Unexpected EOF error

If you wanted another example where good error handling in code could save some time, here's one -

Last week I was making some changes to a small process I've had for ages; the process was all working when I wanted to extract a small bit of information from the message and decided to use the XLANG/s xpath method to do so.

I planned to get the actual xpath string from a helper class (I always do, to help with maintenance), so I've create a variable for it, but as a temporary step I've put the xpath as the initialising value for the variable.

When trying to build the project I've received an error in the error list  - "Unexpected EOF".  not very useful.

This could have taken me ages to figure out, but luckily I knew the only thing I changed was adding that variable, so I changed the initialising value to TEST and tried to build; this time the error was much more meaningful -

"identifier 'TEST' does not exist in '<process name>'; are you missing an assembly reference?"

as well as -          

"cannot find symbol 'TEST'

Just out of curiosity I tried again with the value of TEST THREE WORDS, this time it looked like

"identifier 'TEST' does not exist in '<process name>'; are you missing an assembly reference?"

and

"cannot find symbol 'TEST'

as well as

unexpected identifier: 'THREE'

and

expected ';'          

So what do I make out of it -

To start with I should stop forgetting that the values for string variables should be provided in quotes

That having a single quote in a variable value results in the rather cryptic "Unexpected EOF" error

That BizTalk could have done slightly more to validate values of variables at design/compile time and save us developers precious time :-)

Zombies

A lot has been written about Zombies in BizTalk, but somehow they never really caused me too much problem. until today.

In one of our processes we are calling a web service to perform a certain lookup.

That web service’s implementation (whether correct or not) is to return the instance of the class if found or null otherwise.

When the service returns a null BizTalk receives an empty message, which triggers an exception in the process -

Microsoft.XLANGs.Core.MissingPartException: The XLANG/s message has no part at index '0'.  The total number of parts found in the message is '0'. If you expect a multipart message, check that the pipeline supports multipart messages such as MIME.

I’m quite all right with that, BizTalk is definitely correct – we have returned an empty stream as a message which means nothing.

And so - we've added an exception handler around the call to the service and our orchestration’s flow carries on correctly all the way to the end shape. as expected.

However, contrary to the expected at this point the orchestration instance gets suspended by BizTalk with the error - “The instance completed without consuming all of its messages. The instance and its unconsumed messages have been suspended.”

I believe this behaviour has been introduced in BizTalk 2006 - 2004 was quite “happy” to have the so called "zombies" in the process, and it was up to the developer to handle those (or not, as the case often was); since 2006 this had become an error and if a process ends with unconsumed messages the process gets suspended and the unconsumed messages are available for inspection.

Generally, I think that making a zombie an explicit error is a good idea, although it would have been nice to be able to set whether this behaviour is required on a per-process basis, however - in our case we found this very problematic - we were in a position where we couldn't consume the message properly (due to the exception handler) but although we have handled the exception we ended up with a suspended in stance as it appeared as a zombie. we could not win.

I have opened a support call on this one, waiting to hear if the product group agrees that this is incorrect behaviour or not (I seem to be getting conflicting thoughts). I'll try to keep this post updated when I get an answer.

(You can read more details about this here  - http://technet.microsoft.com/en-us/library/bb203853.aspx and http://blogs.msdn.com/biztalk_core_engine/archive/2004/06/30/169430.aspx

An interesting point is that this case is not listed in the list of causes for zombies in these articles, which makes me wonder – should an exception raised through a receive shape, which was properly handled by an exception handling result in a zombie?)

Which System Exception?

I'm almost embarrassed to post this one, but it send me chasing windmills for a couple of hours, so if I can save that from any other unfortunate person it served a purpose I guess -

I have a call from one orchestration to another.

In the calling orchestration I wanted  to catch any exception that would occur on the called orchestration (or anywhere else down the line) and so I swiftly wrapped the call in a non-transactional scope and added an exception handler.

Being the keyboard-fan that I am (BizTalk is completely unfriendly to us keyboard types, but still...) when the type selector opened I did what I always do - I types the name of the class I wanted (System.Exception) and pressed enter.

The problem was that I did not type the class name ('Exception') but the full name ('System.Exception') and that got me System.SystemException class as the selected type, only that I did not realise that at the time.  apparently the type selector ignores special characters being keyed in?!

It was much later that I realised exceptions that were thrown in my called orchestration were not being caught in the calling one and even then it took me a good couple of hours to realise what was the reason (I simply did not look carefully enough at the exception type - System.SystemException just looks like System.Exception when you (well - I) glance at it!

Three take ways for me -

1. Pay more attention when doing even the most trivial tasks. (trivial, isn't it?)
2. Consider being less zealous about using the keyboard when doing BizTalk development
3. But seriously - consider always having a general exception handler to catch whatever exceptions were not caught by other handlers. I should have been mostly covered by catching System.Exception (although BizTalk, unlike C# or VB.net does support exceptions that do not inherit from System.Exception, see Charles Young post on the subject), but this could have helped at least identify the problem.

Where is that orchestration?

I like visual designers. what I don't like is visual designers that try to protect developers from themselves; they always seem to go just one or two steps too far.

I find myself (and others, I believe) complain fairly frequently about the orchestration designer not letting us do this or that...and while in many cases it all makes sense (you really can't, or shouldn't be doing that), being over protected can cause a lot of confusion and waste precious time. here as example of something that happened to me a couple of weeks ago -

I was configuring a start shape in one of my processes, and when I tried to select the orchestration I wanted to start, it simply did not appear on the list of available orchestrations.

It took me a while - I rebuilt the assembly with the started orchestration and refreshed the assembly with the starting one - twice, but could not get it to appear. I've tried everything - including resorting to restarting visual studio -but could not see my process.

Only when I checked carefully the parameters requested by the started orchestration did I spot the problem - the orchestration had a 'ref' parameter.

Apparently orchestrations that have out or ref (I suspect they are pretty much the same in BizTalk) parameters do not appear on the list of processes in a start orchestration shape.

This makes sense really, as the start shape is asynchronous there's no way to use ref/out parameters with it. I just think a better design decision would have been to flag such selection as an error (at design and compile time) and not simply hide the irrelevant orchestrations.

This way it is immediately clear why I can't select the orchestration I planned to, I don't have to figure all of this out every time.

Have you spotted what the terminate shape does?

The list of shapes in the BizTalk toolbox is not a very long one and so - 4 years after BizTalk 2004 was released I find it strange to discuss the behaviour of a simple shape like the terminate shape; the thing is - that I never quite paid attention to how useful it can be, and so I figured others may have missed it as well.

Now - it is not like I'm talking about anything revolutionary here, just a small oversight on my part.

In essence the terminate shape is as straight forward as anything can be really - you put it in your workflow and behold - your process, upon reaching this shape, would terminate!

The shape takes one "parameter" - a string (or anything that returns a string) which would be the "reason" for termination.

I've known for a while that this string would appear in the HAT query results list as the reason for the termination, and so it is quite useful from operational perspective (why did that process stop again?)

However, I always claimed that terminate is really intended to be used for error scenarios, and not to situations where the process simply reached a point where it should stop processing; I have event, in a previous post, suggested introducing a new "end" shape; in that post I have mentioned a possible alternative - using the Terminate shape - which was also suggested in a comment by an anonymous, but as I believed that the shape is really meant to be used for error scenarios I felt this was not ideal, but at the time this was more a gut feeling thing then anything else.

Today I noticed for the first time the label used for the termination "reason" - it is called "error info", which to me is the "proof" I needed (at least to satisfy myself) that the terminate shape was indeed intended to be used for error scenarios; same error info appears when you view the message flow in HAT of a process that has been terminated , at the top section with all the general details about the process you will find an 'error info' label; any text provided for the terminate shape will be shown there.

I've spotted this only because, being the hard headed guy that I am, I have a few orchestration that have a terminate shape as the last shape in the process. "what is the point in that??" I hear you ask...well - this is why I looked at my decision again.

Well - there are a few possible scenarios - here's one - we have a process that is exposed as a web service. the process initiates several sub-processes (using a mixture of call and start orchestration) and then returns to the caller with a response.

If we have errors in the process we keep track of them in a helper object and return them as a soap header (with or without a response) so that the client is aware of them.

using the terminate shape at the bottom of the orchestration (if my errors collection is not empty) I can report the errors to HAT and make them visible to our operators as well; instead of the orchestration showing just showing as completed, they get a hint through the status that something did not go smoothly and by inspecting the error info field they can find out what it was.

yes - I know I can use the event log - but this way it gets logged with the process in HAT which we can easily find.

...and then just when you actually needed HAT..

I'm pretty sure I'm not alone suggesting that the HAT tool is somewhat, let's say, lacking....

There's quite a few annoying things about the tool, but there's one thing in particular that has to be at the top of the list, because, in my view, it means that in the one case that you really need HAT to help you out, it fails you miserably.

The "orchestration debugger" is a nice selling point for BizTalk: you develop your process and, assuming you have the relevant tracking settings turned on you can go back to processes already completed and "replay" them to see which shapes have been executed and which haven't.

This is really great when viewing processes already completed, and also somewhat useful when setting  a breakpoint in the process and attaching to the process in HAT (although not as useful as one might think).

However - it is completely useless when dealing with suspended orchestrations.

If your orchestration get's suspended for whatever reason, you get a nice error message in the event log, in most likelihood the event log message will even contain the name of the shape in your orchestration in which the exception occurred; however - find the suspended instance in the admin console or in HAT, open the orchestration debugger - and you're into a surprise: the viewer will only show you execution up to a few shapes BEFORE the actual shape that failed.

I'm not sure I have the story right, but I believe this "bug" (it's really a "side effect", more on this in a second) was introduced in 2006 (but I no longer have 2004 installed to prove it was actually better before hand).

As far as I know, one of the changes in BizTalk 2006 is around the way orchestrations handle exceptions - in BizTalk 2004 all unhandled exceptions in an orchestration would (if my memory serves me right) result in a suspended non-resumable instance; in 2006 these instances are resumable; this suggests that BizTalk 2006 has to keep the state of the orchestration BEFORE the error occurred - so that if an administrator chooses to resume the process (possibly after fixing whatever caused the suspension) the process could start again and retry the action where it got suspended before).

In order to achieve this BizTalk probably keeps the last GOOD state of the orchestration in the database (from the last persistence point executed); in other words - where before a suspension would cause a persistence point, from 2006 it does not and the orchestration's last persistence point is what's kept in the message box.

If that is correct it would explain why the orchestration debugger only shows information up to a point before the shape that caused the suspension - it would only have information up to the last persistence point.

I don't know if this was an oversight when releasing 2006 or a conscious sacrifice, but I think it's a big pain point; it would have been great to see it all - see where the exception occurred, see the state of the orchestration at this point as well as having an indication as to where was the last persistence point - so we could tell what will get executed when we resume the orchestration.

There are quite a few good uses for HAT - it's a great tool to know what's been executed on the server over time; it's not a bad tool to take a look at the duration it took for a service to run, it's even somewhat useful to check the flow of a particular message through the engine using the message flow or the orchestration debugger view - but when it comes to helping out finding out a cause for a suspended orchestration - it is quite pointless for that reason.

So if you counted on it bailing you out when your process fails - you may as well switch off orchestration shape tracking for your processes.

Why the New Configured Port Wizard is confusing

It is generally known, I believe, that BizTalk has somewhat of a steep learning curve.

BizTalk server is by no means a simple product, but that's ok because we tend to do complex stuff with it :-)

Microsoft, over the years, from version to version, invested a lot in making the server easier to learn and use, some improvements were made in the UI, but mostly through investments in the documentation, tutorials, examples and "community content"; I do believe this has made BizTalk much more accessible.

There are still quite a few things in BizTalk which, from my experience, tend to confuse new starters; one of them is the concept of orchestration ports and port type and the "New Configured Port Wizard".

I find that too many developers use BizTalk without fully understanding the fact that it is a strongly typed system and without understanding the relationship between ports and port types (and messages and [multipart-]message-types).

This is not helped by the fact that you can quite easily develop an orchestration without explicitly defining either.

The new configured port wizard defaults to creating a new port type whenever you configure a port; I suspect many developers never give it a second though and simply create a type for each port they use; this way you can easily create quite a few copies of the same type and not even recognize it.

further more the wizard creates the port and the port type, but only completes the port type definition when you connect the port to a receive/send shape that has a message configured (the message-type portion); I believe this further confuses people as 1) it does not make it clear that the message type is indeed part of the port type definition (along-side the access modifier and the message exchange pattern) and 2) as the port type definition does get completed when you create the link, many developers do not understand why they cannot replace the message in the receive/send shape, or connect another receive/send shape to the recently created port (because the message types may not match).

If I had to guess I would say that the reason for the way the wizard works is the assumption that it lowers the entry barrier to developing BizTalk processes as people can develop processes without understanding the concept of types) but, in my view, here lies the problem - developers produce code they do not fully understand with all the problem that creates.

Orchestration Statuses

I was surprised to find out there's some confusion around the possible statuses a deployed orchestration can be in, but after spending 20 minutes browsing MSDN I realised I simply couldn't find one clear description of those, so here's my attempt -

An orchestration deployed into BizTalk server can be in one of four states -

1. Unenlisted (unbound) – the process has been deployed to the server, but is unconfigured (host and/or port bindings are not set), unsubscribed and is not running.
2. Unenlisted (bound) - the process is configured, but is still unsubscribed and is not running.
3. Enlisted (stopped) – the process is fully configured, subscriptions have been created, but it is stopped.
4. Started – process is ready to run (and will do so when activated by a message)

The first two are obvious I should think, so is probably the last state, the Enlisted state however, may cause some confusion - when enlisted but stopped the orchestration's subscription is active on the message box meaning it will get evaluated for every message published. if the evaluation succeeds, the message will get queued for this orchestration but it would not start (as it is in stopped state). the messages will remain waiting until the orchestration is started or they are terminated.

Creating a message "from scratch"

A question I get asked repeatedly is how to create a message in an orchestration “from scratch”, i.e. – when the message is not meant to be created from any other message.

Initially one might think this does not make sense, but I find myself doing it quite often actually; two typical scenarios are when one has to create a message before branching the process to satisfy the compiler (and avoid the “use of unconstructed message” error), a second is when one needs to return a message that simply contains few values obtained from, say, a database call, or some calculation etc.

In this case often it is simpler to create the shell of the message with the elements/attributes required and then use xlang’s xpath function to push the values into the right places.

Whatever the scenario is, there are, I believe, two options to create empty messages (corresponding to the two options to create messages in orchestrations in general, actually) –

1. Using a map
2. Using a message assignment shape (and a helper method).

The first one is quite obvious – you create a map, pick any message you may have in the process (you are likely to have at least one message in your process already) - use that as your input message and the message you want to create as your output message; then you’re facing two alternatives – the first one is to use xsl – you can create an xsl file that effectively has the XML you want to create hard coded in it and instruct the map to use that xsl.

The input message is completely ignored (and so is completely irrelevant) and the output message is always the way you want it to look like. You are not sensitive to any changes in the schema of your input message.

The other alternative, which I like less, is to actually use the mapper; in this alternative you would probably map the root node of your input message to the root node of your output message and then set up default values in the output message for any element/attribute that you wish to include in your output message.

The reason I believe this is not as good as the first alternative is that it is less obvious how the output message would look like; one has to follow up on the nodes to see what is set (or test the map) to see the output while in the xsl alternative one look at the xsl is usually enough to show what the output is going to look like.

Either alternative you choose – I used to think that using the mapper is a better option (as opposed to using the message assignment shape which I will describe shortly) – mostly because I thought this is a more standard way to create messages, and so it is more obvious, looking at the process and the project, where such constructions take place but mostly,I believed, if you’re using xsl files to create the output it would be very easy to spot, read and change them in the solution when necessary (simply find the xsl files and change them, no need to look for anything else).

Thanks to several people, but mostly to Ben Gimblett with whom I work with in one of the projects I'm involved with and who has insisted not to follow my advice and use helper methods to create messages, I now agree that using helper classes is the better way, mostly because using them you don’t have to use a dummy input message (as you do in the map) – which, I have to agree, can get quite confusing to anyone trying to understand the process but also because, I suspect (but have not tried to prove), a helper class will perform better than the mapper option.

When using a helper class you again have to alternatives –

You could have a message assignment shape in which you call a method that returns a .net type (class) that has been generated (using xsd.exe) from your schema.
All you need to do in your method now is create an instance of that generated return type – populate whatever members you require and return it.

In the assignment shape you assign the return value from the method to your message[part] and so BizTalk will take care of the serialization and will convert the class to the schema and because the class and the schema both represent exactly the same thing the serialisation would work just fine.

The benefit of this approach over the mapper option is mostly that there's no need for any dummy input message, no need to write xml or xsl; only very simple (and quite minimalist) code is required.

The downside – you need to generate those .net classes to represent any schema you wish to return, and maintain them as your schemas evolve.

The second alternative is simpler on the one hand as it does not involve generating and maintaining classes; it does, however, require a bit more wiring –

It starts the same way – a message assignment calls a method whose return value is assigned to the constructed message[type].

The difference is that the method does not return a strong type; instead it returns an XmlDocument whose contents are loaded from compiled resource within the assembly.

The function takes in the name of the xml that needs to be used to create the constructed message, retrieves the resource from the resources in the assembly, loads it into an xml document and returns it to the caller.

I find that this last approach works best for me in most cases – all the generated xmls are in one place which makes them easy to maintain (which I liked in the xsl option), there’s very little co-ordination that’s required – only the name of the xml file (or any other key one wishes to use) must be known to the caller and the xml resource should match the schema – but as it is stored in one location AS XML this is very easy to achieve and maintain.

Wish list: "End" shape.

Often orchestrations have conditions that, when met, mean the orchestration should stop executing.

In .net code a developer could simply put a “return” keyword in the function to stop the execution and return to the caller; unfortunately there is no equivalent in orchestration development.

The only “clean” way to end an orchestration currently is to reach the built-in, fixed, red shape at the bottom of the orchestration designer.
This often means that decision shapes exist all over the orchestration with branches that have to drop all the way to the bottom. In large orchestrations this can be hard to follow up on.

It could have been made much nicer by allowing a developer to add as many “end” (or “return”) shapes as needed in various places in the process. Decisions, of course, will still exist, but their scope will be much smaller.

An alternative that exists currently is to use the terminate shape, but I find it to be a bit artificial – my view is that termination of processes may be relevant in exceptional scenarios, as part of error handling, etc. but not when the orchestration ends correctly (albeit before reaching the end of the schedule). The main problem with the terminate shape possibly is that the orchestration gets a “terminated” state and not “completed” which will make it difficult to report in HAT (and otherwise)

Recursive Orchestrations - follow up

A few weeks ago I've posted this entry about my short, unsuccessful attempt to call an orchestration from another instance of itself using the call orchestration shape and thus introducing recursion.

In comments to that post G Michels pointed out that it seems to be impossible by design according to a very specific error he'd seen.

Adam, however descirbed a very neat approach that is pretty close to recursion (although not quite there in my view) and so I spent some time looking into this. here is the outcome -

The scenario that Adam intorduced is calling an orchestration from another instance of itself. as such this is not exactly a recursive call as, unless more effort is made, the call is completely asynchronouos - the calling orchestration will not wait for the called orchestration to complete, nor will it receive any feedback from it's operation.

The idea is to use the publish-subscribe mechanism BizTalk provides to kickstart another instance of the process; however, Adam's idea of using the Listen shape when activating the orchestration and using partner ports for exchanging messages between instances makes the implemenetation much simpler than it would have been if we tried to do the same thing using basic direct ports and subscriptions and thus it is much more maintainable.

Here's a screenshot of the process I've created following Adam's description -



There are two key points to this idea -

The first one is apparent from the start - the orchestration is started using a couple of activating receive shapes inside a listen shape.

This allows the orchestration to be activated by one of two ports; without this we'd have to fiddle around with promoted properties and subscriptions to get this to work which would have been much more confusing.

In my example the left receive port is late bound to a file drop that would start the process initially.
The right branch is another activating receive shape, directly bound to the message box, but we'll get to that in a minute.

The message I drop has a count attribute initially set to 1, and is configured as a distinguished proeprty.

This allows me to check in the decide shape that whether count is lower than 3 (just so I have some stop condition).

If it is - I contruct a new message by copying the existing one and increasing the count by one.

the message is then sent out using a send port, and this is where the second key point comes into play - the send port is direct bound to a partner port - the second receive port in the same orchestration.

What that means is that, again, without worrying too much about subscription, we are able to publish a message to the message box and ensure it will be picked up by an orchestration of our choice - in this case the very same orchestration, into a particular port - the second receive port linked to the listen shape.

The second half of this - the partner receive port is configured, quite counter intuitively, as Kevin Lam explains in his fantastic post, to have itself as the partner port.

..and so, by sending a message from the partner send port, we are triggering a second instance of the same process in a recursive-like manner.

As I've mentioned earlier this is still somewhat lacking as the calling orchestration will simply carry on (and complete in my test case) and not wait for any called orchestration to complete and will not be able to receive any feedback from such processes.

To support that I had to add a few bits to the scenario, which did increase it's complexity beyond what I would have hoped - here's the modified example -

The first half remains largly unchanged -



As you can see I've added a boolean variable to indicate whether I'm "inside" a recursion or not and am setting it to True on the left branch of the listen shape (after making sure it's defaulted to False) - this would tell me later whether there's something waiting for a response from this instance of the orchestration or not.

Also, for convinience and nothing else, I'm now keeping the iteration number in a variable.

But in order to achieve my goal I had to make a few bigger changes - here's the second half of the process -



The first addition is a receive shape after publishing the recursion request - this would make the process sit and wait for a response from the initiated process before proceeding.

As you can imagine that involved initializing a correlation set in the previous send shape which uses the iteration number, and following on it in the receive shape, which in turn meant I had to introduce a property schema to define my context property (and changing the schema to promote it).

Another consequence of this is that I could no longer use the same messsage type all around - if I did I the receive shape will simply get the published message before any second process is initiated; I had to create a response schema and use that for the received message type.

The second change was - after the resposne is received - to check whether the current iteration is the first one. this is done by checking the boolean variable I've set in the listen shape - if I'm not within the first iteration, I need to publish a response to the message box initializing a similiar correlation set (with the iteration number, but using iterationNumber-1 to link it to the parent iteration) to ensure the calling process is resumed.

This message could be used to provide any feedback to the calling process if needed.

By now it should be obvious this is way more complicated than simply adding a call orchestration shape and selecting the same orchestration, which is all we had to do if BizTalk did support recusrions out-of-the-box. in fact - this grew to be so complicated I think I will have to consider 10 times before choosing to use this approach in any production environment.

Either way - I found this a very interesting exercise to go through. hope you have as well. and thanks to Adam for sharing the idea.

Thoughts about static members and local caching in BizTalk Server

(sorry for the long post)

I've recently read Richard Seroter's very useful post around static objects in BizTalk server and singleton classes, which also lead me to Saravana Kumar's post about using static classes to implement caching.

Like most BizTalk developers - I assume - I use static methods quite often; mostly they come handy to developers in environments like BizTalk Server because in our world we already have a good "container" for state - the orchestration class, and so quite often we simply need a piece of code to help us complete a very specific, atomic, task that does not require a separate state to be maintained.

Of course having static, stateless, methods also helps avoiding certain serialisation issues we would have had to deal with had we used a class instance.

However, that is not exactly what Richard and Saravana are talking about - both posts (although they have tackled this from slightly different perspectives I believe) - their classes maintained state but provided a static interface - both examples had static members to hold information;

While under some circumstances having static members is a necessity, or at least very useful, it introduces a fair amount of complexity into the design of the process - with the level of complexity varying depending on the actual requirements from the static data.

As I believe Saravana described a more specific use of static members - local caching - I thought I'd tackle this one first; I share the view that while caching is often introduced to improve performance, it also often has exactly the opposite effect (or no effect at all); Saravana writes at the start of his post -

"There is no necessity to explain the importance of caching in any server based developments like BizTalk, ASP .NET etc. You need to plan early in your development cycle to cache resources in order to make most out of your limited server resources."

I think this is inaccurate, at least as I understand it, but I guess it really depends what "resources" means in the context -

With a server application such as BizTalk server "limited server resources" frequently refers to the server's memory, in which case, local caching of data will actually put more strain on this resource; it might improve performance as it can save roundtrips to the database server, especially if the database server is particularly busy or the network is not as fast as it should be, or it might slow the server down as less memory is available for it to do it's job. anyway - if you include memory in your "resources" - local caching will hardly help you make the most of it.

In many cases going to SQL server to do that lookup every single time ends up being faster (or at least not slower) than caching the data locally. The reason being that storing all this information in the server's memory consumes resources as well; if you have several hosts that use the data you will need to cache it in each host (or AppDomain) consuming even more memory; now - considering having enough available memory is key requirement for a server to run fast - this can easily get quite painful.

To make matters worse - if you have multiple servers in your farm (and you should have) - you are now repeating this on each server; but more importantly - now you have to worry about making sure both caches are ALWAYS in synch, otherwise you'll get unexpected results from your processes.

You also need to worry about having some mechanism that ensures data is up-to-date (maybe you have to go and retrieve newer data periodically, after the data's expiry time or anytime expired record is being requested), maybe you need some mechanism to invalidate the data in the cache when someone/something updates the source of the data?

The list goes on and on but bottom line is - your code becomes quickly more complex - which means it is slower, it is harder to maintain and harder to fix - which means it may not be as optimal as you want it to be and also you've used a lot more memory.

Now, if we're talking about data coming from SQL one has to remember that SQL, as far as I know, is quite smart about it's own caching strategy, so if you're using the data frequently enough it will be cached anyway (and if not you shouldn't be caching it anyway).

Again - I'm not saying that caching is always bad - but it is definitely something that needs to be justified before it should be implemented and so should not necesarily be "planned early" but rather implemented when it becomes obvious data access is indeed a problem.

If, for example, you have to work with a very busy SQL server, or the network is not as fast as it should be, or the process of retrieving the data is quite long and complex and thus slow (probably worth revisiting in that case, but anyway) - than it might make perfect sense to cache the data - but if it's a simple lookup of some sort or something similar - more often than not caching will not bring a significant improvement in my experience.

Richard's post, as I understand it, speaks much more generically about how to implement a singleton pattern to provide access to static data in a thread-safe manner; my thoughts around this are quite similar - if you have to have static members, this is probably the way to go, but it should be avoided whenever possible -
The singleton pattern is a very useful pattern, in particular when you need to share data between processes, for example between different instances of an orchestration, or between instances of a running pipeline (in a host).

What I think was not clear from Richard's post (but it could well be just me) is when this is useful and when it is not - Richard clearly illustrated that, in BizTalk, the very limiting fact that the messaging engine and the orchestration engine are separate systems, means there's no simple way to share data between them and while technically, within any one sub-system (in an AppDomain, in a host), a singleton pattern can provide a good way to share data; the hosting model in BizTalk means that you cannot assume, for example, that information that exists in a receive pipeline will exist in the send pipline; or that information that is available for one process will be available for another (as they all may be hosted by different instances) although

What is worth stressing (and again, it might just be me being difficult) that you don't have to introduce a singleton pattern every time you need to use static methods - if you don't have to ensure that in 100% of the time there will be only one instance of the class you can safely, I think, simply use static members.

And more importantly - if you don't have static members - you don't need worry about this - and so while this pattern is very useful it mostly demonstrates the complexities around keeping static/shared information in products like BizTalk Server; data that is not owned by the specific instance of the process; and so cannot be "managed" by it.

I'm not even going into the complexities around the persistence of the shared data (I don't believe static members will be persisted when an instance dehydrates, so what happens when the server/host is being rebooted? What happens when BizTalk decides to free resources and remove assemblies from memory, etc.

I guess the main point I'm trying to make, in what has become probably a too long post, is that processes manage state and a lot of effort is going into that, and anything outside that has to be maintained separately which adds complexity and has to be justified.

Recursive orchestrations?

I never imagined I'd find a case for this, but I did today - I could use a recursive orchestration.

I have a simple orchestration that takes a request message and calls a web service with parameters our of that reuqets (after doing some work on them and making a couple of checks).

In certain situations the orchestration needs to call the web service more than once - so I figured - theoretically, instaed of reaeating the web service call in the orchestration or trying to get it in a loop, I could simply create my own request message with the new parameters and call myself. classic recursion.

Only that BizTalk does not let you do this, and I suspect this is just a UI limitation - it is as simple as the fact that when you open the call orchestration dialog you see all the orchestrations in the assembly but "yourself".

If I had some time I would try editing the ODX file directly to get this to work just to prove whether this is just a UI limitation or not, but I can't afford taking the time now and anyway it would not be helpful as editing the ODX files is ALWAYS a bad idea!

Interesting

(I've added a follow up on this here)

Orchestrations and the Obsolete attribute

I've recently posted this message; which meant we had to go back and change quite a few methods shared between quite a few orchestration.

considering the size of our project finding all the places that needs changing was not a simple taks, nor could we break the interface and assume we'll find and fix all the places before our next build is due; a more planned and carefull appraoch was required and so we've decided to add the Obsolete attribute to the offending methods and overload them with ones that take in XlangMessage to be used everywhere.

We were hoping that adding the attribute would result in nice warnings appearing in every build that uses the obsolete overloads which would help us identify, and eventually change them all.

However, to our surprise, BizTalk (in the most general term one could use 'BizTalk') is being a bit inconsistent in it's handling of the Obsolete attribute -

If we use a method called MyMethod in an expression shape in an orchestration, and we add the Obsolete attribute to the method -

[Obsolete("Some Message Here")]
MyMethod(XlangPart part)

When we build the orchestration we don't get any warnings as we'd expect which makes the Obsolete attribute quite useless (and is very inconsistent with .net in which, obviously, the use of the attribute is supported)

Whats strange is that if we change the attribute to raise an error -

[Obsolete("Some Message Here",true)]
MyMethod(XlangPart part)

Not only we see this error when trying to build the orchestration we would also see any other WARNINGS caused by the use of the attribute elsewhere in the project. suddently they are all visible.

Downcasting and orchestration parameters

Here's a tricky one that, for some odd reason, made us all smile.

We have a lightweight .net class (I'll call it X) we often use in our orchestrations (to do some tracing work for us, if you must know); we will quite often pass this class as a parameter to any called (or started) orchestrations.

Recently we've created another flavour of this class through inheritence; the new class (Y from now on) inherits the original class and extends it.

Having the new class defined we went to one of our processes and simply replaced the variable type from X to Y; thanks to the inheritence we did not really have to change any code other than adding the new logic we needed.

However, this orchestration was calling another one, in which the parameter type was still X while the paramter we were passing in has now changed to be Y.

In .net this is perfectly legal and the object would be down-casted to the base type with no issues.

I'm happy to report that BizTalk is no different (which shouldn't really be a surprise) and indeed our orchestrations still executed just fine. Y was being used by the first process, and then was passed in to the called process which used it as X. nice.

There is one major problem though - the orchestration desginer does not seem to support this scenario - if, like we did, the change of type is an aftermath thing (i.e. type change is done after configuring the call orchestration shape) everything is working just fine.

If, however, we need to introduce a new call orchestration when we have Y and the orchestration parameter is configured as X, or we try to modify an existing call orchestration shape the designer will not let us select a variabel of type Y for a parameter of type X.

We see the parameter type of X in the deisgned and quite simply are not given the change to select any variable of type Y.

This is unnecesary and quite limiting as it means we cannot call the orchestration from different processes passing either X or Y depending on what we have in the calling processes, we have to either restrict ourselves to one type (i.e. migrate all of the system to Y, which is not ideal) or do the downcasting ourselves before passing the parameter.

We were almost tempted to fool the designed by always changing the type after configuring the call orchestration shape, but this is unmaintainable, so we're resulting to changing the paramter type where possible or downcasting ourselves before the call orchestration shape where not.

Are you passing XlangPart to a helper class?

Then dont!

(note: there's an important update relevant to this post, you should read it here)

Generally I'm not a big fan of manipulating messages in code (as many people will testify after having long discussions with me on the subject); however - reality often comes into play and I find myself being able to clearly justify it.

This is uaully the case when the complexity of the operations required is such that it would be very difficult (and so less maintainable) or much slower to do so in a map (using custom xsl or not).

Then, of course, there are all the other cases where you simply need to pass in a message part to a helper method for other reasons as well, not necessariliy to modify them.

In any case - I've recently stumbled accross this article that suggests passing around message parts may cause a reference leak.

Bottom line is that one should always pass in the xlang message and resolve the part in code rather than pass in the xlang part itself.

Just though it's worth sharing...

Boolean variables in orchestration

Can anyone explain why the default to a boolean variable in orchestrations is True?
This is counter intuitive (as it is the opposite in c#) and I'm constantly hit by it (when will I learn!?)