Which System Exception?

I'm almost embarrassed to post this one, but it send me chasing windmills for a couple of hours, so if I can save that from any other unfortunate person it served a purpose I guess -

I have a call from one orchestration to another.

In the calling orchestration I wanted  to catch any exception that would occur on the called orchestration (or anywhere else down the line) and so I swiftly wrapped the call in a non-transactional scope and added an exception handler.

Being the keyboard-fan that I am (BizTalk is completely unfriendly to us keyboard types, but still...) when the type selector opened I did what I always do - I types the name of the class I wanted (System.Exception) and pressed enter.

The problem was that I did not type the class name ('Exception') but the full name ('System.Exception') and that got me System.SystemException class as the selected type, only that I did not realise that at the time.  apparently the type selector ignores special characters being keyed in?!

It was much later that I realised exceptions that were thrown in my called orchestration were not being caught in the calling one and even then it took me a good couple of hours to realise what was the reason (I simply did not look carefully enough at the exception type - System.SystemException just looks like System.Exception when you (well - I) glance at it!

Three take ways for me -

1. Pay more attention when doing even the most trivial tasks. (trivial, isn't it?)
2. Consider being less zealous about using the keyboard when doing BizTalk development
3. But seriously - consider always having a general exception handler to catch whatever exceptions were not caught by other handlers. I should have been mostly covered by catching System.Exception (although BizTalk, unlike C# or VB.net does support exceptions that do not inherit from System.Exception, see Charles Young post on the subject), but this could have helped at least identify the problem.

Is Michael right?

Michael Stephenson has suggested here that in his opinion the pipeline as a component is obsolete.

While something told me as I was reading this I could not agree with him 100%, I simply could not put my finger on the reason, I could not come up with a good enough argument against it; in fact - I still can't - which probably means he is right, but I have thought of the following two points -

The first one is not a justification, but a possible explanation of how things came about - as far as I understand it pipelines have always meant to be created at design time, then "compiled" and deployed; in 2004 there wasn't a straight forward way of using per-port configuration of pipelines, but soon enough the information that this can be done, and how to do so, became public and some developers (not all, obviously) have started using this method.

It was only in BizTalk 2006 that the ability to configure a pipeline was exposed in the UI; I guess that this points out that it was not the planned use initially (and also that MS definitely "listens" to the community, or market - call it as you wish)

It is worth noting that Michael does not stop at the pipeline component's configuration, but suggest the entire pipeline, i.e. - which components exist in the pipeline and in what order - should be configurable; this is taking the per-port configuration to another level, but it does make sense.

The second point I have in mind is that there's one benefit of having the pipeline as an artifact of sort, and that is re-use (and I'm guessing this was at least one of the considerations by MS, when making the decision to make the pipeline a compiled component) and in my experience I do find pipelines fairly highly reusable artifacts.

But this last point is not a very strong counter-argument (not that I intended it as such), just expanding on Michael's opinion and suggesting that the latter should be provided similar to ports, which are completely configurable through binding files, but are re-usable.

If we could create a pipeline in the admin console as an entity on it's own right (or through bindings) and then select it in ports, we could have the best of both worlds, mostly.

There would only be one thing I can think of that we would be missing out on, and that is the rich design environment we get when creating the pipelines in the pipeline designer - and I'm not talking necessary about the drag-and-drop way of adding components, I could not care less about that if I'm honest, but the rich property editor which is not currently supported in the generic per-port-pipeline-configuration.

Again - I'm sure this can be handled in the same way that adapter settings are handled in ports, given enough attention.

So - Michael - I am officially supporting your idea - where do I vote?

Microsoft has announced the publication of the new "Performance Optiomisation Guide" written by Ewan Fairweather and Rob Steel.
I would expect this one to do wonders for some of us out there trying to give a little bit more "juice" to our BizTalk impelementation - grab it here.

Where is that orchestration?

I like visual designers. what I don't like is visual designers that try to protect developers from themselves; they always seem to go just one or two steps too far.

I find myself (and others, I believe) complain fairly frequently about the orchestration designer not letting us do this or that...and while in many cases it all makes sense (you really can't, or shouldn't be doing that), being over protected can cause a lot of confusion and waste precious time. here as example of something that happened to me a couple of weeks ago -

I was configuring a start shape in one of my processes, and when I tried to select the orchestration I wanted to start, it simply did not appear on the list of available orchestrations.

It took me a while - I rebuilt the assembly with the started orchestration and refreshed the assembly with the starting one - twice, but could not get it to appear. I've tried everything - including resorting to restarting visual studio -but could not see my process.

Only when I checked carefully the parameters requested by the started orchestration did I spot the problem - the orchestration had a 'ref' parameter.

Apparently orchestrations that have out or ref (I suspect they are pretty much the same in BizTalk) parameters do not appear on the list of processes in a start orchestration shape.

This makes sense really, as the start shape is asynchronous there's no way to use ref/out parameters with it. I just think a better design decision would have been to flag such selection as an error (at design and compile time) and not simply hide the irrelevant orchestrations.

This way it is immediately clear why I can't select the orchestration I planned to, I don't have to figure all of this out every time.

UK SOA/BPM user group meeting

The first meeting of the UK SOA/BPM user group is just around the corner on the 17th of July.

You can find all the details here

See you there.

Failed to register adapter <...> for receive location

Everyone gets this error in the event log every once in a while -

The Messaging Engine failed to register the adapter for "<...>" for the receive location "<...>".

Please verify that the receive location exists, and that the isolated adapter runs under an account that has access to the BizTalk databases.

Mostly I get this when I publish a new BizTalk web service and forget to selected the correct application pool (using windows service 2003).

Often this comes up when setting up an XP (or Vista) workstations for the first time and forgetting to sort out ASPNET permissions.

Sometimes it is simply a typo in the receive location's url, or a problem the script that was supposed to enable it, all of those are pretty much suggested through the event log entry message.

But earlier this week I had another case, one which never happened to me before (and, arguably, shouldn't have happened at all, but if it did for me, there must be another idiot who would do the same! :-)), and that one is not mentioned in the message -

I have published an orchestration web service, selected the correct application pool and made sure the receive location was enabled but still this error would be logged.

What I didn't do (I blame it on having a cold) is to actually create an instance of the host under which the receive location is configured to run; I had a host configured and selected but no host instance...

Flush failed to run error when using DES

A quick one -

If you're using one of the DirectEventStream BAM API and you're seeing an error "Flush failed to run", check the connection string (and related permissions) to the database.

Happened three times to me recently (don't ask!) - all of which were related to configuration problems.

Do we need schemas?

This is somewhat of a recurring theme with me recently, but I want to discuss the contents of the management database; more specifically I want to discuss the fact that schemas get deployed to it and that most other things deployed will have a strong dependency on schemas.

As schemas are always at the bottom of the dependency chain, this means is that on top of the expected difficulties one can experience when needing to change schemas and the impact on other system, the actual act of deploying a new schema.

At best this is simply an annoyance to a developer who needs to re-deploy his entire solution as the schema evolves through the development cycle (versioning is not applicable in this scenario);

At worst this is an operational nightmare if a solution has to be updated/patched/evolved where a good versioning story does not exist (as is all too often the case, not that versioning would have solved this all).

As we are forced to remove the entire solution and then re-deploy with the new schema, we can expect, from my experience, the process to take quite a while for large solutions, which may take the business offline for a couple of hours.

Taking the risk of making a point about something I don't know enough about - the internal behaviour of BizTalk server with regards to deployed schemas (but one could say this is often the case...) - I would argue that as far as I can tell, schemas are not actually used all that often by the runtime.

(and because I accept I could be completely wrong here, please do share any thoughts/ideas/comments/insights/whatever on the subject - put a comment on this post or email me if you prefer. I'd love to hear some feedback on this.)

Anyway - as I was saying -

When you define a message type you select the schema at design time, and the designer may refer to that schema to do various things - draw the map designer, check validity of assignments in expression shapes, build intellisense, it would even check serialisation an de-serialisation attributes on classes vs. your schema when you try to assign a .net class to a message in an expression shape, but as far as I'm aware, the schemas are rarely used by the runtime.

At runtime, when message is received into an orchestration (and set to a pre-defined message type), it's contents are not checked against the schema; neither does it get validated at the end of a transform or message assignment shapes.

When you run a map you select a schema, but again - that map could well return something completely different; BizTalk couldn't care less.

When do I know schemas get used? in the pipelines. sometimes.

If you're using the XmlDisassembler for example it would try to resolve the message type based on the message's root node and namespace, and then try to get the schema from the database.

the disassembler may then use this schema to promote some properties, if configured it may debatch the message according to the schema and possibly use it to validate the message; all are very valid usages for the schema but - they are not always used, and they require specific configuration, either in the schema at design time or in the pipeline component (or both).

Also, at least with regards to property promotion, all that get's used is a bunch of xpaths provided in an annotation in the schema, not the actual schema information.

There are, of course, other cases where schemas are required - FlatFileDisassembler, XmlValidation, Xml and FlatFile Assemblers all need schemas for their work (to some extent at least) and definitely the design time environment uses them extensively, but what I'm arguing is - can we do without having to deploy schemas if they are not used?

BizTalk works in a late-binding fashion anyway, where assemblies and their contents are loaded from the GAC/database as needed (and may be unloaded after a period of them not being used), couldn't we get away with only deploying the schema when it is needed at runtime, and simply 'register' message types when it is not?

In fact - even if a schema is needed at runtime - why does it need to exist in the database? how is it different from maps, pipelines, orchestrations? all of which are 'known' to the database but physically exist only in the GAC? (well, that's not accurate - the orchestration's structure is stored, as XML in the database, but that's to be displayed in HAT, and possibly a bad design decision on it's own)

I can't help thinking I'm missing something, I'm sure the guys behind BizTalk's decision had given it a lot of thought and found good justification for it, wouldn't they? anyone can comment on what those might be?

One argument could be that BizTalk wants to know which messages are 'supported' by the solution - just as a message arriving with no subscription is considered an error, a message arriving which is not of a known 'type' should be considered an error. but in a sense - the two are the same, and in any case BizTalk is quite happy to support 'blob' messages through the use of passthrough pipelines and XmlDocument as a message type in the orchestrations.

Throttling in full action

Here's another one from the archives (=the list of things I have waiting to be blogged)

At some point we had a sudden peak in system load on our BizTalk processes and, as a result, our BizTalk solution that was running so nicely seem to have gotten "stuck".

In "stuck" I mean - we ended up with lots of processes in "Active" state, but they did not seem to be active at all; a closer inspection (of trace that should have been emitted) showed that although the instances status says "Active" they were all very passive indeed - nothing was executing on the server - close to 0% CPU and no trace whatsoever.

This is where you might expect me to describe the long hours we've spent investigating the issue, the sleepless nights and empty cartons of pizza... - but really what happened is that, not being able to afford any more down time, we called out premier support which turned out to be a great thing because the first thing they did (well, not literally, but anyway) was to ask us to check the state of the server using the MsgBoxViewer which in turn pointed out that we have simply "max-ed out" our memory consumption throttling level.

You see - we use a lot of caching of data in our processes; mostly because we access a lot of reference data frequently - data that does not change very often; this is by design. what we forgot to do is estimate the amount of memory this caching will require when many different clients use the system and adjust the throttling level accordingly.

As you can see from the image below - out of the box the BizTalk hosts are configured to throttle at 25% of the server's physical memory. the idea is to prevent the BizTalk processes from taking up too much memory and killing the server, and the assumption is that if throttling kicks in, and stops processing instances, memory consumption will slowly reduce until the server gets back to a more healthy state. however - from it's very nature - caching does not really release memory that often and so instances have stopped progressing but no memory was released as a result and so we got "stuck".

In our case, the solution was straight forward - as we know our memory consumption will be high, and we know there's nothing else running on the server to compete with that memory consumption (more or less) we could increase the threshold to 50%, which is enough to grant BizTalk Server enough memory for the caching and all the processing requirements.

In the process we monitored the situation by investigating two BizTalk performance counters - "Process memory usage threshold" (here shows as 500MB) compared to "process memory usage" (here showing around 130MB).

As long as there was large enough gap between the two we knew our processes are going to be just fine; it is always important, of course, to monitor these over time to ensure there's no memory leak in the processes, which we have done, on top of peak load tests - which we have not.

Now, while all of this is down to a test or two we may have neglected on our side, there are a couple of interesting points at the back of this from a product perspective -

1. We were confused by what we saw mostly because of the "active" state of all instances (and we had quite a few); we would have diagnosed the problem much quicker, and on our own, had the admin console indicated that the server is not actually processing anything due to it's throttling state.
   
   
2. I can't help but wondering whether the throttling mechanism couldn't be a bit more clever and identify it has reached a dead end and is not actually helping in improving the situation. following on our case the engine realised memory usage has gone too high and has stopped processing instances. wouldn't it be great if after, say, 10 minutes it realised that memory is not actually reducing and so it will never exit the throttling state and would write something to the event log?

Again - not trying to make any excuses, just thoughts with the power of hind sight...

Just in case there's someone out there who didn't hear yet -

Microsoft have just publicly announced BizTalk Server 2006 R3 on Steve Martin's blog here.

Not much more to say on top of that I guess...that's the whole story.

GAC assembly as a post build event

With BizTalk server every DLL we use has to be in the GAC.

Much too often, after making a change to such DLL, I forget to GAC it in time before running a test which would, naturally, result in the test failing; in most cases the error is obvious and I simply have to return to the infamous build-gac-restart host cycle before running my test, but every now and then I get thrown by the error and do not realise it is a simple case of me forgetting to GAC an assembly.

To avoid those annoying moments I have developed a habit of adding the command to GAC an assembly to the project's post-build event.

The command required looks like this -

"$(DevEnvDir)..\..\SDK\v2.0\Bin\GacUtil.exe" /i "$(TargetPath)" /f

and it can be used to make sure that on a successful build the assembly generated will be added to the GAC.

Of course this has downsides - it is quite possible that you do not want to GAC on every build; further more - the post-build event is part of the project properties and as such goes into whatever source control you're using - now you have to worry about all those other developers who might get that code and build it - do they want it in the GAC?

Having said all that I find that - for me - having the post build event is usually better than not having it.

Have you spotted what the terminate shape does?

The list of shapes in the BizTalk toolbox is not a very long one and so - 4 years after BizTalk 2004 was released I find it strange to discuss the behaviour of a simple shape like the terminate shape; the thing is - that I never quite paid attention to how useful it can be, and so I figured others may have missed it as well.

Now - it is not like I'm talking about anything revolutionary here, just a small oversight on my part.

In essence the terminate shape is as straight forward as anything can be really - you put it in your workflow and behold - your process, upon reaching this shape, would terminate!

The shape takes one "parameter" - a string (or anything that returns a string) which would be the "reason" for termination.

I've known for a while that this string would appear in the HAT query results list as the reason for the termination, and so it is quite useful from operational perspective (why did that process stop again?)

However, I always claimed that terminate is really intended to be used for error scenarios, and not to situations where the process simply reached a point where it should stop processing; I have event, in a previous post, suggested introducing a new "end" shape; in that post I have mentioned a possible alternative - using the Terminate shape - which was also suggested in a comment by an anonymous, but as I believed that the shape is really meant to be used for error scenarios I felt this was not ideal, but at the time this was more a gut feeling thing then anything else.

Today I noticed for the first time the label used for the termination "reason" - it is called "error info", which to me is the "proof" I needed (at least to satisfy myself) that the terminate shape was indeed intended to be used for error scenarios; same error info appears when you view the message flow in HAT of a process that has been terminated , at the top section with all the general details about the process you will find an 'error info' label; any text provided for the terminate shape will be shown there.

I've spotted this only because, being the hard headed guy that I am, I have a few orchestration that have a terminate shape as the last shape in the process. "what is the point in that??" I hear you ask...well - this is why I looked at my decision again.

Well - there are a few possible scenarios - here's one - we have a process that is exposed as a web service. the process initiates several sub-processes (using a mixture of call and start orchestration) and then returns to the caller with a response.

If we have errors in the process we keep track of them in a helper object and return them as a soap header (with or without a response) so that the client is aware of them.

using the terminate shape at the bottom of the orchestration (if my errors collection is not empty) I can report the errors to HAT and make them visible to our operators as well; instead of the orchestration showing just showing as completed, they get a hint through the status that something did not go smoothly and by inspecting the error info field they can find out what it was.

yes - I know I can use the event log - but this way it gets logged with the process in HAT which we can easily find.

...and then just when you actually needed HAT..

I'm pretty sure I'm not alone suggesting that the HAT tool is somewhat, let's say, lacking....

There's quite a few annoying things about the tool, but there's one thing in particular that has to be at the top of the list, because, in my view, it means that in the one case that you really need HAT to help you out, it fails you miserably.

The "orchestration debugger" is a nice selling point for BizTalk: you develop your process and, assuming you have the relevant tracking settings turned on you can go back to processes already completed and "replay" them to see which shapes have been executed and which haven't.

This is really great when viewing processes already completed, and also somewhat useful when setting  a breakpoint in the process and attaching to the process in HAT (although not as useful as one might think).

However - it is completely useless when dealing with suspended orchestrations.

If your orchestration get's suspended for whatever reason, you get a nice error message in the event log, in most likelihood the event log message will even contain the name of the shape in your orchestration in which the exception occurred; however - find the suspended instance in the admin console or in HAT, open the orchestration debugger - and you're into a surprise: the viewer will only show you execution up to a few shapes BEFORE the actual shape that failed.

I'm not sure I have the story right, but I believe this "bug" (it's really a "side effect", more on this in a second) was introduced in 2006 (but I no longer have 2004 installed to prove it was actually better before hand).

As far as I know, one of the changes in BizTalk 2006 is around the way orchestrations handle exceptions - in BizTalk 2004 all unhandled exceptions in an orchestration would (if my memory serves me right) result in a suspended non-resumable instance; in 2006 these instances are resumable; this suggests that BizTalk 2006 has to keep the state of the orchestration BEFORE the error occurred - so that if an administrator chooses to resume the process (possibly after fixing whatever caused the suspension) the process could start again and retry the action where it got suspended before).

In order to achieve this BizTalk probably keeps the last GOOD state of the orchestration in the database (from the last persistence point executed); in other words - where before a suspension would cause a persistence point, from 2006 it does not and the orchestration's last persistence point is what's kept in the message box.

If that is correct it would explain why the orchestration debugger only shows information up to a point before the shape that caused the suspension - it would only have information up to the last persistence point.

I don't know if this was an oversight when releasing 2006 or a conscious sacrifice, but I think it's a big pain point; it would have been great to see it all - see where the exception occurred, see the state of the orchestration at this point as well as having an indication as to where was the last persistence point - so we could tell what will get executed when we resume the orchestration.

There are quite a few good uses for HAT - it's a great tool to know what's been executed on the server over time; it's not a bad tool to take a look at the duration it took for a service to run, it's even somewhat useful to check the flow of a particular message through the engine using the message flow or the orchestration debugger view - but when it comes to helping out finding out a cause for a suspended orchestration - it is quite pointless for that reason.

So if you counted on it bailing you out when your process fails - you may as well switch off orchestration shape tracking for your processes.

Why the New Configured Port Wizard is confusing

It is generally known, I believe, that BizTalk has somewhat of a steep learning curve.

BizTalk server is by no means a simple product, but that's ok because we tend to do complex stuff with it :-)

Microsoft, over the years, from version to version, invested a lot in making the server easier to learn and use, some improvements were made in the UI, but mostly through investments in the documentation, tutorials, examples and "community content"; I do believe this has made BizTalk much more accessible.

There are still quite a few things in BizTalk which, from my experience, tend to confuse new starters; one of them is the concept of orchestration ports and port type and the "New Configured Port Wizard".

I find that too many developers use BizTalk without fully understanding the fact that it is a strongly typed system and without understanding the relationship between ports and port types (and messages and [multipart-]message-types).

This is not helped by the fact that you can quite easily develop an orchestration without explicitly defining either.

The new configured port wizard defaults to creating a new port type whenever you configure a port; I suspect many developers never give it a second though and simply create a type for each port they use; this way you can easily create quite a few copies of the same type and not even recognize it.

further more the wizard creates the port and the port type, but only completes the port type definition when you connect the port to a receive/send shape that has a message configured (the message-type portion); I believe this further confuses people as 1) it does not make it clear that the message type is indeed part of the port type definition (along-side the access modifier and the message exchange pattern) and 2) as the port type definition does get completed when you create the link, many developers do not understand why they cannot replace the message in the receive/send shape, or connect another receive/send shape to the recently created port (because the message types may not match).

If I had to guess I would say that the reason for the way the wizard works is the assumption that it lowers the entry barrier to developing BizTalk processes as people can develop processes without understanding the concept of types) but, in my view, here lies the problem - developers produce code they do not fully understand with all the problem that creates.

2 BizTalk Groups, 1 SQL Server

A couple of months ago we needed to deploy our existing solution to a new BizTalk group to serve a different client.

For various reasons we have decided to dedicate a BizTalk server for this client, so we don't have to worry about the impact of deploying changes to our existing clients etc, but to share the SQL server.

Obviously this is not the most ideal setup, but as the new client is in a completely different time zone to the existing ones, and both environment are not (yet) expected to have high volume of traffic, we could be quite confident that the SQL server can support both environments (one during the day, the other at night)

Anyway - BizTalk is quite happy to support that and you can easily configure it to use different databases on the same DB server using the configuration wizard and it all works quite well. nothing to report.

That is - until we wanted to deploy our BAM activities on the server.

For the sake of this discussion let's assume that when we've configured the first BizTalk group we left the default 'BAMPrimaryImport' database name for the BAM main database, and that for the second group we use 'BAMPrimaryImport2'.

Using bm.exe deploying BAM activities is usually a no-brainer and the tool takes all the hassle of creating tables, relationships, views, indexes etc as well as registering them all in the BAM repository.

However, the tool also generates SSIS job for each activity for purge-and-archive purposes; these jobs are simply named based on the activity name they support and are then deployed to the SSIS server (which is not partitioned by an entity such as database as far as I know), and this is where we faced a problem:

As the first group already had our BAM Activities deployed, the corresponding SSIS jobs were already deployed to the server with the generated names; when we came to deploy the activities for the second group, bm.exe went on to try and generate the SSIS packages using exactly the same (generated) names and failed saying such packages already exist.

As far as I know there is no way to control the names of these packages bm.exe would use and so we were a bit stuck.

Fortunately - changing the name of the existing jobs was fairly easy in the SQL management studio, and as they are not referred to by anything (other than a schedule to run in SQL) was fairly safe and so - what we did was to rename the packages created by the first group, so that bm.exe would create the packages required for the second group with the old name without failing.

Orchestration Statuses

I was surprised to find out there's some confusion around the possible statuses a deployed orchestration can be in, but after spending 20 minutes browsing MSDN I realised I simply couldn't find one clear description of those, so here's my attempt -

An orchestration deployed into BizTalk server can be in one of four states -

1. Unenlisted (unbound) – the process has been deployed to the server, but is unconfigured (host and/or port bindings are not set), unsubscribed and is not running.
2. Unenlisted (bound) - the process is configured, but is still unsubscribed and is not running.
3. Enlisted (stopped) – the process is fully configured, subscriptions have been created, but it is stopped.
4. Started – process is ready to run (and will do so when activated by a message)

The first two are obvious I should think, so is probably the last state, the Enlisted state however, may cause some confusion - when enlisted but stopped the orchestration's subscription is active on the message box meaning it will get evaluated for every message published. if the evaluation succeeds, the message will get queued for this orchestration but it would not start (as it is in stopped state). the messages will remain waiting until the orchestration is started or they are terminated.

Extracting values from a message in the pipeline

Another question I was ask recently is how to extract a fields value from a message in a pipeline.

One example I’ve seen goes something like this –

   XmlTextReader xtr = new XmlTextReader("books.xml");
   XPathCollection xc = new XPathCollection();
   int onloanQuery = xc.Add("/books/book[@on-loan]");
   XPathReader xpr  = new XPathReader(xtr, xc);

Then, in order to get the value the stream needs to be read -

while (xpr.ReadUntilMatch())
{        
   Console.Write("{0} was loaned ", xpr.GetAttribute("on-loan"));
}

you can, of course, replace the Console.Write with any action required on the data located, you should also have a check to see exactly which xpath was hit (if you have more then one in the collection)

The problem with this is, as most of you may well know, that it requires that the component reads the entire stream in it's execution.

This, actually, has two disadvantages – one is performance - assuming this is a receive pipeline BizTalk will have to read the stream anyway to write the message to the message box (ina send port the send adapter will do the same). If we could avoid the need to read the stream ourselves, and simply event on the stream as BizTalk’s internals read it we would significantly improve the performance of our pipeline.

Secondly – since we’ve read the stream, we may have a problem now when we go back to return the stream to the pipeline; some streams we might receive are not seekable (such as anything coming from the HTTP or SOAP adapters) and so we can’t simply rewind them and we surely can’t return a stream pointing at the end of the message to the pipeline. It is enough to read Charles Young’s great series of articles about receive pipelines in BizTalk 2004 (http://geekswithblogs.net/cyoung/articles/12132.aspx) to see what sort of issue you might face.

Although some of these issues have since been address the underlying problem remains and that is that by reading the stream we have to then return a “touched” stream to the pipeline which may or may not cause issues, and as we can’t always be sure in what context our component will be used (can you assume send vs. Receive? Can you assume a particular adapter will be used, can you assume port maps will not be used?) we should look for a better way to do this. Luckily such a way exists that helps in most circumstances – BizTalk’s XPathMutatorStream.

Luckily for me Martijn Hoogendoorn already wrote about it a couple of years ago, check his blog entry here I just thought I’d point this out.
Obviously this stream was designed to allow replacement of values, but nothing prevents you from setting the output parameter to the input parameter and thus avoid any changes.

As you can see in the post using this stream means you never actually need to read the message’s stream yourself, you simply need to wrap the original stream with an instance of the xpath mutator stream, add the xpaths you want to the collection and return the wrapped stream with the message. When BizTalk will read the message’s stream it will now be reading your stream which would raise the appropriate event when your xpaths are being hit. Marvellous!

From a performance point of view I have not looked at the implementation of the stream so I can’t say for sure it is faster than reading the stream completely in your pipeline, but my gut feeling says it is, but definitely from robustness perspective this solution is much better as it eliminates all the problems one might encounter by reading a message’s stream in the pipeline.

When is this solution not good – when you need an entire XmlNode. This stream would be able to return a single value (element or attribute I believe), but if you need the entire contents of an XmlNode (with child elements or various attributes) it would not server you well.

Creating a message "from scratch"

A question I get asked repeatedly is how to create a message in an orchestration “from scratch”, i.e. – when the message is not meant to be created from any other message.

Initially one might think this does not make sense, but I find myself doing it quite often actually; two typical scenarios are when one has to create a message before branching the process to satisfy the compiler (and avoid the “use of unconstructed message” error), a second is when one needs to return a message that simply contains few values obtained from, say, a database call, or some calculation etc.

In this case often it is simpler to create the shell of the message with the elements/attributes required and then use xlang’s xpath function to push the values into the right places.

Whatever the scenario is, there are, I believe, two options to create empty messages (corresponding to the two options to create messages in orchestrations in general, actually) –

1. Using a map
2. Using a message assignment shape (and a helper method).

The first one is quite obvious – you create a map, pick any message you may have in the process (you are likely to have at least one message in your process already) - use that as your input message and the message you want to create as your output message; then you’re facing two alternatives – the first one is to use xsl – you can create an xsl file that effectively has the XML you want to create hard coded in it and instruct the map to use that xsl.

The input message is completely ignored (and so is completely irrelevant) and the output message is always the way you want it to look like. You are not sensitive to any changes in the schema of your input message.

The other alternative, which I like less, is to actually use the mapper; in this alternative you would probably map the root node of your input message to the root node of your output message and then set up default values in the output message for any element/attribute that you wish to include in your output message.

The reason I believe this is not as good as the first alternative is that it is less obvious how the output message would look like; one has to follow up on the nodes to see what is set (or test the map) to see the output while in the xsl alternative one look at the xsl is usually enough to show what the output is going to look like.

Either alternative you choose – I used to think that using the mapper is a better option (as opposed to using the message assignment shape which I will describe shortly) – mostly because I thought this is a more standard way to create messages, and so it is more obvious, looking at the process and the project, where such constructions take place but mostly,I believed, if you’re using xsl files to create the output it would be very easy to spot, read and change them in the solution when necessary (simply find the xsl files and change them, no need to look for anything else).

Thanks to several people, but mostly to Ben Gimblett with whom I work with in one of the projects I'm involved with and who has insisted not to follow my advice and use helper methods to create messages, I now agree that using helper classes is the better way, mostly because using them you don’t have to use a dummy input message (as you do in the map) – which, I have to agree, can get quite confusing to anyone trying to understand the process but also because, I suspect (but have not tried to prove), a helper class will perform better than the mapper option.

When using a helper class you again have to alternatives –

You could have a message assignment shape in which you call a method that returns a .net type (class) that has been generated (using xsd.exe) from your schema.
All you need to do in your method now is create an instance of that generated return type – populate whatever members you require and return it.

In the assignment shape you assign the return value from the method to your message[part] and so BizTalk will take care of the serialization and will convert the class to the schema and because the class and the schema both represent exactly the same thing the serialisation would work just fine.

The benefit of this approach over the mapper option is mostly that there's no need for any dummy input message, no need to write xml or xsl; only very simple (and quite minimalist) code is required.

The downside – you need to generate those .net classes to represent any schema you wish to return, and maintain them as your schemas evolve.

The second alternative is simpler on the one hand as it does not involve generating and maintaining classes; it does, however, require a bit more wiring –

It starts the same way – a message assignment calls a method whose return value is assigned to the constructed message[type].

The difference is that the method does not return a strong type; instead it returns an XmlDocument whose contents are loaded from compiled resource within the assembly.

The function takes in the name of the xml that needs to be used to create the constructed message, retrieves the resource from the resources in the assembly, loads it into an xml document and returns it to the caller.

I find that this last approach works best for me in most cases – all the generated xmls are in one place which makes them easy to maintain (which I liked in the xsl option), there’s very little co-ordination that’s required – only the name of the xml file (or any other key one wishes to use) must be known to the caller and the xml resource should match the schema – but as it is stored in one location AS XML this is very easy to achieve and maintain.

Wish list: "End" shape.

Often orchestrations have conditions that, when met, mean the orchestration should stop executing.

In .net code a developer could simply put a “return” keyword in the function to stop the execution and return to the caller; unfortunately there is no equivalent in orchestration development.

The only “clean” way to end an orchestration currently is to reach the built-in, fixed, red shape at the bottom of the orchestration designer.
This often means that decision shapes exist all over the orchestration with branches that have to drop all the way to the bottom. In large orchestrations this can be hard to follow up on.

It could have been made much nicer by allowing a developer to add as many “end” (or “return”) shapes as needed in various places in the process. Decisions, of course, will still exist, but their scope will be much smaller.

An alternative that exists currently is to use the terminate shape, but I find it to be a bit artificial – my view is that termination of processes may be relevant in exceptional scenarios, as part of error handling, etc. but not when the orchestration ends correctly (albeit before reaching the end of the schedule). The main problem with the terminate shape possibly is that the orchestration gets a “terminated” state and not “completed” which will make it difficult to report in HAT (and otherwise)

Casting between messages and classes demystified(?)

I wrote in the past already, as I've argued many times before, that messages should be manipulated using maps and xsls and not using code.

However, as stubborn (and purist?) as I may be I have to admit that some cases just call for the odd manipulation (or creation) of messages as classes in .net helper functions.

And so - it is very fortunate that BizTalk is intelligent enough to allow us to cast one to the other without any fuss.

To those of you who do not know yet it is quite possible to have the following -

A serializable class in any .net language.
A schema that correclty represents the xml representation of that class.
A class (idealy a different one) with a function (usualy static, but doesn't have to be) that accepts a paramter of the class above.
An orchestration that has a message of the schema above.
An expression shape that calls the method in that second class passing the message.

BizTalk will take care of the serialization of the message and will pass an instance of the class to the helper method.

Similarly it is possible to return a class from a function and use that function in an assign shape to create a new message (and, of course, the two approached can be combined as well)

Anyway, we've known that for a while now, and it has worked wonders for us; this technique makes both the functions and the orchestrations that need to use them so much simpler; but recently I nearly went mad when I tried to do just that and kept getting casting errors at build time for - as I thought at the time - no reason at all.

For some reason BizTalk was unwilling to accept that my schema and my class are one.
only after quite a few nerve racking attempts did I find out what I did wrong -

All of our classes have serialization attributes to control the xml generated; mostly we're making sure we use attributes whenever possilbe (as opposed to the default element serialization) and that the attribute/element names are short-ish.

In the particular class I used we had an XmlRootAttribute that set a (different, shorter) name to the root element representing the class; as soon as I removed the name from the attribute (it is optional) VS was happy to compile my orchestration.

It appears that BizTalk needs the root element of the class to have the name of the class; no shortcuts there.

By the way - a few months ago I've published this article suggesting that passing XlangParts as paremeters to helper functions may cause a memory leak.

As this would have prevented us from using this wonderfull casting technique we've asked Microsoft to kindly confirm if indeed my understanding at the time was correct and that one should not pass an XlangPart as a parameter to a function.

As I understand it now, and although it is not very clear from the KB article I've referred to in my post, passing XlangParts is ok as long as the lifetime of the message/part is not shorter than the the lifetime of the parameter - in other words - as long as you don't keep the message or message part in memory in your helper class you should be safe.

Wish list: searching for expression in subscriptions

And while I'm on the subject of subscriptions (see my last post) -

In the BizTalk Administration Console you can run a query to view subscriptions (which is a huge step forward from the old subscription viewer for BizTalk 2004, I have to admit)

You can, however, only filter based on the subscription type (activation or correlation), the service name or the service ID.

What that means is that while you can find all the subscriptions that start a particual orchestration, you can't find all the orchestrations that would be started by a message arriving through a particual receive port, which would have been quite useful, don't you think?

What's even worse though, is that even if you were happy to do a lot of manual work, there doesn't seem to be any practical way of finding out this information using the admin console, although it is all there in the database -

When you bind an orchestration recieve shape to a receive port, the subscription is is created for you using the receive port's id rather than the port's name, so it would look something like -

http://schemas.microsoft.com/BizTalk/2003/system-properties.ReceivePortID == {E1E7FE08-D421-4D5A-8CD8-CA51E25FA508}

This is not very readable, but, unfortunately there is a much greater problem with it -

There doesn't seem to be a way to know a receive port's id through the admin console; so even if you did have the patience required to go through all your receive ports, matching their GUIDs to the one in the subscription hoping to figure out this way which ones will get your orchestration activated, you would find it impossible to find out which receive port actually has the id -'{E1E7FE08-D421-4D5A-8CD8-CA51E25FA508}'

The only way to find this out is to go to the management database and lookit up yourself in the bts_receiveport table, baring in mind, of course, that this id will change on the next deploy.

BizTalk's Pub/Sub

The publish/subscribe mechanism in BizTalk is one of the key features of the product and is very useful and powerful.

I guess there's some learning curve around it, and that most first implementations of any BizTalk developer do not make much use of it (as they often start with all ports being directly bound) and that it takes a while befor a BizTalk developer and the organization involved establish a good architecture and move more towards using true publish-and-subscribe, losely-coupled, approach to implementation.

However, the existing model is not perfrect; in my view (and I suspect it is shared by many) it has two main weak points -

The pub/sub is implemented on top of MS-SQL which introduces a significant performance overhead

The orchestration subscriptions are 'compiled' and cannot be configured withouth a build-and-deploy cycle

The first point is quite an obvious one - there would be a latency associated with any implementation of publish/ubscribribe mechanism;. in the BizTalk case it involved writing the message and it's meta data (context) to the message box (a SQL database) and having a separate process locate newly published messages, figuring out which subscribers need to receive a copy of the message and manage the activation/correlationthe of message-process interaction (as well as keeping a list of references for house keeping etc).

Reading and writing to the database, the the polling interval of the subscription evaluation process, etc. all introduce latency, which, in certain scenarios, can be crucial.

If to believe the fractions of information floating around regarding Oslo then we might see an in-memory pub/sub mechanism in future version of BizTalk (in addition, not as a replacment to the existing model I suspect) which, while will no-doubt come with a price (persistance, and therefore scalabiltiy and durability to some extent), will no-doubt make supporting low-latency scenarios much easier.

As for the second point -

At first look the pub/sub in BizTalk is very flexible; in all the BizTalk demonstrations I can remember from the past the presenter would create a recieve port and a couple of send ports and will edit the subscriptions of those ports in the administration console to show how easy it is to create content-based routing in BizTalk server and configure it at runtime.

In BizTalk 2006 you even did not have to restart the host to speed things up (as you did in demos with 2004), it happens pretty much instantly.

However, the case with orchestrations is not that simple...

The subscription for orchestrations is specified as a a filter in the properties of the initalizing receive shape in the process; this gets compiled into your assembly together with the process, and will be used to create the subscription when you deploy the orchestration.

As far as I know, short of manipulating the management database yourself (which would not be supported) there's no way to change those subscription at runtime.

If you want to change the subscription you have to change the filter in the orchestration, build, undeploy the old version and deploy the new one (or version the process and perform the side-by-side deployment)

This is, in my view, an un-necessary pain, in dynamic organizations (aren't they all?) that require changes often; and to that extend developers had to find a solution to the "I need to be able to change that subscription from outside the process" requirement.

That solution is often adding some routing metadata to messages in the form of context properties ('nextProcess', 'Operation', etc.) which would be set by publishing processes and/or pipeline components and use these in the filters (rather than the actual content data).

So you could often see a pipeline component, often driven by some external configuraion, that would check for certain bits in the message or it's meta-data and set these context properties based on the values it found; the premise is that pipeline components are easier to replace, but also - thesee components often use database or a rules engine in one form or another to decide what goes into the message context and by doing so introducing real flexibility as is advertised.

What all of this means is that we, developers, end up developing a pub/sub mechanism on top of the existing pub/sub simply because we need flexibility the product does not provide.

I don't like this apprach, but I end up doing this myself occasionally, simply because I have to.

I could possibly understand why MS has decided to do so - there are benefits to editing the subscription expression within the orchestration (known types would be one thing), and also - one could argue that the process subscription is part of the process design and so changing it is likely to involve code changes as well which will require a re-build, but really - I think we would all have benefited from the ability to edit the orchestration subscription in the same way we can edit send port subscriptions - through the admin console.

The message box as a service boundary

For the last 18 months or so I've been working on a very exciting, and quite large, BizTalk implementation here in the UK, I'll leave the full details of it for now, but I can tell you that it involves all the nice buzzwords we keep hearing about SOA, SaaS, S+S, ESB at least to some degree (and with various level of quality, if we're honest)

Anyway, as you can imagine we're using web services quite extensively - we expose a lot of them, and consume even more; some are internal to the company (but cross teams, although not so much platforms) and many are external (which do cross platform as well)

The reasons to use service oriented architecture should be very clear to everyone by now, as are the famous four tenants of SOA.

In out implementation we've abstracted the calls to all the internal web services through utility orchestrations which would take a message in our canonical format , convert it to the service's format, call the web service and transform the response to the canonical format before returning it to the calling process; this way we can re-use those transformations, and have a central place to deal with each request, apply error handling, etc.

From the parent process we then use call orchestration to initiate these utility orchestrations passing in the request cannonical message and receiving the response canonical message as an out param, which is quite efficient (when initiating orchestrations through the call orchestration shape the request does not go through the message box)

As we're doing the transformations in these utility processes, we consider them to be in the boundary of our process, and not, obviously, within the boundary of the called service, for this reason we call the web service from the process rather than the actual assembly behind the web service.

When we, within the utility process call the web service, what actually happens is that the request message (now in the WS' format) gets published to the message box, being picked up by the send port which would pass it to the SOAP adpater which, in turn will serialises it and transmits it over the wire to the service; the service then deseralises the message on the other end before executing whatever code needs to be executed and the entire process now repeats in the opposite direction.

In this case the service boundary is the web service endpoint.

A few weeks ago I had what I thought was a brilliant idea - why not treat the message box as the service boundary!?

If I had a process that takes in the service's format of the message using a directly bound receive shpae and a filter, execute the code internally (as we're now inside the service boundary we can use the service code directly from expression shape, no need to go through a web service) and when finished publish the response back to the message box (in it's own format), I could have simply published a request message for that service, and get the response published back for me; correlation should be used, but this can be handled using self-correlating ports or a correlation set.

The client process would do pretty much the same - it would use a utility process to transform the canonical format to the service's format and publish the request. it would then use correlation to receive the response and transform it back to the canonical format before retunrning it to the calling process(synchronously).

What would we save? - following this approach for at least some of our internal services can save us the need to serialise the messages over the network; in the web service case we have to go through the message box from the process to the send port anyway, so going through the message box from one process to aonther would not make a difference, but all the network traffic and the work by the SOAP adapter (which is far from being efficient) can be saved.

This was a good idea (I thought anyway), but I suspect it won't work, as it has two main flaws (and I will be extremely happy to get some ideas around those) -

Firsly - both subsystems will need to exist on the same BizTalk group so that they share the same message box and so we could use pub/sub to exchange messages between them (on it's own this is not necessarily a problem, but it is the main cause for the next one, which is the big one)

Secondly - the schemas will have to be shared -

When you're adding a web reference to a web service from a standard .net project a proxy gets generated for you; that proxy will include a local version of all the classes used by the web service (these will be in YOUR code namespace rather then ther service's but will serialize to the same XML).

Equally - when you add a web reference in a BizTalk project, you get schemas generated so you can create messages to send and receive to/from the web service; these will be in the service's XML namespace as they have to represent the XML supported by it, and here lies the problem.

If both the service implementation and the client implementation are on the same BizTalk group, the schemas will have to be shared as there's no way to deploy two schemas using the same root node and namespace and we all know that sharing schemas is a bad idea as it strongly couples the implementation together and that pretty much renders the idea useless (this, confusingly I suspect, means we're sharing a class and not a contract).

Of course one could play around with the idea of having two BizTalk groups and communicating between them, and although you can choose better transports than SOAP for that internal communication I suspects that brings us closer to simply calling the web service and so I'd rather stay with that standrad approach.

Mapper vs. XSLT round 2

I've received a good question today -

"we had a little debate in the office today - what is faster - running a map with pure xsl or the standard way with functoids, what you think?"

As I've blogged before - I'm a big supporter of writing custom XSL and not using the Mapper and Functoids in anything other than the simplest of maps; so - although performance is only one of my arguments - the answer should be obvious.

Nevertheless I'll take the chance to answer properly again, although I suspect the question is not accurate enough -

At runtime there's no difference between the two; the Mapper generates XSL (which you can see by "validating" the map in visual studio and following the link to the XSL file generated which would appear in the output window, so the question should be, in my view, whether the Mapper can generate as good XSL as a developer could, but as you can imagine the answer really depends on a particular scenario - how many functoids are you using? how are they working together? what's the size of the map? what's its complexity?

Anyway, in my view there is a bottom line answer to that question and that is that under most real-world scenarios custom written XSL will almost always be better than generated one, but I'll try to explain a little bit more -

When you're using Functoids in your map you're generally doing one of two things - you're either calling external assemblies or you're adding some XSL lines to perform some actions for you.

The former one is easier to tackle - if you need external assemblies you can call them from custom XSL as well (as I've explained here ); as the Mapper will do exactly the same, the performance impact will generally be identical in both cases (using mapper or custom XSL).

The latter is harder to tackle, as there's no one-rule-fits-all statement one can make - but here's a shot at it -

The Mapper is a visual, generic, designer that generates code.
As is always the case with these tools they come with a price, and that price is often the quality of the code generated; now - don't get me wrong - I don't argue that the Mapper is bad, or that it always generates bad, slow XSL; but if you know XSL well, there's no doubt you will write better code than a generator will.

When you're adding a Functoid that does not call an external assembly you'll be doing one of three things -

You will be adding an embedded c# code - most Functoids do this, look at the string manipulations as a simple example.

You will be adding a template based on input nodes - the Looping Functoid for example.

Or - You will be adding XLS structures or functions - the record count or value mapping Functoids for example

All three are perfectly fine, and even more so - if you'll try them out you'll see that the designer does generate quite a nice XLS in all cases.

The problem starts when, and this is inevitable in the real-world, the maps get more complex.

Once you move out of the playing ground and into real scenarios, the maps get more complicated and the inefficiency of the generated code becomes both more apparent (as multiple Functoids need to work together to achieve the desired output the XSL gets 'uglier and uglier') and that inefficiency becomes a greater problem as it is repeated many times over a large-ish map.

Bottom line is from my perspective - if you feel comfortable with XSL (and the rest of the team) - you will always achieve better scripts than any generator would so use it. If you don't feel comfortable with XSL - learn it! It's easy! (and in the mean time use the mapper).

Exception, Orchestration, Serialisation.

I was adding a custom exception yesterday to a helper class I’m calling from an orchestration.

Usually, my exception handlers in the orchestration are quite short; this time, however, I wanted to do a bit more, which included calling a web service when a particular expcetion is caught.

While implementing this I learnt something interesting (which, arguably, I should have known a long time ago – just to show how difficult it is to catch up on all the changes in the .net framework) -

In.net framework 2.0 a Data property of type IDictionary was added to the Exception class, which by it's own is not a problem, only that IDictionary is not serialisable and so could have proved rather difficult to anyone using Exception, especially in a BizTlak environment.

Luckily (but not surprisingly) the .net framework team have implemented ISerializable in the Exception class, which helps, but does cause a small headake to the unexpecting BizTalk developer (me).

But first - I have to apologise - again I'm not familiar with all the details around this, and am resorting to pure guesses of a couple of points (will be happy to get more information if you care to enlighten me), still - I'm sure this will be useful to most people...

When you mark a class as [Serializable], as the runtime deserialises a class it attempts to call a parameterless constructor to create an instance of the type; the serialiser will then populates all the members of the class through their public properties (I suspect that this is, partly at least, why Xml Serialisation serialises public members only).

When working with ISerializable, however, the runtime expects a constructor that takes SerializationInfo and StreamingContext as parameters; it is expected that the constructor will populate the members out of the SerializationInfo collection.

I believe that the runtime interrogates the type to be deserialised and, once it finds that the type or any type in its inheritance path implements ISerializable it takes the second approach mentioned.

Not realising the Exception class implements ISerializable ,I did not have the expected constructor in my class, which meant that when BizTalk tried to deserialise the object (between the send shape calling the web service and the receive shape expecting the response) it failed, which now exaplains the error reported in the event log -

The constructor to deserialize an object of type ‘[custom exception class name here]’ was not found.

Adding the constructor with the two parameters to my custom exception class allowed it to pass the deserialisation with no errors; however – I was now facing a second problem – after indicating that my class implements ISerializable and addin the constructor required the members of the Exception class, from which my class inherited, including the Data member now deserialised correctly; my own class' member,however, did not.

There are two ways to overcome this - I could have simply mapped my properties (after all I only had a couple of strings to keep with the exception) to the Exception's Data property (have the getter and setter of each property use the collection internally, and so all the data will be capture in the Exception base class and so serialised with it, or - I could implement ISerializable fully which really only means

1. Firstly - adding my members to the serializationInfo member of GetObjectData:

        public override void GetObjectData(SerializationInfo si, StreamingContext
        context)
        {
            base.GetObjectData(si, context);
            si.AddValue("member1Name", member1Value);
            si.AddValue("member2Name", member2Value);
        }

2. Secondly - populating the members back in the constructor:

       protected MyCustomException(SerializationInfo info, StreamingContext context)
            : base(info, context)
        {
            member1Name= info.GetString("member1Value");
            member2Name= info.GetString("member2Value");
        }

Voila! it all serialises and deserialises ok now. if only I didn't have to spend a whole day to figure this out!

Wiki is coming to MSDN!

This has been talked about for a while now but I've only seens it in action now -
MSDN now supports wiki at parts - and most importantly for anyone reading this blog (I suspect) - in the BizTalk 2006 R2 documentation.

Check out this page and see how Eric Stott has kindly pointed out a huge improvement that was mentioned almost as a by the way statement in the docs. way to go Eric!

So here ya go, now there's an easy way to share thoughts, ideas, additions and corrections to the msdn content. brilliant!