BizTalk Admin Console

Anything you missing there? anything you don’t like?

I’m trying to compile a list of suggestions for it, put a comment or drop me a line….

More on using the SOAP adapter without a web reference

Every now and then I am drawn to implement a call to a web service using the SOAP adapter and without adding a web reference.

I wrote about it in the past here, as did others, for example consider Richard Seroter, Saravana Kumar and Jon Fancy’s articles on the subject.

Last time I had to do this, though, a couple of months ago, I bumped into a puzzling error, which took some time to understand.

In the scenario I had I needed to modify the message in the pipeline, and - being the good citizen that I am (?) - my pipeline component created a new message and did not return the same message it had received.

When my message hit the adapter it got suspended with the error -

Failed to load "" type.

Please verify the fully-qualified type name is valid.

Details: "".

The type must derive from System.Web.Services.Protocols.SoapHttpClientProtocol.

The type must have the attribute System.Web.Services.WebServiceBindingAttribute.

In my scenario the proxy was provided for the SOAP adapter via its configuration in the send port explicitly as per Richard’s post, so I was very confused about the empty type definitions in the error message.

Some investigation revealed the cause – the send port configuration is read at the beginning of the pipeline and these settings are set in the outgoing message’s context.

My pipeline component created a new message and did not copy these properties to the new message, so when it arrived at the adapter, which tries to read them, the empty definitions were found.

On MSIs and Default Bindings

Some observations (I’ll you to draw your own conclusions, for a change) -

• When you import an MSI, any default binding in it will always be imported, then any binding matching the environment selected in the import MSI wizard (if selected)
• If the second binding contains any settings that were already created as part of the first binding, they will be overwritten
• When you export an MSI, a binding file is created with the existing configuration as the default binding, even if you did not add one as a resource explicitly. This binding will be imported first, before any other binding.
• In the case of receive locations – if two bindings in the MSI contain receive locations for the same port each marked as primary, the last one imported will be the primary receive location.

Why am I writing this?

We had an odd case (which shouldn’t really happen) where we created an application with a particular receive location.
We have then added a binding file for another environment in which we have misspelled the receive location name (the URI was the same)

When we tried to import the MSI to the second environment, selecting the environment name in the import MSI wizard we received an error that looked something like this -

(for the search engines: “Cannot update receive location”, “Address should be unique for Receive Location”)

The reason was that the wizard imported the MSI and initially applied the default binding creating the receive location with the first name. It then went on to import the second, environment specific, binding file, now trying to add a new receive location with the different name but failing as the URIs are the same. had the name been the same the receive location would have been overwritten, had the URI been different a second one would have been successfully added, but given this weird combination the error was produced.

What I’ve learnt about BizTalk Hosts

Here’s something I learnt from Guru Venkataraman on my one of my visits to Redmond last year -

Like most seasoned BizTalk developers, I suspect, I usually follow the best practice around host planning nicely summarised by Marcel here.

What I did not think about (and nor did Marcel, it appears) is the actual queues behind these hosts, how they are used, and the impact on performance; in comes Guru’s wisdom-

Each host has a queue, implemented as a table in the message box database, so - loosely speaking – when a message is published to the message box, BizTalk determines which subscribers are interested in the message, and places the message (well – logically) in the queues of the relevant hosts.

Each hosts polls its own queue periodically (default is 500ms) to get any pending messages and acts on them – start a new instance or correlate to an existing instance, potentially rehydrating it.

Now imagine you have one host running two different processes and that two messages are received both of which should trigger a new instance of process A.

The host picks up the first message (for this discussion to be clear enough we’ll have to pretend it’s working on a single thread, although I suspect several aspect of this are) and passes it to a new instance of process A.

Imagine now that process A publishes a message intended for process B; as process B runs under the same host, the message gets into the very same queue in which we already have the second message queued for process A.

Now – process B cannot start until the host reads the pending message and starts the second instance of process A. it is only at this point that the message intended for process B is at the ‘front’ of the queue and can be read by the host and an instance of process B can be started. effectively the execution of process B has been delayed unnecessarily because there were messages queued for process A.

Now – of course BizTalk much of this is indeed multi threaded and so the problem is not that severe. BizTalk was clearly designed to host many processes under any single host and there’s definitely not a requirement (or a recommendation) to have a host per-process, that would be ridiculous, however -

Where you have a flow where process A often publishes messages that end up in another process (B, C or D to make some names), and you have a reasonable throughput, you would get better performance if the publisher of the messages (process A) is hosted separately from potential subscribers (B, C or D).

With the subscribers configured with a different host they can pick up their messages without being affected by other messages awaiting for process A, and vice versa – the same goes for any replies published by them.

Two things worth remembering in this context:

1. When you ‘call’ an orchestration, as opposed to ‘start’-ing an orchestration, the host of the calling orchestration is used, regardless of what’s configured in the admin console
2. Whilst I’ve specifically mentioned processes, the same considerations apply to send ports, and even receive locations, and when looking at the former it is worth remembering that dynamic send ports, where used, always use the default host for the adapter used.

What not to do – hard coded values

This always seemed like an obvious one to me, but as I do see it out there occasionally, I guess it should be mentioned in my little ‘series’ – think carefully whenever you are hard coding any data into your code, as it is almost guaranteed to change one day, however remote you think that chance is.

This post could probably end here, but that would be almost rude, so I’ll give an example -

A custom disassembler is being developed to identify a message type before parsing it.

The messages are expected to be received over POP3, and the decision which message it is is done by identifying the sender of the email (rather than by looking in the actual message), which is fair enough.

So – the component has code to read the context property storing the sender’s email address and then a switch statement on the property’s value determines the message type required.

Of course this works very well. until a third party decides to change the email address, or another one is added. now you have to update the code, re-deploy and re-test. why? because you did not want to spend another day moving this to configuration?

Even a simple AppSetting would be better than hard coded, but of course usually you’d be looking at least at defining a custom configuration section if not using a database or other store (SSO?)

..and it looks like it’s not

Back in December I’ve posted ‘This should not be a routing failure’ about an experience we’ve had on our BizTalk 2006 environment that I found very weird.

Tim Wieman, a senior program manager on the BizTalk Customer Advisory Team, was kind enough to drop me an email letting me know that this behaviour was indeed identified as incorrect, and was actually fixed.

A hotifx was created for BizTalk 2006 which later versions already incorporate, so if you’re using 2006, and you’re having this issue - get that hotfix, if you’re using a later version – you probably don’t need to worry about it.

I have managed to test the hotfix on BizTalk 2006 and it does allow my scenario to run without a problem (don’t forget the registry change for each host, as described in the link to the hotfix); I haven’t been able to try this on later versions yet, but there’s no reason to think that would not work.

Thanks Tim!

This should not be a routing failure!

Important update: it looks like it’s not after all! and I’ve posted the details here; I suggest you check it out after reading this post.

Some routing failures are straight forward – you have a message, it has context, and no one subscribes to it. Routing Failure Report notifications are very useful on helping troubleshooting these – and most likely someone has mistyped a property’s value, a subscription value or forgot to ensure that the relevant properties are promoted and that relevant services are enlisted.

Some routing failures are a little bit more tricky to map out -

Consider a simple scenario where you have a two way receive port publishing a request, which is being picked up by a two way send port -

Now add a second two-way subscriber to the mix…

In this case BizTalk cannot know which of the two potential responses it may get (from the two different two way send ports) it should deliver to the [send side of the] receive port, and so it opts to not even publish the request and instead deliver a routing failure.

This is fair enough - race conditions are not a good thing and it’s pretty clever that BizTalk identifies this up front and prevents this.

However - the same routing failure will happen, it appears, even if the publisher is NOT a two way port, and that – I did not expect.

If the publisher of the original message (the receive port) was not a two way port, but one-way, it would not expect any response from anybody, which removes the race-condition situation, and therefore should have been supported in my view, but it does not appear that it does.

Imagine a scenario like this -

1st Process publishes message A, through a one way port
1st Send port (2-way) picks up message A and calls a service, publishing the service’s response - Message B.
2nd Send port (also 2-way) picks up message A as well and calls a different service, publishing this service’s response - Message C.

As far as I can tell, this is a valid scenario – call it message enrichment via external services.

Subscribers should be able to subscribe to message A, B or C as they wish, and all is nice and loosely coupled, if you only ignore the fact that this does not seem to work!

Building such a scenario I'm getting a routing failure for message A, which - unless I'm not seeing correctly - I can only explain according to the rule mentioned above, which I find really frustrating.

What not to do – avoid reading entire message’s to memory unnecessarily

This one is fairly common, I suspect, and I can certainly see why – the temptation is simply to big – but too many pipeline components start by reading the message into memory, when, with a little bit more effort this could have been avoided.

One pipeline component I’ve seen, for example, receives a flat file, and needs to remove records already processed (duplication elimination) – quite a good thing to do in a pipeline, and I also liked the approach of doing so before the disassembler, to make the xml produced smaller.

V1 of the component used a memory stream – the incoming stream was read line by line, each line was assessed, and – if was not a duplicate – it would get written to the memory stream.

When the component had finished going through the entire incoming stream, the memory stream would be assigned to the message, replacing the original stream, and the message would get returned to the pipeline.

There are two downsides to this approach – the first is memory consumption – the component will always consume at least as much memory as the size of the (outgoing) message; done properly BizTalk would then clean this memory, but only after completing the processing of the message; the second downside is potentially unnecessary delay in further processing of the message – one of the huge benefits of the pipeline, in my view, is its streaming fashion, where subsequent components, if developed in the correct manner, can start working on parts of the message before preceding components completed their processing; basically each component passes back to the pipeline the portion of the message it already processed, whilst working on the next portion.

It appears that the customer in question encounter memory issues as the component’s code was changed to use virtual stream instead of memory stream; a virtual stream is effectively a stream that uses disk for storage instead of memory.

This solves the memory consumption issue, but merely replaces it with IO operations which may have an even bigger impact on the server’s overall performance (and does not address the processing delay point at all).

What would have been the correct way to implement this in my view?

The component should have create a custom stream, wrapping the original stream from the message; It would then replace the message’s stream with the custom stream immediately returning the message back to the pipeline. Note that so far the component hadn’t touched the message stream – zero bytes have been read.

As BizTalk (and not the component!) would read the message (for instance when persisting it to the message box), the custom stream’s read function would be called which would contain that reads the underlying stream (the original stream received by the component), probably buffering reads until the end of a line for simplicity (although in many cases this is not necessary) and assessing whether the record is a duplicate or not; if it is a duplicate the function will simply read the next line and so on until a non-duplicate record is found, at which point the line would be returned as the output byte stream from the read function.

This effectively means that the next component, or the message box, will receive the message line by line, duplicate records removed, without having to wait for the component to process the entire message, and with only a maximum of one line ever loaded into memory.

LOB Adapter SDK

I’ve been doing some work recently with the LOB adapter SDK.

The adapter I’ve been creating is effectively a wrapper around a web service we’re using extensively, to which we often need to make several consecutive calls.

Doing so from an orchestration, using a send port, is inefficient, as each request involves a pub/sub cycle; on the other hand – I’m not a big fan of ‘inline sends’, which was one of the options considered.

Building a custom adapter would give us the best of both worlds, if you like – the process would create a message containing all the requests to be sent and publish it; a send port, configured with the custom adapter, would pick this message and deliver it to the adapter which would, in turn, debatch the requests, send them to the service, aggregate the responses back and return the aggregated message.

This way we still have all the tracking and management BizTalk provides but only one pub/sub cycle.

The LOB adpater SDK does a very good job in lowering the barrier of creating adapters (I’ve created ‘native’ adapters before) which is a good thing.

Several points I’ve been left with from this, very initial, exercise -

Implementing the metadata browser, resolver and search handler was a bit tedious; I ended up creating an xml file with my metadata, embedded this in my project and wrote code to generate the metadata off that. I wish this was part of the SDK.

Once the metadata was set I needed to write the outbound handler (in my case), for which I needed to know how the incoming messages and outgoing responses should look like; with no generated samples this was a bit too painful to my liking – figuring out the request format is easy although tedious again – I’ve ended up consuming my adapter from a client, making a call, setting a breakpoint and going over the message at runtime.

This is, of course, not possible for the response – where I need to create the message in the adapter, so trial and error was the only way (ok – and reading the docs, and applying some common sense) – in my case I actually made a mistake in my metadata configuration which took me a while to figure out (and Rupert’s help, thanks!)

Bottom line is that if there was a tool to look at the metadata and generate request/response samples for each operation this work would have been much easier. I might be tempted to create one if I get the time.

The last thing that phased me was working with metadata from the client.

I needed the client to provide some information outside the message; I knew the mechanism in BizTalk would be to use a context property, for which I will need to deploy, with the adapter itself, the property schema, but I wasn’t sure if and how the adapter gets access to that or how I could consume this from a .net client.

Figuring out the adapter access was easy enough, turns out all the BizTalk message’s context properties are added as properties to the request WCF message received from the framework, so

token = message.Properties[PROPERTY].ToString();

where PROPERTY is the context property’s namespace#node would return the value of the property requested.

I was a bit stumbled with how to provide this property outside BizTalk, this time Manas Garg came to the rescue in this thread

Turns out you can simply do -

ConcurrentPrograms_ARClient client1 = new ConcurrentPrograms_ARClient(binding, address);

using (new OperationContextScope(client1.InnerChannel))

{


          OperationContext.Current.OutgoingMessageProperties.Add("Property Name", "Property Value");


          client1.OPERATION(params...);


}

What not to do – ‘bundle’ unrelated components together’

So – here’s another real-world they shouldn’t have done this –really! item; somewhat related to my previous post about projects, but a different slant -

MyP had used pipeline components fairly extensively, a very good idea in my view, especially in this case.

But, as discussed in that previous post all pipeline components, regardless of where they were used, were bundled together in a single assembly.

To make matters worse, a single pipeline component often served more than one purpose, for more than one client.

So – for example – a pipeline initially created to remove an unwanted trailer from a message from a particular sender, ended up also converting the message to xml, and then extended to support another format, from another sender, only that the two don’t share any code – the execute method of the pipeline component has a switch statement on the sender name, and runs two separate functions.

Now – considering all the components are in the same assembly already, how can this make matters worse?

Well – single responsibility principle is one that I generally like – I’m a new developer working on this project, I see a component called TrailerRemover, used in a pipeline called <someSender>_Receive I assume this is processing messages from <someSender> and that it removes a trailer.

I eventually discover it does a lot more, and processes messages from another sender as well.

One of the side effects of this is time wasting – it is much more difficult, in my experience, to maintain systems that don’t follow the single responsibility principle.

This is aggrevated by the fact that this is usually a symptom, if not a cause, for bad architecture – I shouldn’t be able to mix logic for two different senders, not unless specified common logic is factored out and reused properly, from a shared assembly.

Now when I come to change some code I find it difficult to know what the impact may be – where exactly is this thing used?

What not to do – projects

So – what not to do?

Here’s one – don’t take 15 different interfaces, from 15 different providers, and the canonical format, and your internal process and bundle it into the same set of projects.

My predecessor (MyP in short) in this project had followed the best practice and ensured (for the most part) that schemas, maps, pipelines and processes each have their own project.

This is good – to start with, in the old days, mixing them used to cause all sorts of build issues, but – although I haven’t tried in several years now – I assume that’s all in the past, but – more importantly – this is bad practice because different artifacts have different ‘resilience to change’ – if you need to add a small shape to a small process that gets instantiated as a result of a receive shape (and not being ‘started’ or ‘called’) you can usually un-deploy it fairly easily.

Change a schema, on the other hand, and usually there’s a whole raft of artifacts you now have to un-deploy with it.

For that reason – mixing two artifact types in the same assembly, whilst not technically problematic, usually suggests you will have maintenance nightmare in the near future (unless you don’t mind down time, and regression tests, that is).

Anyway, as I was saying, MyP did separate the different artifact types to different assemblies, but equally MyP only had one assembly of each; so – when we are processing a message arriving from partner A, for which we have a schema, a pipeline, a map or two and an orchestration – these were split across four assemblies; equally - when we are processing a message arriving from partner B, for which we also have a schema, a pipeline, a map or two and an orchestration, all different – these were also split across THE SAME four assemblies.

On the surface not much wrong with this, the problem is what happens when you want to change a small thing in, say the schema for partner B, used from within a map, which in turn gets used within an orchestration – you now have to un-deploy the orchestration assembly, so that you can un-deploy the maps assembly, so that you can un-deploy the schema assembly, so that you can redeploy the new version of the schema assembly followed by the rest. (ok – so I’m assuming versioning and side-by-side deployment are not being used – which I have to – for my story);

So – all that is the general pain that is part of the BizTalk developer’s day, why am I bringing this here? well because you’ve taken a BizTalk ‘challenge’ and doubled it - for one – you took process A down due to a change in process B; why? had they existed in separate assembly sets you wouldn’t need to… and two – as you’ve released new code – you have to test new code; only that now you have to test two sets of code, including one you haven’t intended to change (but may have, by mistake or otherwise); again – had there been two sets of assemblies, you could probably get away with testing just the scenarios related to the ones you’ve changed.

Now multiply this by about 15 partners, and you see how it can be quite wrong. I hope.

What not to do - premier

Again I find my self having to apologise for not writing for a while (it’s been one month since my last post, two since my last ‘real’ post), the usual suspects are to blame – too much work, too much bureaucracy, kids, new xbox games….

Work wise I’ve taken on a few more small engagement recently, one of which was to enhance/fix/maintain a small-ish solution someone else built.

I could not resist the temptation to take some notes of all the stuff I would have done differently, and am in the process of compiling them as a proper report for the client.

As I was not posting for a while, partly due to this work, I thought its only fitting that I publish some of these ‘recommendations’ here.

For obvious reasons, I will not name names and will try to generalise any samples/explanations provided; I can’t resist mentioning, though, that this other person, I learnt as I looked up his/her name on the web, had also nicked the entire text off my web site and used it on his/her own, which I found rather annoying (I believe the pages have since been removed, after a polite nudge…)

Anyway….keep in mind that many of my notes may come down to style, or MY best practice; they don’t necessarily mean that the other approach is completely wrong -  I don’t presume I know better than anyone else (oh well….) but that there’s value in another point of view – in this case- mine.

So – a few posts coming, likely to be very short and to the point, hopefully someone will findthem useful.

Yossi

Don’t be scared of custom disassemblers

During the last few weeks I’ve been asked to review two separate projects, for two separate companies, developed – naturally – by two separate teams.

The two things both projects had in common were that they both had to deal with legacy “flat files” and they both chose to process these files outside BizTalk using custom code.

In both these cases I completely agree with the decision to use custom code to parse the incoming files – as good as the out-of-the-box flat file support in BizTalk is (made significantly better with the introduction of the flat file wizard, once one gets the hang of using it) – there’s no avoiding from writing custom code to parse flat files every now and then – some file formats are pretty challenging with different records types, conditional records, interleaving segments etc.

I do not agree with, however, the decision to perform this custom parsing outside BizTalk.

I’m pretty sure I would not even bother posting this point had I not seen two of these in the same month, but the fact that I did suggests it may be worth posting a quite note.

One of the projects had the code in a console app, called from a windows scheduled task; the application would pick up files from a folder, parse them, and drop the xml representation in another folder, for BizTalk to consume.

The other had a windows service monitoring a folder and pick up any files, parse them to a different, simplified, flat file format (!), and drop them in another folder for BizTalk to consume.

Both of these introduce another component to the mix; such component needs it’s own error handling, it’s own monitoring, deployment strategy, operations manual etc. similarly it includes a fair bit of re-inventing the wheel – writing code to monitor folders, read files, and write files – stuff that BizTalk is doing out of the box.

What would have been the correct approach then? quite simple – a custom disassembler in the receive pipeline -

Writing a custom disassembler it quite simple - at the end of the day, it boils down to developing a class library, which implements a few simple interfaces, the main one – IDisassemblerComponent defines two methods - Disassmeble and GetNext (the other interface are even simpler, almost insignificant in terms of effort)

Disassemble gets the source message and potentially parses it up front, GetNext is called repeatedly by the pipeline to receive 0 or more parsed messages, until it returns null. simple.

In one of the projects I’ve since taken their existing code (console app), refactored it into a class library, and wrapped it in a custom disassembler class that calls it; converting the scenario to a BizTalk pipeline and performing the key “developer testing” took less than a day.

Why did they not do it to begin with? whilst sometimes there are valid reasons, technical or otherwise, I suspect that in this case it was just unfamiliarity with with BizTalk and some lack of confidence in the development team’s ability to learn and implement (or their belief in themselves); these are valid concerns to any project manager, but I would suggest that a better course of action would have been to spend some time looking at what it takes to implement a custom disassembler, seeing that’s its not at all that scary, and by doing so learning more about a product used in the solution (BizTalk) and achieving a better architecture, and more maintainable approach.

Loosely Coupled….Part II

In a previous post I’ve mentioned our constant attempt to strike the right balance when it comes to loosely coupled services; I’ve mentioned that we were looking at two different scenarios – loosely coupling calls to services outside BizTalk and loosely coupling calls to services inside BizTalk (once implemented within the BizTalk group)

I’ve also mentioned that our solution is composed of a few distinct ‘areas’ (each one generally encapsulated as a BizTalk Application), which we consider, in most cases, to be service boundaries, and – within one ‘flow’ of an incoming request message, we will often have to cross one or more of these boundaries to achieve our end goal.

In most cases, in our solution, the ‘subscriber’ service would use the schema of the ‘publisher’ service for its incoming message; this roughly follows the principle of a service’s proxy, albeit a bit upside down - for practical reasons. Only that - and that’s a much bigger difference - we don’t create a copy of the schema as a service proxy would, but rather reference the schema of the publisher directly (through a shared assembly); this, of course, creates a strong dependency between the two and - over time – this has caused us a lot of headache around deployments as whenever we wanted to update the publisher, we’d have to remove the subscriber too.

Recently we have experimented with following more closely the service proxy approach and instead of using the same referenced schema (using a shared assembly), we’ve created an identical copy of it – same root node and namespace - in the ‘subscriber’ side.

The assumption was that we will be publishing a message using the copy of the schema the publisher holds and be receiving it using the copy of the schema the subscriber holds, but as the message itself will look exactly the same, and will inevitably have exactly the same message type, and so it would be picked up by the subscriber successfully.

Had it worked, we would be have been able to avoid the dependency between the subscriber and the publisher, which would help us gain much needed flexibility in the publisher to support, and change for, multiple subscribers.

Theoretically - if the publisher schema had to change (say – to support functionality required by other subscribers), as long as the change is backwards compatible such as added elements, we could replace the publisher copy of the schema, but leave the subscriber copy as is, until such a point that we need/want to update the subscriber process.

Well - in BizTalk 2006 – this would have worked just fine; unfortunately – from R2 onward, it no longer works – when an orchestration receives a message, it often does so based on a subscription that included the ‘messaging message-type’ (root node and namespace); however – starting with R2 – an additional check has been introduced – to compare the full .net type name of the schema used by the publisher message with the full .net type name of the schema used by the subscriber, assembly, version and all.

This check obviously fails in our scenario, and our fancy loosely coupling solution no longer works (in R2 or 2009).

I think this check is actually a result of code introduced as a hotfix for BizTalk 2004, which – for on reason or another did not make its way into BizTalk 2006 but did into later versions, but I’m not sure, either way – it’s important to note the workaround described at the bottom of the hotfix description, as it appears this behaviour can be turned off, but one would have to check carefully the potential impact.

What else could we do?

Well – one pattern we know that works fairly well is the broker pattern – there’s the publisher, with it’s own schema, there’s the subscriber, with its own – completely different schema, and there’s a broker – a third process that has dependencies on both and contains a map to convert one to the other; on the plus side – this gives us all the flexibility we need – at any one point we only need to deploy two entities – the publisher or the subscriber and the broker, which is good enough; having the process, with a map, allows us to use multi-part messages if we deem them suitable, and all the complexity we need in the mapping; on the down side there are more artifacts to deploy and manage but, more crucially, one extra message box hop which, in a low latency scenario as ours, is not a small price to pay.

Another option is to simply expose the subscriber as a service and call it as such – there are big benefits to doing that – including the fact that we can now have a copy of the schema, in the form of a proxy or without one, and we have also decoupled the services in terms of BizTalk groups –the other service can be anywhere, although this was never a requirements for us; however – we’re paying in more pub/sub again, as well as more IO and quite possibly more complexity.

Theoretically we could have also use XmlDocument (or any other generic wrapper, for that matter) to convey the message, but a) I don’t like typeless intechanges and, b) this does not work well in cases where correlation is required, as the following receive tend to short-cut the subscriber and pick up the request as a response, that is unless you’re willing to introduce two wrappers – one for the request and another one for the response.

Debugging Expression shape code

Sandro Pereira has posted a question, and answer, in the BizTalk newsgroup (he also described his answer, in detail, on his blog) about debugging expression code in Visual Studio

He wasn’t referring to debugging code in helper classes, but code in expression and assignment shapes.

My answer was that this was not possible, but Sandro quickly proved me wrong, as he demonstrates in his answer and blog post, and this got me thinking –firstly – despite knowing about the option to use the generated code (and actually using it on very rare cases to understand a certain BizTalk behaviour) I had never thought of using it for debugging, and that is an interesting thought.

However – I had to wonder – how come I never came across the need to? in all those years of BizTalk development, not even once can I remember thinking – oh! I could solve this if only I could debug the piece of code in this shape..

The reason, I think, is two fold -

1. I rarely have more than 2-3 lines of code in an expression shape of any kind; if it’s not straight assignments, its going into a helper method; it’s cleaner, it’s more reusable, and it’s easier to debug.

2. I use trace. a lot. and so every few shapes or so, and certainly in expression shapes, I will have a trace line that outputs to a log file important information about the state, and the flow of the process; this proves to be invaluable when troubleshooting issues on the live environments, but also really helpful in development.

On ASMX, WCF, namespaces and generated schemas (in BizTalk 2006)

Recently we’ve started to consume a new version of a web service we’ve been using for a while.
We’ve known that, as a whole, not much had changed, only that they have now moved to WCF; they would have migrated their classes to VS 2008 but would expose pretty much the same functions, using pretty much the same parameters.

Still – it appears that BizTalk now insists on generating multiple schemas for the web reference, and as more of the service is moved across more schemas are introduced.

This caused Oleg a fair amount of pain as, when new schemas would be introduced, they would re-order the existing schemas, so reference1.xsd (in the web reference) would suddenly become reference2.xsd, which in turn break out maps.

The process of finding out the logic behind which schemas are created was fairly short and simple, but as I’ve documented it I thought I might as well share it -

Initial observation revealed that whilst the ASMX services’ WSDL file contains all the schemas needed, the WCF services using import statements in the WSDL file; the schemas exist in separate ‘files’.

The ASMX services always uses the XmlSerializer, WCF services use the DataContractSerializer by default, but can be configured to use the XmlSerializer if required.

Here’s a walk thorugh of the scenarios we’ve compared (using BizTalk 2006) -

Standard WCF project, DataContractFormat
We’ll start by comparing the standard WCF sample project generated when you create a new WCF service application in Visual Studio 2008 -

[ServiceContract]
public interface IService1
{
    [OperationContract]
    string GetData(int value);

    [OperationContract]
    CompositeType GetDataUsingDataContract(CompositeType composite);
}

[DataContract]
public class CompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    [DataMember]
    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    [DataMember]
    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Looking at the WSDL generated, 3 schemas are imported –

1.       The usual generic types


2.       The definition of the compositeType type


3.       The definition of the service’s messages (GetData, GetDataResposne, GetDataUusingDataContarct, GetDataUsing DataContractResponse)





Adding a web reference to this service from a BizTalk 2006 project we can see it represents this fairly accurately -


We can see all 3 schemas downloaded from the service, but within the reference.map generated code  a single reference.odx defined the methods in the form of ports and web-messages, and reference.xsd defineds the compositeType schema.

Equivalent project in an ASMX service

I’ve created an equivalent ASMX service, which looks like this –

[WebService(Namespace = "http://tempuri.org/")]
public class Service1 : System.Web.Services.WebService
{

    [WebMethod]
    public CompositeType HelloWorld(CompositeType composite)
    {
        CompositeType response = new CompositeType();
        response.StringValue = "Hello World";
        return response;
    }
}

public class CompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Publishing this service I can see it’s WSDL contains (does not use import, but that proved to be insignificant) a single schema that represents the service’s messages and the compositeType definition.

Consume this service from a BizTalk 2006 project and only the WSDL file is downloaded (there are no ‘external schemas’ to worry about) but within the reference.map pretty much the same odx and xsd files are generated, no real difference between ASMX and WCF here.

Next I’ve looked at changing the serializer the WCF service works with from DataContract to XmlSerializer –

Standard WCF project, XmlSerializerFormatter

Now we will change the serializer to XmlSerializer  by adding XmlSerializerFormatAttributre to both the service and the data contracts  

   [ServiceContract]
    [XmlSerializerFormat]
    public interface IService1
    {

…and

[DataContract]
  [XmlSerializerFormat]
  public class CompositeType
  {

The WSDL in this case includes only one import, for a single schema representing both the service messages and the compositeType schema (basic types are not exposed) and BizTalk now only has one schema downloaded, but again – the reference.map code remained identical (one ODX, one schema)

How will adding a second namespace affect these behaviours? Lets investigate -

WCF project, two namespaces DataContractFormat

To demonstrate this I’ll add another data contract - AnotherCompositeType, specify an explicit namespace for it and include it as a second parameter to the GetDataUsingDataContract operation -

[DataContract(Namespace="HttpL://SomeNamespace")]
    public class AnotherCompositeType
    {
        bool boolValue = true;
        string stringValue = "Hello ";
 
        [DataMember]
        public bool BoolValue
        {
            get { return boolValue; }
            set { boolValue = value; }
        }
 
        [DataMember]
        public string StringValue
        {
            get { return stringValue; }
            set { stringValue = value; }
        }
    }
 
        [OperationContract]
        CompositeType GetDataUsingDataContract(CompositeType composite, AnotherCompositeType anotherComposite);

Using DataContractFormat again, but with two classes, representing two different namespaces, we’re now getting yet another schema - the fourth one - representing the added data contract (if the namespaces of both data contracts were the same, the DataContractFormat would have included them in the same schema)

On the BizTalk side, the reference.map code now also contains a second schema, one describes the original CompositeType, and a second describes the second type – AnotherCompositeType and here as well – were the two types in the same namespace, a single schema would exist, describing both.

Let’s look at the same again, using the XmlSerializerFormat

WCF project, two namespaces XmlSerializerFormat

Adding the XmlSerializerFormat, I also have to remember to include the XmlRoot attribute to set the namespace, as the serializer does not look at the DataContract attribute -

[DataContract(Namespace = "http://SomeNamesapce")]
[XmlSerializerFormat]
[XmlRoot(Namespace = "http://SomeNamesapce")]
public class AnotherCompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    [DataMember]
    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    [DataMember]
    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Now the WSDL for this service, using the XmlSerialiserFormat imports two schemas – one for the service messages, and the CompositeType schema, all reside in the same namespace, and a second for AnotherCompositeType which is defined in a separate namespace

Consuming this from BizTalk and again I’m getting two schemas – one for each namespace.

So far – switching between DataContractFormat and XmlSerializerFormat made no difference to the generated code under reference.map, but it did change the way the WSDL is constructed (import vs. embededed schemas) and therefore the downloaded components (wsdl and schemas, vs. wsdl only)

Note - another thing I’ve noticed is that when a new schema needs to be generated under the reference.map code, as a result of a change to the service, updating web reference does not seem to do so; I had to delete the web reference and re-add it to see the newly added schema.

Last  - let’s look at how the ASMX service behaves with two namespaces –

ASMX service with two namespaces

I’ve added the second class, and added it as a parameter to my web method

[XmlRoot(Namespace = "http://AontherNamespace")]
public class AnotherCompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

And, still consistent – when consumed from BizTalk 2006 I’m getting only the WSDL downloaded (two schemas are embedded) but the reference.map code contains two schemas – one for each namespace.

To summarise -

Using the DataContractFormat you will always get one schema for generic types, one schema for the service’s messages, and then one schema for each namespace any other types are declared in (0..n)

Using the XmlSerializerFormat  schemas are embedded in the WSDL file, and you would get one per xml namespace used.

As far as BizTalk generated code is concerned, however, there’s no difference between the two.

 

What this meant to us – well – we understand better, but there’s still not much we can do.

In our case – we control the service, and – in fact - we know that the only reason we encounter multiple xml namespaces in the service contract is because the various classes exist in several .net namespaces, and they have not supplied the DataContract attribute on any class, they have certainly no supplied the namespace parameter to that attribute, which meant the .net namespace was used as the xml namespace, resulting with multiple namespaces and therefore multiple schemas.

One that team added the attribute, and used a consistent xml namespace throughout, our immediate problem was solved; however - had it been a third party’s service we would not have that luxury and we would have had to update our code whenever we update our web reference, even if only new types were added in a backwards compatible way (as the schema ordering may have changed)

On that – it’s probably easier to simply rename the schemas (and the underlying .net types) under the reference.map generated code rather than the referencing maps and messages.

Creating messages from scratch - revisited briefly

A while back I’ve posted about the different ways to create messages in an orchestration, and later some performance comparison between them.

Mostly for fun I run a quick test on my newly installed laptop; I did not put nearly as much effort as I have previously, so don’t make out of these numbers too much, but I was amazed to see that all the results were running pretty much 10 times faster.

Now – it’s a new BizTalk (2009), new SQL server (2008), new operating system (Windows 7) and a new(-ish) laptop (Thinkpad T61), so there’s no way to know how much each component contributed to the improvement, but it is amazing how much difference can exist after just one year!

Well – not at all scientific, but I found it interesting anyway!

Loosely coupled - Part I

I’m working on a fairly large BizTalk implementation, which in many ways does not fit the classic BizTalk project pattern in that, whilst it has a large element of integration to it, it is even more a business process rich application.

One of the things that constantly keeps us busy is trying to strike the right balance between complexity/performance and maintenance, and a big part of that is finding just how loosely coupled the solution can and should be; this is something I find ourselves spending more and more time on.

There are two parts to this challenge for us – one is about ensuring our processes are not too tightly coupled with any of the many services we’re calling (internal or external) and the other is that different ‘areas’ of our solution, although implemented on BizTalk, and even in most cases within the same group, are treated as independent ‘services’ and are decoupled from each other; we’ve learnt the hard way the price of not getting good enough – mostly in terms of maintenance complexity and cost and are keen to find the right answers, if there are any.

In this part I would like to start tackling the first point – calling services -

In a recent post I ranted somewhat about the WCF’s adapter lack of support for multi-part messages, a topic I’m likely to come back to every now and then I suspect.

This was triggered by my attempt to prototype a way to call services from orchestrations in a loosely coupled way – ideally without the orchestration having to know about the service implementation (isn’t that the promise behind BizTalk Server?)

Naively, I started with a very simple plan – the orchestration would have its own representation for the service call, which it would publish to the message box; the send port, configured with the WCF adapter, would pick up the request based on some subscription, a map in the send port would convert the request to the service’s format and the service would be called; the response would be handled in a similar way.

I have documented that approach here

This worked pretty well in my little prototype, and I was rather pleased with myself for a short while, at least until Ben was tasked with taking this approach into our real world; it took very little time for it to break – all he had to do is call a real service - one that takes more than one parameter (how’s that for a lesson to not over simplify prototypes…)

To be fair, this needn’t be an issue, had we not used multipart messages in our processes, but – accustomed to using the SOAP adapter were using multi-part messages to represent the numerous parameters to the service, and – as I’ve hinted in my previous post – I do sincerely believe this to be the better approach.

That meant two things – 1. the WCF adapter would not be able to transmit the message, as it does not support multipart messages, and - 2. we would only be able to map the body part of the message.

These pretty much put a lid on my idea.

The WCF adapter’s lack of support for multi-part messages we could probably work around using a custom assembler or encoder – such a component could take a multipart message and convert it to a single part message using some pre-defined rules and/or part names, context properties, etc. but the map limitation really puts the lid on this approach.

What can we do then? well – we’d have to create a schema in our process that contains all the parts needed for the service calls, this does not have to look like the service contract, but it does have to contain all the information required to construct the service’s request and effectively means describing the multipart message in XSD; this makes me slightly uncomfortable as, in a sense, this schema exists specifically for this service; it also means we would have to work harder in the process to construct this message – on top of creating and deploying another schema, we would also need some map to convert the multiple messages containing the information to the single entity (or use some code to achieve the same), but it would work – once we have the single part message, we could publish it, map it to the service’s format in the port and deliver it to the service, in the exact same way that worked in my prototype.

At the moment, this looks to me as our only bet, but I’m looking for alternatives, if anyone has any suggestions?

A transform shape gotcha Update

A while ago I blogged about this problem we've found with the transform shape; what I should have done, and haven't (I believe) is blog about the fix Microsoft issued as a result of our bug report.

You can read the details here

Interestingly I think the scope of the problem was bigger than we originally thought, certainly bigger that what I blogged about anyway, in that even if you don't have the tranform shape problem (for exmample - if you've created the orchestration on a machine with the hotfix, but then opened it in a machine without it) you can have some issues -

In one case we had a map, where the input was composed of several schemas; as you probably know BizTalk effectively creates a schema that aggregate all the parts into a single schema; to achieve this it had to impot several schemas from our solution, all share the same namespace;

When we tried to build the project with the map, we received errors saying a certain element type could not be found, the cause was that BizTalk was looking for it in the wrong schema, as it had several schemas imported into the same namespace, it chose the wonrg one and could not find the type definition.

Once we've installed the fix, and with no other changes, the map compiled all right.

Parallel shape behaviour in BizTalk 2004 and >2006

‘Shiri’ had posted this question in the newsgroup -…

…After open the orchestration
debugger both at BT 2004 and BT 2006 we've recognized a different behaviour:
at 2004 executes through all the branches first shape and then back to the
first branch at parallel and executes the second ahape, the second branch
and etc. at 2006 executes the first branch - first and second shape, then
goes to the second branch and executes the first and second and etc.
It seems that 2004 works more like multithreaded then 2006….

I vaguely remember reading/discussing this difference in the past, but – unfortunately - I can't remember the exact details; I've floated this question around again, and I believe (but treat this with care – this may well be at least somewhat inaccurate) that there has indeed been a change in the behaviour of the parallel shape in BizTalk 2006, here’s some background -

The parallel shape was never intended to provide ‘true’ parallelism, and Microsoft has been fairly clear from the start that BizTalk will not process each branch on it’s own thread (which would have been required).

Darren Jefford explains very well the intention, and the expected behaviour, of the parallel shape in his book Professional BizTalk 2006, which I highly recommend; if you want a quick peek you can read the relevant piece here, the key point he makes is that when thinking about the parallel shape, you need to wear your business analyst hat, and not the developer hat – the parallel shape effectively “says” – hey - I’ll run this code (=branch), but if I reach a point where I’m sitting idle waiting for something to happen (receive, delay or listen shapes) I will go and run that other code (=branch) in the meantime; when I’m through with that (or reached a waiting point again) I will check if I’m ready to process the rest of the first bit of code…and so on…

This is not quite your techie run-things-in-parallel-on-multiple-threads approach, but – from my experience – it is more than enough (if you wanted to run things completely in parallel you could use BizTalk’s pub/sub, which would allow you to potentially get a lot more than just one thread – you might end up on a different machine altogether, for a price :-))

So – indeed – the BizTalk 2006’s (and subsequent versions) parallel shape behaves exactly as I understand it should do (and as is described in Darren’s book) and is consistent with Shiri’s observation - if you have three branches, and neither have a blocking shape – the left most branch will be executed completely, then the next one to its right and then the right most branch; however –and that’s a very important point to remember - Microsoft does not, to the best of my knowledge, guarantee any order of processing between the branches, and this might change in future versions (as indeed it has been seen from 2004 to 2006), so all you can assume is that all the branches can theoretically run in any arbitrary order or indeed in parallel.

Back to the 2006 behaviour - if, however, you had a receive shape as the second shape in the left most branch, when BizTalk would hit this shape it would move on to execute the second branch while it’s waiting for the message to be received; it would come back to the first branch at the earliest point once two things had happened – 1. the message it was waiting for was delivered and 2. it had reached a point in the currently executing branch in which it can stop and re-enter the first branch; this would be a receive shape, delay shape, listen shape or the end of the branch.

So – if that’s the 2006 behaviour, was the behaviour in 2004 different? yes, I believe it was – in 2004 the engine was, some would say, trying to be too clever - BizTalk 2004 would try to run branches on multiple thread if it can; where “if it can” depends on several factors, not the least the state of the thread pool at the time of evaluation; if it managed to do so, you would get code running truly in parallel, as Shiri observed, but there are no guarantees that this would be the case; in that sense BizTalk 2004 is less predictable than later versions of BizTalk, which is exactly the problem with this approach, and - considering that this was never the intention to begin with – I can fully understand the decision to simplify the model in BizTalk 2006.

Oslo based solution for deploying BizTalk applications – published

Unfortunately life is a bit hectic for us at the moment, and it’s been a while since I’ve posted anything, or was able to do anything other than family or work.

As part of this general “neglect” I was unable to spend the time required to complete, and publish, my Oslo based solution for deployment BizTalk applications (given the somewhat ugly, but suitably short, name – BTSDF), but as many waited (ok – more than 5), and some kept asking (2), I thought I’d do my best to get something out of the door, so I have.

 

in Codeplex  you can now find an initial version published, which includes – the language definition, the core runtime I have as well as two “executors” in various stages of [non] completeness – my MsBuild generator is already quite useful for simple apps, it generates a set of MSbuild files you can use to deploy the application on any machine as well as the required SDC tasks dlls and targets file; the API executor deploys the app on the current machine, and is quite basic, but a good sample (I think) and a reasonable starting point.

I’m happy to entertain requests for changes/additions, and even more happy to add any one who’s willing and able to put the time as a contributor; Oslo knowledge is optional! :-)

 

You can download the source, build and run it locally – but you will have to remember to copy the two executors dlls from their folders to the main bin\debug folder as there is no compiled reference to them.

 

I’ve also uploaded an “Alpha release” which includes the compiled assemblies and the supporting files required.

There’s still more work to do, but it’s getting shape now, and I’m using the MSbuild executor today; it certainly needs a bit more documentation, which I hope to get around to…

 

I hope this works for you, let me know what you think (the good and the bad), and – if you have some time and will – drop me a line and I’ll add you to the team!

 

Yossi

Oslo based solution for deploying BizTalk applications – the runtime

This is a third post in a series describing my Oslo based solution for deploying BizTalk applications; I’ve used this exercise to play around with ‘M’, but it was important for me to work on a real solution, with real benefits – something I could actually use…in Part I I discussed the concept and presented both the “source code” of my app and the output I was working toward; Part II was all about the MGrammar part of the solution.

In this, third, part I will discuss the last missing piece –the runtime.

Before I start, though, I would say that I did find getting into ‘M’ somewhat confusing at first; and while it’s more than just possible I’m still missing some things , I hope this series could help one or two people in their journey with Oslo – which is, without a doubt, an exciting one!

There are two things, I believe, that contributed to my confusion - the first is the fact the M is really many things, quite different things, actually; from what I hear Microsoft have identified the challenge some of us (me) are having getting a grasp on ‘M’ and are hard at work to bring things [closer] together; hopefully it won’t be long before we know how the converged language looks like, in the mean time one simply has to remember that -

There’s MSchema - which you could use to define models,  a bit like xml-schema, or declaring your classes in code or even tables in SQL; I haven’t really touched on MSchema in this series butI might come back to that later.

Then there’s MGraph, which is a way to define instances of things, possibly ones that have been modelled using MSchema, but, as is evident from my little project, not necessarily - MGraph can be very useful even if you don’t have a model- as long as you have your grammar – in comes MGrammar, the third spect of ‘M’, which can be used to define a syntax for your very own [domain-sepcific-]languge for describing things;
A ‘runtime’ could then be used to processes instances described as MGrammar as a result of inputs in your language.

And that is the second thing that really confused me – what is that ‘runtime’? in all the ‘M’ presentations I’ve seen, the ‘runtime’ was merely mentioned and has never received enough “floor space” and yet – an MGrammar without a runtime, in the majority of cases, is, quite useless; you have to have a runtime that would act on your source code; in fact – the runtime would act on the MGraph resulting from your language, which is what makes it all so brilliant, because in a sense, this is where everything comes together – you runtime can work on instances described in your language, on MGraph instances stored in the repository created using MSchema and possibly even ones defined using Quadrant.

The point is that there must be a runtime that understand the model behind your language , can parse its graph and then do whatever you need it to do; and it is your job to build that runtime.

So what have I done for my runtime? here’s a quick overview (reminder: the full source code will find its way shortly onto codeplex) -

My runtime is a console application, one that takes a source code file path as an argument and outputs MSBuild files (and dependencies) that can be used to deploy the application described in the source code onto BizTalk Server.

The first part of my runtime - which I will not bore you with -  is about validating the command line arguments; standard stuff.

The second part is about creating  the parser for my language, where, thankfully, the Oslo SDK does all of the heavy lifting – it includes a class called DynamicParser which, once created, you can use to parse your source code.

To create the DynamicParser you must first compile your language, and that’s easy enough to do – you start by creating a compiler

MGrammarCompiler compiler = new MGrammarCompiler();

and continue by supplying your grammar

compiler.SourceItems = new SourceItem[] {
        new SourceItem {
            Name="BTSDeploy",
            ContentType = ContentType.Mg,
            TextReader = new StreamReader(GetLanguageDefinition())
        }
    };

(GetLanguageDefinition() is a simple helper method I wrote to get the grammar file embedded as a resource in the exe)

Now you’re ready to compile your language, but to make things manageable you want to provide it with an error reported; the compiler would report any errors to the stream you would provide, I’ve naturally used the console

TextWriterReporter errorReporter = new TextWriterReporter(Console.Out);
if (compiler.Compile(errorReporter) != 0 || errorReporter.HasErrors)
{
    Log("Failed to compile language definition\nSee above for details");
    return null;
}

If the compilation succeeded you are ready to create your parser -

DynamicParser parser = new DynamicParser();
compiler.LoadDynamicParser(parser);

That’s part one of three done.

The next step is to use the dynamic parser to parse your source code, the output of which would be a graph representation of the source; luckily the SDK does virtually all the lifting here as well, and it comes down to one line -

object rootNode = parser.Parse<object>(sourceCodeFileName, null, errorReporter);

Note that the output type is object – which, as you will find out if you try this out, is quite painful– currently all the types used in the Graph are internal, which makes debugging quite difficult (you can’t quite look at any variables you hold in any meaningful way, you have to keep calling methods, as you’ll see next; hopefully this will change one of the next updates to the SDK.

In any case rootNode is now pointing at the root of a graph – a tree like structure you could ‘walk’ to extract the pieces of information you care about in the source code; here you’re expected to use methods like GetLabel, GetSequenceElements and GetSuccessors to reach nodes and their values in the graph and, of course, to do that you need to know exactly how your graph looks like; my first instinct was to look at the PreviewMode pane in intellipad (usually the right most pane when working with MGrammar) as it shows you a representation of the MGraph created for the source code and language used; this worked quite well, but, as I found out, wasn’t the most trivial thing – the two didn’t align completely and I ended up having to resort to trail-and-error to get the parsing logic right.

The reason is that M has a few shortcuts one could take, but the graph you would be working on is the very basic, more verbose format; some information on this is mentioned here.

Then, on a recent visit to Redmond, Dana Kaufman passed on a great tip – if you ‘compile’ your grammar using mg.exe to create the mgx file (basically a ZIP file containing XAML representation of language) and then use mgx.exe on your source file adding  a reference to the mgx file you just created, you end up with an ‘M’ file which is exactly the graph your runtime would be working on.end up with; so useful!

So – here are a few example of how I worked the graph – to start with I knew my root node should be a node with the label ‘Application’, so I checked it this way -

string label = graph.GetLabel(rootNode).ToString();
if (label != "Application")

I then knew that the application name would be a child element of the root node, so I extracted it like this

// extract the application's data - this should contain two nodes - the application name and the list of items in the application
List<object> appData = graph.GetSequenceElements(rootNode).ToList<object>();
//first line should be the application name, make sure it is not a node and extract the label
if (!graph.IsNode(appData[0]))
    Contents.AppName = appData[0].ToString();

the second node in the appData collection is where the graph ‘continues’, so to get the list of things that compose my application I needed to walk down that path -

//the second element should be the list of lines
foreach (object section in graph.GetSuccessors(appData[i]))
{
  //each successor would be a category (reference, importing binding, resource, etc), with a list of items
  processSection(graph,section);
}

with processSection start with

string sectionName = graph.GetSequenceLabel(graph.GetSuccessors(section).First()).ToString();
List<object> items = graph.GetSuccessors(graph.GetSuccessors(section).First()).ToList<object>();

Log("Found section '{0}'", sectionName);
switch (sectionName)

I hope that from these few examples you can see what it takes to work the graph – the graphBuilder (which is a somewhat confusing name, as I’m using it to walk the graph, not build it) has all the methods you need to access the various nodes ( but there’s no xpath-like- support), but as all the types are (currently) internal to the MS assembly you’re always working with objects, which is less then ideal.

Again – my full source code is on its way to codeplex, I just want to make sure it’s commented well enough to be well understood, and am struggling with time, but the bottom line is that once you figure out how the graph builder works, learnt how to see your graph visually (using mg.exe and mgx.exe) and got used to the fact that you’re dealing with objects for now, parsing the source code is very easy.

Obviously it is completely down to you what you then do with all the information you’ve extracted from the source code; in my case  my runtime is using a plug-in model so the first part is all about using the Oslo SDK to get an instance of a BizTalkDeployment class populated based on the contents of the input file, this class looks like -

    public class BizTalkDeployment
    {
        public string AppName { get; set; }
        public List<object> References { get; set; }
        public List<object> Build { get; set; }
        public List<BizTalkAssembly> BizTalkAssemblies { get; set; }
        public List<object> ImportBindings { get; set; }
        public List<Binding> AddBindings { get; set; }
        public List<Assembly> Resources { get; set; }
    }

I then use late binding and configuration to load a plug in that would take an instance of this class and do the work, be it generation of msbuild scripts, deploying to the local machine using BTSTask or anything else.

On Atomic Scope and Message Publishing

A few weeks back I worked on a process that looked something like this -

It was triggered by the scheduled task adapter and then used a SQL send port to call SP to return list of ‘things’.
It needed to split the things in the list to individual records, and to start a new, different, process, through pub/sub (to avoid the binary dependency with the called process), for each ‘thing’.

Fairly simple.

A lot of have been said on the different ways to split messages, I won’t repeat this discussion here; I would just say that initially I used a different approach – I used the SQL adapter in the initial, triggering, receive port and then used a receive pipeline, with an XmlDisassembler component, to split the incoming  message so that each record was published individually thus avoiding the need to have a ‘master process’; that back fired though, in my case – I quickly realised I’ll be choking the server with the amount of messages published and needed a way to throttle the execution; I’ve played a bit with host throttling but then came to the conclusion the best approach for me would be to throttle in a process, which is what I’ve done.

And so - to make things interesting, and because I already had it all ready - I decided to use a call to a pipeline from my process to split the message.

The first thing I realised, trying to take that approach, was that I had to change type of the response message received from the SQL port to be XmlDocument (which is an approach I generally dislike – I’m a sucker for strongly-typed-everything) – but my schema was configured as an envelope so that when I call the pipeline from my process it knows how to split it correctly, but, when used in the SQL port BizTalk split the message too early for me – I needed to whole message in the process first, which was no good to me; if , however, I removed the envelope definition from the schema when I would call the pipeline directly from my process it won’t know how to split the message, which is no good either; nor could i have two schemas (BizTalk, as we all know, dones’t like that bit at all, not without even more configuration); XmlDocument it is.

It then came back to me (in the form of a compile time error :-)) that the pipeline variable has to exist in an atomic scope, and so I added one to contain my pipeline variable; I then added the necessary loop with the condition set to the GetNext() method of the pipeline and in each iteration constructed a message using the GetCurrent() method; all standard stuff.

I would then set some context properties to route my message correctly and allow me to correlate the responses (I used a scatter-gather pattern in my master process) and published it to the message box

What I noticed when testing my shiny new process was that all those sub-processes that were meant to start as a result the published messages in my loop were delayed by quite a few minutes (6-8), which seemed completely unreasonable, so I embarked on a troubleshooting exercise which resulting in that big “I should have thought of that!” moment.

While the send shape in my loop successfully completed its act of publishing the message in each iteration, moving my loop to the next message and so on, being in an atomic scope BizTalk would not commit the newly published messages to the message box database, allowing subscriptions to kick in, before the atomic scope would finish; that is to allow it to rollback should something in the atomic scope would fail.
What it meant for me though, was that all the messages were still effectively published at once, which brought me back to square one (or, minus one, actually, considering that the great delay caused my this approach means I’m even worse off from my first debatch-in-pipeline approach).

And so I went back to the old and familiar approach of splitting the messages using xpath in the process, which allowed me to carefully control the publishing rate of messages for my process and throttle them as needed.

ASMX, WCF and enums woes

We’ve been slowly migrating our services from asmx to WCF, but as we’re still using BizTalk 2006 with no support for WCF we’ve been exposing endpoints configured for basicHttpBinding and consume them using the SOAP adapter.

Generally speaking things have been going well, although we completely gave up on the idea of moving the services to WCF and NOT have to change the client, until yesterday we’ve stumbled into a serialisation issue –

The SOAP adapter, as part of its work deserialises the request message arriving through the send port into t he web service proxy class it generated, before calling the web service (which would result in the class now being serialised back into xml, which is another story); that deserialisation failed.

The error message was clear enough and indicated it failed to deserialise an enum parameter the service was expecting, and that ran a bell – I posted on exactly that back in September, but after carefully checking and re-checking everything we could swear that our message (which was now suspended) matches perfectly the schema generated by the add web reference wizard; what’s going on then??

After chasing our tail for a short while we brought up reflector to the rescue and found out the cause of our woe is a combination of a difference in behaviour between WCF and ASMX and the use of BizTalk – here are the details –

Consider the following asmx web method  –

[WebMethod]
 public string GetDataUsingDataContract(CompositeType.someEnum myEnum)
 {
     return "Hello World";
 }

With CompositeType being

public class CompositeType
 {
   public enum someEnum
   {
      Value1,
      Value2
   }
 }

(..and pretend CompositeType has many more things, but these are irrelevant to this topic)

The definition for myEnum in the WSDL looks like

<s:element minOccurs="1" maxOccurs="1" name="myEnum" type="tns:someEnum" />

Where the type tns:someEnum looks like

<s:simpleType name="someEnum">
    <s:restriction base="s:string">
    <s:enumeration value="Value1" />
    <s:enumeration value="Value2" />
    </s:restriction>
 </s:simpleType>

As a result the definition of the enum in a proxy generated via the add web reference VS 2005 option (which is what BizTalk would use) looks like –

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Xml", "2.0.50727.3053")]
 [System.SerializableAttribute()]
 [System.Xml.Serialization.XmlTypeAttribute(Namespace="http://tempuri.org/")]
 public enum someEnum 

 {
     Value1,
     Value2,
 }

All makes sense.

Now, let’s look at what WCF does in the same case; consider the following service –

[ServiceContract]
 public interface IService1
 {

   [OperationContract]
    string GetDataUsingDataContract(CompositeType.someEnum myEnum);
 }

 [DataContract]
 public class CompositeType
 {

    public enum someEnum
    {
        Value1,
        Value2
    }
 }

The WSDL generated looks like

<xs:simpleType name="CompositeType.someEnum">
 <xs:restriction base="xs:string">
   <xs:enumeration value="Value1" />
   <xs:enumeration value="Value2" />
 </xs:restriction>
 </xs:simpleType>
 <xs:element name="CompositeType.someEnum" nillable="true" type="tns:CompositeType.someEnum" />

The key difference is that the name of the class containing the enum has made it into the type name for the enum, which never happened in the ASMX version.

As a result the proxy is generated as such -

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Runtime.Serialization", "3.0.0.0")]
 [System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum", 
                  Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
 public enum CompositeTypesomeEnum : int 

{
   [System.Runtime.Serialization.EnumMemberAttribute()]
    Value1 = 0,
   [System.Runtime.Serialization.EnumMemberAttribute()]
    Value2 = 1,
 }

Again – note the name given to the element now contains the class name and, crucially, a dot (‘.’).

On it’s own – nothing to malicious – although it’s another nail in the coffin for the idea that you can substitute web service with WCF service, configured them to use basicHttpBinding and all should be the same (ok – am I the only one still wishing this was possible?)

Enters BizTalk.

When you use the add web reference wizard to add a reference to the WCF service, BizTalk generates all the schemas and proxy for you, which is what you would use to create requests going to the service (and process responses).

Because the WSDL of the WCF service contains the longer name of the enum (with the class name, the dot and the enum name) the .net proxy generated is identical to the one created for the WCF service above; the schema, however, is generated incorrectly!

BizTalk “kindly” decides that having dots in the element name is not a good idea and removes it so the schema generated looks like this –

<xs:schema xmlns:tns="http://schemas.datacontract.org/2004/07/WcfService1" elementFormDefault="qualified" 
          targetNamespace="http://schemas.datacontract.org/2004/07/WcfService1" 
          xmlns:xs="http://www.w3.org/2001/XMLSchema">
 <xs:element name="CompositeTypesomeEnum" type="tns:CompositeTypesomeEnum" />
 <xs:simpleType name="CompositeTypesomeEnum">
   <xs:restriction base="xs:string">
      <xs:enumeration value="Value1" />
      <xs:enumeration value="Value2" />
   </xs:restriction>
 </xs:simpleType>
</xs:schema>

“CompositeTypesomeEnum”??????

Well, we’ve seen this, and created a message with exactly that element, which – of course – the SOAP adapter failed to deserialise into

[System.Runtime.Serialization.DataContractAttribute(Name="CompositeType.someEnum", 
                                                    Namespace="http://schemas.datacontract.org/2004/07/WcfService1")]
 public enum CompositeTypesomeEnum : int 

{
   [System.Runtime.Serialization.EnumMemberAttribute()]
   Value1 = 0,
   [System.Runtime.Serialization.EnumMemberAttribute()]
   Value2 = 1,
 }

The solution was fairly simple – we’ve simple change our xsl to put the element name as the .net proxy requires it, and not as the schema describes it, and it all worked well.