BizTalk WCF adapter and multipart messages

I wrote myself notes for this post months ago, but haven’t taken the time to write it down properly; a passionate discussion today with Rupert Benbrook prompted me to get back to this.

The WCF adapter does not support multi-part messages, and some would argue that’s a suggestion of the feature possibly being dropped in some future version of the product.

This small fact seems to have been glossed over by most – I, for one, haven’t heard this discussed anywhere, to the point that I’m left wondering if I’m really just “stuck in my ways”, but I honestly think that’s a poor decision.

Trying to search for some detail about this fact I found the following, rather old, but still very valid and informative, post; here are some of the key points made -

The reason support for multi-part message in WCF adapter is not being recommended is because WCF does not support it and the model proposed is that whoever wants to use something equivalent, need to aggregate the messages into one xml and send it as one single-part message.

(Tapas Nayak, Principal SDE, BizTalk Adapter Pack)

Multipart messages are a marginalized feature in BizTalk, the pipeline components like the flat-file and xml don’t understand them, same story for the majority of the adapters. So we decided to not support them in the WCF Adapter too

(John Taylor, Senior Technical Lead, BizTalk Server)

 

With regards to the first statement – I think that is somewhat down to interpretation – WCF did not change how web services work; WSDL is the same WSDL, specifications have not really changed; so – WCF is just a new (and of course – much better) implementation; this is an important basis for any discussion because it indicates that the WCF adapter could have taken the same path the SOAP adapter did.

Consider the following Service definition in WCF:

        [OperationContract]
        CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut);

or the equivalent web method in asmx:

        [WebMethod]
        public CompositeType GetDataUsingDataContract(CompositeType composite, CompositeType composite2, out CompositeType compositeOut)
        {
        }

the WSDL for both methods is pretty much identical, the operation definition looks as follows -

<wsdl:operation name="GetDataUsingDataContract">

<wsdl:input wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContract" message="tns:IService1_GetDataUsingDataContract_InputMessage"/>


<wsdl:output wsaw:Action="http://tempuri.org/IService1/GetDataUsingDataContractResponse" message="tns:IService1_GetDataUsingDataContract_OutputMessage"/>


</wsdl:operation>

the messages are defined as so -

<wsdl:message name="IService1_GetDataUsingDataContract_InputMessage">

<wsdl:part name="parameters" element="tns:GetDataUsingDataContract"/>


</wsdl:message>


<wsdl:message name="IService1_GetDataUsingDataContract_OutputMessage">


<wsdl:part name="parameters" element="tns:GetDataUsingDataContractResponse"/>


</wsdl:message>

And the schemas, in both cases, define the aggregated messages – with all the in/out parameters as required.

So – what is the difference? the way I see it is it is in the way the different adapters handle the messages as they pass through the adapter; the SOAP adapter has a layer in which, when receiving messages from the outside,  breaks them to individual parts, delivering a multi-part message to the message box; similarly, when delivering messages from BizTalk to the service, it composes them, aggregating the various parts in the request message, to a single SOAP envelope; the WCF adapter does not include such functionality – it will take the message BizTalk provides and will wrap it with the SOAP envelope pretty much as is.

All my grievances about the SOAP adapter aside – I think its approach is a good one; it allowed the process to focus about the flow of the business process and avoided the need to define and construct artificial entities in the form of aggregated schemas, which often make very little sense (as the various parts can be largely unrelated from a process perspective, but still necessary for the service to perform its operation).

In my discussion with Rupert the same point was raised – the fact that SOAP supports one message in and one message out (i.e. no multipart messages), but in my view that’s a problematic argument – again – the whole point of the BizTalk architecture is that the send port can mask the transport mechanisms; ok – the SOAP adapter is not doing a particularly good job at that, but that’s hardly an excuse to make matters worse – I don’t want to have to know that I need an aggregated message to call this particular message because it uses the WCF adapter;

Moving on - I particularly disagree with the second statement mentioned - I think multipart messages are a very important feature in BizTalk, and would be surprised if any seasoned BizTalk developer would tell me they prefer not to use them (do let me know if you think I’m wrong); to start with – it allows one to define a type once and then use it multiple times; this makes maintenance infinitely better when types need to be changed (I thought it ought to be this schema, but found that this other schema is better); ok – so you have to be careful about how and where you re-use, to not introduce unwanted dependencies, but if you have large orchestrations which follow several steps,  this is very useful.

Secondly, it allows one to easily link pieces of information that are only ‘somewhat-related’, especially important, in my view, when using pub/sub; having to pass several pieces of information from the publisher to the subscribers; the alternative, not surprisingly, would be exactly what is suggested above – create a schema defining the various pieces of information required and use that instead, but I think that’s a much larger maintenance overhead than a multi-part message definition; it is much easier to change a message definition that it is to change a schema; especially in large implementation with many different assemblies;

This was the point most passionately discussed with Rupert today – he argued that if a message is important enough to be published (with all its parts) it is important enough to create a schema for it; the point I was making, possibly not good enough, is that – to me, specifically in this area, the two are the same – I can define my aggregated message in a schema, that would have, say – 2 elements, each using a type defined in my other, existing schemas, or I could use a multi-part message, with two parts, each pointing at existing schemas; there’s very little difference in principal – I’m defining something – strongly typed and well defined,  that is composed of two other, well known things, and I’m passing it around.

Obviously interoperability can only be achieved using standards like XSD, but that’s not a consideration at this point I’m publishing a message from BizTalk, which would be subscribed to by BizTalk, what’s wrong with describing it using BizTalk terms where conceptually there’s virtually no difference?

If there is no difference – why do I care? because multi-part messages are easier to create than aggregated schemas, and there’s less deployed artifact; I can simply put an assign shape and assign existing messages to the relevant parts; if I was to go down the aggregated schema route I would have to define the schema and create a map that would aggregate the messages; I could use code, or I could use xpath to assign the parts to the various elements, but I’d have to use a map or something to create the shell of the message first; seems like a lot of overhead for simply passing a couple of messages between processes.

I agree that many adapters don’t support multipart messages (the file and ftp adapters are the most prominent examples, outside the WCF adapter), and in many cases this makes perfect sense, but some do – SMTP and POP adapters, for example, use it very well; so does the SOAP adapter, of course.

There was one piece where we were in mostly in consensus, about the benefits of a single schema versus multi-part messages – when you wish to have  a map in the port, I believe BizTalk would only consider the body part of the message, but again – to me that’s a shortfall in the product, not a proof that the existing feature is not useful; for both this and the pub/sub scenario I would have lover to be able to treat a multi-part message type in the same way a single-part message is treated – with a specific type that you can subscribe on (I can always set the message type manually, I guess, but would have been great to get out of the box support)

Similarly I agree that the built in xml and FF disassemblers don’t support multipart messages ( only in the sense that they will not disassemble them), but the model does – you can easily access, and parse, multipart messages from within a pipeline component, even a custom disassembler, so I can’t help wondering if the answer is in the following statement made by John Taylor, as quoted in that blog post -

The feature set for the WCF Adapter was potentially very large indeed and we were under some pressure to make cuts where we could.

and I think it’s a big shame.

I, for one, would be very disappointed if multi-part messages stop being supported, and I’m slightly worried – WCF is obviously taking more and more front stage, generally and with BizTalk specifically (even more so with the LOB adapter SDK and the BizTalk adapter pack that makes use of it), and as it does not support multi-part messages, it may well set the tone for future BizTalk developments.

MSBuild support in BizTalk 2009

In my notes on BizTalk 2009 post I have highlighted the fact that now, as part of the BizTalk installation, you can select the “Project Build Components” which allow you to build BizTalk projects without Visual Studio or BizTalk installed on the build machine.

This is very cool feature of the product, but I have to admit that I have missed something quite important; luckily - Mikael Hakansson hadn’t – the provided tasks do not support deployment of BizTalk projects, but only build (see Mikael’s article here).

What this means is that, while you can use the tasks deployed through the installation to verify that your BizTalk projects build correctly (as part of continuous integration setup, for example), you have no way of verifying that they can be deployed or, more importantly, run any automated tests;

The SDC tasks, with their support for BizTalk deployment, can come in very handy here, and you can certainly use them to augment the built in build support with deployment capabilities, but they add more dependencies to the plate, which has to be planned for.

Of course, attending the MVP summit in Redmond last week we had a great chance of raising this (and various other) subjects with the product group, and we certainly have not let that opportunity go missed, now we just need to wait and see if they “pick up the glove”.

Publishing a service with federated identity from BizTalk Server [2009] – Part II

In my previous blog post I’ve described how I consumed a service that uses ws2007FederationHttpBinding from BizTalk Server; my next task was to expose an orchestration as a WCF service that uses this binding.

In that post I’ve described what I think is a bug in BizTalk R2/2009 which prevents me from setting the issuer configuration through the UI.

When consuming such a service this configuration exists in the send port, and I’ve managed to get enough time to manually edit a BizTalk bindings file to import in order to set the required configuration I can’t set through the UI; I have tried to do the same for the receive location, but the binding file's format is different and my few attempts before my time ran out did not work.

I have submitted this buy to Microsoft, and hopefully they’ll provide a fix/some information about a possible work around; I suspect my best bet is to use the explorer OM to configure the receive location, but I just can’t get the time to do it now, so – I’m afraid this post is mostly a place holder, and I really hope I’d be able to come and update it with some more details soon.

Calling a service with federated identity from BizTalk Server [2009] – Part I

Finally I’ve reached the point where I’m ready to hook up BizTalk to my STS implementation to participate in a federated identity scenario.

My goal is to confirm two scenarios -

1. Being able to call from a BizTalk process a service that uses the ws2007FederationHttpBinding (and requires that the caller provide a token issued by a specific STS)

2. Being able to expose a service in BizTalk that would use the ws2007FederationHttpBinding requiring the caller to provide such token.

If you followed my previous posts you would probably know that I already have a fairly extensive federated identity scenario – I have a custom STS built using the Geneva Framework, a few web sites and a test WCF service all use the STS for their authentication (and, indirectly, authorisation), so it is just a case of hooking BizTalk to all of that, and to start with I decided to try and consume that test service from a BizTalk process.

I’m using BizTalk 2009 Beta to keep things interesting, but I’m pretty sure it would all be exactly the same for R2.

The beginning was easy – in my BizTalk project I’ve used the “Consume WCF Service” wizard from the “Add Generated Items” menu to generate all the artifacts needed to consume my service; the generation went very smoothly and soon enough I had an orchestration that could receive a dummy trigger message, create the service request, receive the service response and deliver it to a send port for me to check it out.

I’ve deployed the project and then imported the binding file generated by the wizard to create the WCF send port.

Checking the port configuration, though, I found a few issues -

The port uses the WCF-Custom adapter, which was correctly configured with the service’s endpoint and the ws2007FederationHttpBinding; however – my STS endpoint is configured to use the ws2007HttpBinding and requires UserName credentials, but these settings were not copied over to the client configuration from the service's contract, the client configuration should have looked like -

<security mode="Message">
  <message algorithmSuite="Default" issuedKeyType="SymmetricKey"
    issuedTokenType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV1.1"
    negotiateServiceCredential="true">
    <issuer address="[STS URL HERE]” binding="ws2007HttpBinding" bindingConfiguration="stsBinding">
      <identity>
        <dns value="STS"/>
      </identity>
    </issuer>
    <issuerMetadata address="[STS MEX ENDPOINT HERE]" />
  </message>
</security>

(with corresponding Ws2007HttpBinding named stsBinding with all the relevant configuration), but the binding attribute and the bindingConfiguration attribute were missing.

It is worth noting here that this problem is by no means BizTalk specific – exactly the same thing (missing configuration) happens when I add a reference to my service from a ny standard .net app.

Also worth noting that in my case, and I don’t know how representing this is, this resulted in WCF trying to open the CardSpace card selector when I tried running the process, resulting in an error - “The channel is configured to use interactive initializer 'System.ServiceModel.Security.InfocardInteractiveChannelInitializer', but the channel was Opened without calling DisplayInitializationUI.  Call DisplayInitializationUI before calling Open or other methods on this channel.”

Easy enough to fix, I thought, as I opened the adapter’s configuration for the send port finding that indeed there are the equivalent fields in the UI (the image shows them AFTER I have typed in the values I needed) -

However, after adding the missing two values and confirming the changes I kept getting the same error; it only took two minutes of head scratching to realise what was going on – opening the send port configuration again showed that the values I’ve carefully typed in have not been persisted, and are now gone again; this must be a bug, which I’ve since confirmed exists in BizTalk 2006 R2 as well;in fact – even the issuer address, which was previously there, was removed from the configuration when trying to apply my additions.

Luckily importing these settings through a bindings file does work (as was evident from the fact that the issuers address did exist initially), and so I’ve edited my bindings file manually to include the missing settings (as well as the issuer identity setting which I later found out was missing for my scenario) and imported it using the admin console; checking the send port configuration I could now see my values for the “binding” and “bindingConfiguration” properties.

This raised another question, though – how do I configure this binding? ws2007FederationHttpBinding is somewhat “special” in the sense that there’s effectively an endpoint inside an endpoint:

Using this binding you have your service’s endpoint and it’s binding configuration; then the message security element of the ws2007FederationHttpBinding has the issuer element which is effectively the endpoint of the STS with the address, the binding and the binding configuration (there is no need for a contract attribute as the contract is known – it is the wsTrust contract); this means that I now need to have another set of binding configuration in my WCF configuration – for the STS – which in my case uses ws2007HttpBinding, but there isn’t a way to add another binding through the send port configuration dialog directly.

Initially I tried to add the relevant configuration to the BizTalk configuration file (BTSNTSVC.exe.config), but it didn’t work – and I can’t quite explain why yet.; what did work, which is good enough for me for now, was to put that exact same configuration in the machine.config; no idea why, but it works.

Another options, described in the documentation  is to use the Import option or the ExplorerOM API.

And so – once I’ve done that, when I ran my process it called the send port going to my service, but the WCF adpater contacted my custom STS first, retrieved a security token from it, and then called my test service passing this token, which meant the service was allowed to execute.

I’m impressed!

There is one, fairly big, inefficiency here now, though – BizTalk does not appear to be caching service proxies, which means it will not cache anywhere the issued token, which in turn means a call to the STS must precede every call to my protected service; not ideal.

Luckily - one of the WCF samples provided by MS demonstrates how to create a behaviour that would cache such tokens locally; at some point (probably not right now, unfortunately) I will have to see how well this plugs into BizTalk (if at all), but I suspect it’ll work well; find that sample here.

Notes on BizTalk 2009

earlier this week Microsoft announced the release of the beta version of BizTalk 2009.

I’m sure detailed posts of various bits will follow soon, but for now I thought I’d list a few points I’ve picked up (in no particular order)-

BizTalk projects are now “first class citizens” of Visual Studio [2008]; in practice it seems they are really “special” c# projects.

This means quite a lot really, to start with, for the most part they look and feel like c# projects (in the beta build the icon for the project is even the c# icon), but this extends to the property pages, existence of assemblyinfo.cs and the project file structure (which is now pretty much standard msbuild file)

It also means that you could add any c# type to a BizTalk project although in the current build you can’t add a new class using the menu, you are always redirected to the template selector which only shows BizTalk types, so –even if I pick the Add new class option

I get the following dialog -

Not sure when this will be fixed, but the workaround is simply (yet annoying) – create the type elsewhere and use the “Add Existing Item” option.

This will allow you, for example, to include any helper classes you use from your orchestration with the process itself.

Another point to notice is that this is only true for C# types. you cannot add VB code files to a BizTalk project.

 

Also – as the project is now a standard msbuild project integration with TFS is tighter.

interesting to see in the installation process the following entry -

Basically installing this on a computer that has msbuild allows you to build BizTalk projects without having BizTalk or VisualStudio installed; using a build server for BizTalk is so much easier now!

 

The build process is completely incremental – each BizTalk artifact is a c# class, and so when building a project only those classes that have changed will be built, which should shorten the time it takes to build a BizTalk project.

Another fantastic addition is more “HAT” features in the admin console, but as Randal already blogged about this one in details I’ll point you there

 

Upgrading seems to be a breeze and both 2006 and 2006 R2 are supported. this can be done by opening any existing BizTalk project in VS 2009 and the migration wizard will pick it up and migrate it for you. as expected.

you can open a solution with multiple projects and they will all be upgraded and you can use devenv /upgrade with both projects and solutions to do that from script?

 

BizTalk 2009 supports debugging maps – this is done by selecting the “Debug Map” from any BTM file context menu -

This would use the test input instance and test output instance properties, and would basically open the XSL generated for this map and break at the root template

Worth noting that currently this does not support maps with multiple inputs or ones using extension objects.

 

I’m sure there’s a lot more to discover, but I have a 4 day old baby at home – so back to those nappies for me I’m afraid.