More on using the SOAP adapter without a web reference

Every now and then I am drawn to implement a call to a web service using the SOAP adapter and without adding a web reference.

I wrote about it in the past here, as did others, for example consider Richard Seroter, Saravana Kumar and Jon Fancy’s articles on the subject.

Last time I had to do this, though, a couple of months ago, I bumped into a puzzling error, which took some time to understand.

In the scenario I had I needed to modify the message in the pipeline, and - being the good citizen that I am (?) - my pipeline component created a new message and did not return the same message it had received.

When my message hit the adapter it got suspended with the error -

Failed to load "" type.

Please verify the fully-qualified type name is valid.

Details: "".

The type must derive from System.Web.Services.Protocols.SoapHttpClientProtocol.

The type must have the attribute System.Web.Services.WebServiceBindingAttribute.

In my scenario the proxy was provided for the SOAP adapter via its configuration in the send port explicitly as per Richard’s post, so I was very confused about the empty type definitions in the error message.

Some investigation revealed the cause – the send port configuration is read at the beginning of the pipeline and these settings are set in the outgoing message’s context.

My pipeline component created a new message and did not copy these properties to the new message, so when it arrived at the adapter, which tries to read them, the empty definitions were found.

Calling a service without adding a reference in BizTalk 2006 and BizTalk 2006 R2

We’ve been experimenting with calling ASMX web services from orchestrations without having to add a web reference (for the SOAP adapter) or use the generated items (for the R2 WCF adapter).

The idea, in short, is to achieve increased decoupling between systems even in a web service scenario -

Generally when you add a reference to a service in BizTalk 2006 or in R2 (although there are some clear differences between the implementation) the schemas for the request and response types are generated for you as well as an orchestration which defines message and port types using those schemas.

When using the SOAP adapter the types generated are somewhat “special” and they encapsulate a little bit of black magic; luckily the WCF adapter which shipped with R2 is much better in the sense that there’s nothing special about any of these artifacts (which also explains why it is now “Add Generated Items” and not “Add Service Reference” – as this is all it’s doing).

What this means is that if you follow the path that BizTalk leads you through you will get all these artifacts in the same assembly with your orchestration, which means you are now tightly coupled with the web service contract; not the end of the world, but if you want to stay true to the idea behind BizTalk - in which your processes can be masked from changes in the other applications you have to play pretend a little bit.

We thought that if we had the web service schemas in a separate assembly, and our process only used it’s own representation of the data (which would, ideally, be less than the entire data provided by the, mostly generic, web service) we could then map between the two in the port rather than in the orchestration, which would mean that if the web service changes, all we will need to do (in theory, at least) is re-deploy the assembly with service’s schemas assembly the and the map.

 

So – how I went about doing that with the WCF adapter -

Following best practice I had an assembly to hold all of “My” schemas – these are the ones describing entities in my domain.

I then created an orchestration assembly to contain my orchestration, which references the schemas assembly; the orchestration assembly has no other dependencies.

I then created a third assembly to include all the types for the service - I went through the “Add Generated Items” wizard to get all the artifacts, but I only really used the schemas (and not the message or port types); this assembly, like the schemas assembly, has no dependencies.

I then progressed to create a fourth assembly to hold the mapping between my schemas and the service’s schemas; naturally this assembly references both projects, but, crucially, it is referenced by no-one.

So – at the end of this we get the following -

 

I then imported the send port bindings generated by the wizard to create the send port; I could have quite happily created it from scratch as there’s nothing special in that port - with the exception of one point, discussed next - so this was really just to save me some time, and added the two maps I’ve created to map the process output format to the service request and the service response to the process input format.

Goal achieved – the process knows nothing about the service – all is done externally to the process through port configuration.

But did it work? Almost - running this scenario I received a soap fault from the service complaining about a misunderstood soap action; makes sense I thought – how would BizTalk know which service operation I wanted?

Well, the WCF adapter has a very nice way to figure out the soap action to use (in my view) – as part of the port configuration there’s a bit of xml that provides mapping between an orchestration send port’s operation name and the required soap action; the setting looks something like this -

 

In the generated port type the operation name matches the operation name in the service description (“HelloWorld”, in my example), which, in turn, is mapped through this xml to the relevant soap action; as I did not use the generated types the operation name did not match – I simply left it as the default “Operation_1” (naturally…); that meant that when the request came the adapter failed to find a matching operation.

Somewhat annoyingly, what the adapter does when it can’t resolve the name is to assume that the entire setting should be used as the soap action and so the entire xml was written to the header;  this behaviour is there to allow one to specify a fixed header to use, but I think the experience could be a bit better there – they could have had two different settings, or at least realise that if I’ve put a BtsActionMapping xml in there I do not intend for it to be used as the header itself(!), and so, if the relevant entry was not found the request should be suspended rather than going out incorrectly to the service; never-the-less the operation could not be resolved, of course, and the service returned a soap fault.

Fixing the issue was easy and simply meant adding the correct entry in the xml and running the scenario again, this time it completed successfully.

 

How does that differ using the SOAP adapter?

Using the SOAP adapter the approach was naturally very similar; pretty much the same assemblies, pretty much the same artifacts; there are three key differences though -

For starters the soap adapter requires a proxy; in most scenarios you’re using a web port type which provides the adapter with a proxy and so in most cases you don’t have to worry about this at all; I can imagine that some are probably not even aware but the send port, using the SOAP adapter, will have the web service proxy set in the “Web Service” tab of the adapter configuration to “Orchestration Web Port”.

Alternatively you can provide a custom proxy class, which is a topic by itself (and you can check it out in Richard Seroter’s post on the topic here), but in most standard cases this is not required.

As I’m not following the “standard” approach I had to create a custom proxy for my send port; I did this by using WSDL.exe and configuring the proxy class in the send port as described in Richard’s post.

In my case, however, unlike Richard’s, I did not wish to pre-defined the method called in the send port; luckily – the configuration allows you to set it to “Specify Later”, which means the method name will be provided per request through the message context (using the SOAP.MethodName property).

Taking the “Specify Later” approach means I don’t have to have a send port per method, which is good of course, but pay attention to my note regarding number of ports in the summary below.

Now that I have the send port and proxy configuration sorted I needed to get the web service’s schemas; I could do that by using XSD.exe and add the output to my service types assembly.

Last thing – when using the soap adapter you don’t generally need to have an XmlDisassembler in the pipeline; however – if you want BizTalk to be able to run a map it needs a “proper” message type in the context, not that awkward one the SOAP adapter puts, and so the XmlDisassembler becomes mandatory in this scenario.

other than that everything else is pretty much the same.

 

So – to summarise –

Calling a service from a process, without the process knowing ANYTHING about the service implementation is very easy, the story is slightly better in the WCF adapter case in my view, but both seem to me quite reasonable.

The only downside to this approach that I could think of so far, is that you are likely to end up with as many send ports as you have response output formats -

As far as requests from the orchestrations to the web service are concerned BizTalk will quite happily pick up the right map from a list of configured maps based on the input; so - if process A has one output format and Process B has a different output format and they both share the same send port – BizTalk will pick up the relevant map to convert either outputs to the service’s request.

On the way back, however, the incoming message (the service’s response) always looks the same, and so BizTalk will have no way of knowing which map to pick from the list.

That means that multiple send ports will have to be created for such cases so that there’s only one map for the service’s response; a large number of those may have some impact on the overall performance of the server group as the number of subscriptions that need to be evaluated increases; what “large” means in this context and how big is the impact is not something I could say easily, so I’d suggest doing some benchmarking to find out in your environment if you are concerned.

Another dive into the SOAP adapter's behaviour

Being keen to fine tune our solution's performance we turned to look into the message exchanges between the SOAP adapter and one of the web services we're usign quite extensively.

Two things became obvious very quickly (well, to be fair - they may have not been without some very useful tips from our colleagues working for the 3rd party providing that service) -

• All request messages we've sent specify the "Expect: 100-continue" HTTP header
• Neither of the request set out use pre-authentication, although we explicitly set credentials on the send port

so we promptly turned to look into them -

100-continue

I'm no network expert; I hold only the basic understanding of HTTP you'd expect from someone dealing with messaging and integration; the way I understand it then is that if a requester sets the "Expect:100-continue" HTTP header on a request, the client would sends the HTTP headers only to the server and then wait to receive a response with "100-continue" status.

It is only when the "100-continue" status is returned from the server that the client sends the actual request.

I also understand that, in order to support older implementations of HTTP (or those that do not support 100-continue), if the client does not receive 100-continue it will send the request anyway, but only after the delay introduced while waiting for the server to respond.

What all that means is about 200ms delay to every message sent from BizTalk.

I'm sure Microsoft have given a lot of consideration to this before deciding this would be the default behaviour of web services in .net but to us, communicating with a long-term partner over a good leased line, it would have been nice to be able to turn it off and skip the extra step.

Unfortunately, although spending some time on this we could not, as of now, find a way to do so.

Our best bet was (and still is) to set the relevant setting in the ServicePointManager in our custom web service proxy, but this does not seem to have the desired effect.

From a bit of reading we've done it seems there's some element of re-use here and that the ServicePointManager may use an existing service point, in which case our newly set property will not be used.

It could be that this is created somewhere in the adapter before our custom proxy or something like that, we truly don't know at this point.

Pre-Authenticate

Fortunately this is an area we’ve had better success in, and it's good because it has a significantly bigger impact on performance than the 100-continue problem.

The web service I've mentioned before uses basic authentication on the wire and then further elements of security in the message, so on the BizTalk port we've set the credentials the SOAP adapter should use when calling this web service, and, as you'd expect, this is working just fine.

However, analysing the traffic going out of BizTalk on the wire we've noticed that for each message we send to the 3rd party, the SOAP adapter would transmit the message out without the security HTTP headers first, receive an HTTP 401 (Unauthorised) error and only then send the request with the HTTP headers baring the credentials we've set in the send port.

Obviously, and especially when relatively large messages are involved, this has a significant impact on the performance of the server (and the farm as a whole as more bandwidth is used for the message exchange).

Again, doing some reading and experimenting, we've learnt that this is pretty much down to the choices made around the DEFAULT behaviour of .net, and again, we thought, we had no control over it.

But, while we could not do anything if we were using the standard configuration of the SOAP adapter, the fact that were using a custom web proxy to call the web service (for other reasons) meant we could try and tweak the way messages were transmitted, and tweaking we did.

It didn't help us in the 100-continue case, but it definitely helped with the pre-authentication requirement - all we had to do is add a line setting the PreAuthenticate property of the HttpWebRequest to true before using it and BizTalk (well, the .net framework web service stack it uses) would quite happily provide the credentials up front avoiding the need to duplicate the round trip to the 3rd party server.

Much better now!

Unnecessary limitation when using custom proxy for web services in BizTalk 2006

We're using a third party web service quite extensively; sometimes the size of messages exchanged between us can be quite big, which is not ideal from a performance point of view.

Luckily - they are quire advanced and mature and have a good support for compressed messages exchange.

To use that we needed to compress the stream going out over the wire through the adapter and set the Content-Encoding HTTP header to "gzip" to indicate the stream is compressed, on their end, if the header is set that way, they would decompress the stream before processing the message , and will compress the response in a similar way.

They were even kind enough to provide us with a class that enhances System.Web.Services.Protocols.SoapHttpClientProtocol (through inheritance) to do all of that, so really - all we needed to do to use their web service in our playground windows application was to change the proxy class generated by Visual Studio when we added the web reference to inherit from their class rather than SoapHttpClientProtocol (their class would ultimately ingerit from SoapHttpClientProtocol).

We've played with it for a while and it worked great, which meant we were ready to move it from our playground application to our BizTalk implementation; As you can guess by the fact I'm bothering with this post - it didn't go smooth…

I've played with custom web service proxies before - generated a proxy from a WSDL and configured the port to use it - so I knew that in general principle that works quite nicely;

And so I went on and generated a proxy class for that web service and changed the inheritance to the class they've provided.

When I tried to select it in the Admin Console as the proxy class to use, I could browse to the assembly I've built but, once selected, I would get the following error:



This did not make sense to me - the proxy I generated inherits from their class which, as I have the source code, I'm sure inherits from SoapHttpClientProtocol. This error did not make sense to me.

Trying to figure this one out I went on to create my own class ("TestClass") that inherits from SoapHttpClientProtocol (and had not other implementation otherwise) and changed the generated proxy to inherit from that - so basically I created the same scenario - proxy inherits from class inherits from SoapHttpClientProtocol.

When I tried to select my updated assembly in the admin console I could not select the original proxy (now inheriting from TestClass), I could, however, select TestClass itself (which inherits directly from SoapHttpClientProtocol ) but only to get another error -



This error was quickly solved by adding a public web method to TestClass; now TestClass could be selected in the UI, my proxy class, however, the one I really needed, that does not inherit directly from SoapHttpClientProtocol, could not be selected.

It has also solved the mistery, or so I I believe - it seems the UI will only display classes whose base type is SoapHttpClientProtocol, when it should havem in my view, displayed all classes whose IsSubclassOf(typeof(SoapHttpClientProtocol)) is true.

For us the solution was simple - we took all the code we were kindly given from the 3rd party and merged it into our proxy class which could then be left to inherit directly from SoapHttpClientProtocol . The problem is that this is quite limiting - there's no real scope for re-use and for no good reason.

In fact - I suspect this is just a design time problem, of the kind we spotted quite often recently where the runtime would actually work with something, but you have no way of configuring BizTalk to use it.