Yossi Dahan [BizTalk]

BizTalk Admin Console

noreply@blogger.com (Yossi Dahan) — Sat, 20 Feb 2010 17:01:00 +0000

Anything you missing there? anything you don’t like?

I’m trying to compile a list of suggestions for it, put a comment or drop me a line….

More on using the SOAP adapter without a web reference

noreply@blogger.com (Yossi Dahan) — Thu, 18 Feb 2010 13:14:00 +0000

Every now and then I am drawn to implement a call to a web service using the SOAP adapter and without adding a web reference.

I wrote about it in the past here, as did others, for example consider Richard Seroter, Saravana Kumar and Jon Fancy’s articles on the subject.

Last time I had to do this, though, a couple of months ago, I bumped into a puzzling error, which took some time to understand.

In the scenario I had I needed to modify the message in the pipeline, and - being the good citizen that I am (?) - my pipeline component created a new message and did not return the same message it had received.

When my message hit the adapter it got suspended with the error -

Failed to load "" type.

Please verify the fully-qualified type name is valid.

Details: "".

The type must derive from System.Web.Services.Protocols.SoapHttpClientProtocol.

The type must have the attribute System.Web.Services.WebServiceBindingAttribute.

In my scenario the proxy was provided for the SOAP adapter via its configuration in the send port explicitly as per Richard’s post, so I was very confused about the empty type definitions in the error message.

Some investigation revealed the cause – the send port configuration is read at the beginning of the pipeline and these settings are set in the outgoing message’s context.

My pipeline component created a new message and did not copy these properties to the new message, so when it arrived at the adapter, which tries to read them, the empty definitions were found.

On MSIs and Default Bindings

noreply@blogger.com (Yossi Dahan) — Sat, 13 Feb 2010 07:43:00 +0000

Some observations (I’ll you to draw your own conclusions, for a change) -

When you import an MSI, any default binding in it will always be imported, then any binding matching the environment selected in the import MSI wizard (if selected)
If the second binding contains any settings that were already created as part of the first binding, they will be overwritten
When you export an MSI, a binding file is created with the existing configuration as the default binding, even if you did not add one as a resource explicitly. This binding will be imported first, before any other binding.
In the case of receive locations – if two bindings in the MSI contain receive locations for the same port each marked as primary, the last one imported will be the primary receive location.

Why am I writing this?

We had an odd case (which shouldn’t really happen) where we created an application with a particular receive location.
We have then added a binding file for another environment in which we have misspelled the receive location name (the URI was the same)

When we tried to import the MSI to the second environment, selecting the environment name in the import MSI wizard we received an error that looked something like this -

(for the search engines: “Cannot update receive location”, “Address should be unique for Receive Location”)

The reason was that the wizard imported the MSI and initially applied the default binding creating the receive location with the first name. It then went on to import the second, environment specific, binding file, now trying to add a new receive location with the different name but failing as the URIs are the same. had the name been the same the receive location would have been overwritten, had the URI been different a second one would have been successfully added, but given this weird combination the error was produced.

Developing Streaming Pipeline Components

noreply@blogger.com (Yossi Dahan) — Sun, 07 Feb 2010 06:48:00 +0000

A couple of months ago I published this post suggesting that pipeline component developers should look to design those in a streaming fashion which avoids loading the message into memory.

Quite a few people contacted me asking for an example so eventually I sat down to create one.

Creating the sample warranted writing down an explanation for it, which turned out rather long and (hopefully) elaborate, so i turned that into an article and, I’m proud to say, has now been published on MSDN.

In the article I wanted I wanted to demonstrate a realistic case, which doesn’t necessarily take the “easy path” and yet is clear enough to demonstrate the principles without overwhelming the reader; primarily I felt it was important to show a sample that modifies the stream (and not just using information from it) as these tend to be more complex.

Hopefully I have managed to provide a good demonstration of the principle. Let me know…

I guess I could consider this a compliment, but I’m not happy!

noreply@blogger.com (Yossi Dahan) — Wed, 03 Feb 2010 17:55:00 +0000

A few months ago I found out that a chap called Jason Hyland, who is another BizTalk consultant here in the UK, has nicked the contents of my web site to his own. word for word.

Despite the hidden compliment (I guess it means I write reasonably well) I was not very happy and so I thought I’d drop Jason a polite note.

I never received a response for my email, but several days later his web site was down, so I assumed he accepted this wasn’t nice and is working on his own wording. I guess I was wrong as today I was reminded of that and decided to check what he has done, and what do you know - http://www.needlecast.co.uk/services still uses, word for word, the wording from my web site.

How bad is that?!

Jason – at the off chance that you’re reading this – here’s some free publicity on my blog (currently averaging around 160 unique visitors per day) – write your own stuff, I’ve done it, so it can’t be that difficult, and it will be the decent thing to do.

Yossi

Wrong impressions with MaxConnection

noreply@blogger.com (Yossi Dahan) — Wed, 03 Feb 2010 17:12:00 +0000

Recently we’ve done some performance tuning on aspects of our solution, and as part of that we’ve looked at configuring the system.net maxCnnections setting (see Darren Jeffords post on this subject) to increase the number of concurrent calls our application can make to various services it uses.

We’ve looked at this for our BizTalk solution but also for a complex, multi-threaded smart client we’re building.

Initially some have been somewhat sceptical about the effects of this, despite all the information on the web, but when I tried to figure out why the reason became apparent – their observations did not match the expectation – no matter they set in the MaxConnections setting, the amount of time it took, and the number of concurrent calls to the external service remained the same.

Trying to figure out why, we’ve used two scenarios – in one we ran our smart client application and monitored using Fiddler we examined how many concurrent requests are sent to the service; on the other we ran a test application we created specifically for this purpose. Confusingly there were two separate reasons for why the observations did not match our expectations -

In the test scenario, the application and the service we used to test were all done on the local machine. Of course in this case the request to the service does not actually go through the network, and so the system.net MaxConnections setting does not play a part; as soon as we’ve moved the service to another machine the results were as expected.

The smart client application, however, was already pointing at a service on a remote machine, and so there must have been something else there; as timing was a little bit more tricky in this case the guys used Fiddler to look into the requests made where they saw all the requests were getting sent out at the same time, despite setting the MaxConnections to 1.

It turns out, and that I did not know, that Fiddler was the cause of the unexpected behaviour.

I don’t know exactly how Fiddler works, but I suspect it hijacks the requests on the local machine so that from our client point of view the requests never actually went through the network and so the MaxConnections setting has been ignored. Fiddler itself does not adhere to this (I’m guessing it is not a .net application) and so multiple concurrent requests have been forwarded by it to the remote service.

As soon as we’ve removed Fiddler the behaviour was as expected.

Lesson learnt.

What I’ve learnt about BizTalk Hosts

noreply@blogger.com (Yossi Dahan) — Fri, 22 Jan 2010 16:22:00 +0000

Here’s something I learnt from Guru Venkataraman on my one of my visits to Redmond last year -

Like most seasoned BizTalk developers, I suspect, I usually follow the best practice around host planning nicely summarised by Marcel here.

What I did not think about (and nor did Marcel, it appears) is the actual queues behind these hosts, how they are used, and the impact on performance; in comes Guru’s wisdom-

Each host has a queue, implemented as a table in the message box database, so - loosely speaking – when a message is published to the message box, BizTalk determines which subscribers are interested in the message, and places the message (well – logically) in the queues of the relevant hosts.

Each hosts polls its own queue periodically (default is 500ms) to get any pending messages and acts on them – start a new instance or correlate to an existing instance, potentially rehydrating it.

Now imagine you have one host running two different processes and that two messages are received both of which should trigger a new instance of process A.

The host picks up the first message (for this discussion to be clear enough we’ll have to pretend it’s working on a single thread, although I suspect several aspect of this are) and passes it to a new instance of process A.

Imagine now that process A publishes a message intended for process B; as process B runs under the same host, the message gets into the very same queue in which we already have the second message queued for process A.

Now – process B cannot start until the host reads the pending message and starts the second instance of process A. it is only at this point that the message intended for process B is at the ‘front’ of the queue and can be read by the host and an instance of process B can be started. effectively the execution of process B has been delayed unnecessarily because there were messages queued for process A.

Now – of course BizTalk much of this is indeed multi threaded and so the problem is not that severe. BizTalk was clearly designed to host many processes under any single host and there’s definitely not a requirement (or a recommendation) to have a host per-process, that would be ridiculous, however -

Where you have a flow where process A often publishes messages that end up in another process (B, C or D to make some names), and you have a reasonable throughput, you would get better performance if the publisher of the messages (process A) is hosted separately from potential subscribers (B, C or D).

With the subscribers configured with a different host they can pick up their messages without being affected by other messages awaiting for process A, and vice versa – the same goes for any replies published by them.

Two things worth remembering in this context:

When you ‘call’ an orchestration, as opposed to ‘start’-ing an orchestration, the host of the calling orchestration is used, regardless of what’s configured in the admin console
Whilst I’ve specifically mentioned processes, the same considerations apply to send ports, and even receive locations, and when looking at the former it is worth remembering that dynamic send ports, where used, always use the default host for the adapter used.

What not to do – hard coded values

noreply@blogger.com (Yossi Dahan) — Thu, 21 Jan 2010 17:46:00 +0000

This always seemed like an obvious one to me, but as I do see it out there occasionally, I guess it should be mentioned in my little ‘series’ – think carefully whenever you are hard coding any data into your code, as it is almost guaranteed to change one day, however remote you think that chance is.

This post could probably end here, but that would be almost rude, so I’ll give an example -

A custom disassembler is being developed to identify a message type before parsing it.

The messages are expected to be received over POP3, and the decision which message it is is done by identifying the sender of the email (rather than by looking in the actual message), which is fair enough.

So – the component has code to read the context property storing the sender’s email address and then a switch statement on the property’s value determines the message type required.

Of course this works very well. until a third party decides to change the email address, or another one is added. now you have to update the code, re-deploy and re-test. why? because you did not want to spend another day moving this to configuration?

Even a simple AppSetting would be better than hard coded, but of course usually you’d be looking at least at defining a custom configuration section if not using a database or other store (SSO?)

..and it looks like it’s not

noreply@blogger.com (Yossi Dahan) — Mon, 04 Jan 2010 08:52:00 +0000

Back in December I’ve posted ‘This should not be a routing failure’ about an experience we’ve had on our BizTalk 2006 environment that I found very weird.

Tim Wieman, a senior program manager on the BizTalk Customer Advisory Team, was kind enough to drop me an email letting me know that this behaviour was indeed identified as incorrect, and was actually fixed.

A hotifx was created for BizTalk 2006 which later versions already incorporate, so if you’re using 2006, and you’re having this issue - get that hotfix, if you’re using a later version – you probably don’t need to worry about it.

I have managed to test the hotfix on BizTalk 2006 and it does allow my scenario to run without a problem (don’t forget the registry change for each host, as described in the link to the hotfix); I haven’t been able to try this on later versions yet, but there’s no reason to think that would not work.

Thanks Tim!

UK SOA/BPM User Group meetings in January

noreply@blogger.com (Yossi Dahan) — Sat, 26 Dec 2009 18:27:00 +0000

The next ‘in-person’ meeting for the UK SOA/BPM User Group will be held on the 27th of Jan in Paddington – London, check out the details here, and if you’re coming – be sure to come and say hi – I’ll probably be the loudest chap in the room. again.

And even before that, on January 12th, an online meeting will be held with the title ‘Developing Composited Solutions with Sharepoint and SAP’

This should not be a routing failure!

noreply@blogger.com (Yossi Dahan) — Fri, 18 Dec 2009 11:18:00 +0000

Important update: it looks like it’s not after all! and I’ve posted the details here; I suggest you check it out after reading this post.

Some routing failures are straight forward – you have a message, it has context, and no one subscribes to it. Routing Failure Report notifications are very useful on helping troubleshooting these – and most likely someone has mistyped a property’s value, a subscription value or forgot to ensure that the relevant properties are promoted and that relevant services are enlisted.

Some routing failures are a little bit more tricky to map out -

Consider a simple scenario where you have a two way receive port publishing a request, which is being picked up by a two way send port -

Now add a second two-way subscriber to the mix…

In this case BizTalk cannot know which of the two potential responses it may get (from the two different two way send ports) it should deliver to the [send side of the] receive port, and so it opts to not even publish the request and instead deliver a routing failure.

This is fair enough - race conditions are not a good thing and it’s pretty clever that BizTalk identifies this up front and prevents this.

However - the same routing failure will happen, it appears, even if the publisher is NOT a two way port, and that – I did not expect.

If the publisher of the original message (the receive port) was not a two way port, but one-way, it would not expect any response from anybody, which removes the race-condition situation, and therefore should have been supported in my view, but it does not appear that it does.

Imagine a scenario like this -

1st Process publishes message A, through a one way port
1st Send port (2-way) picks up message A and calls a service, publishing the service’s response - Message B.
2nd Send port (also 2-way) picks up message A as well and calls a different service, publishing this service’s response - Message C.

As far as I can tell, this is a valid scenario – call it message enrichment via external services.

Subscribers should be able to subscribe to message A, B or C as they wish, and all is nice and loosely coupled, if you only ignore the fact that this does not seem to work!

Building such a scenario I'm getting a routing failure for message A, which - unless I'm not seeing correctly - I can only explain according to the rule mentioned above, which I find really frustrating.

The Microsoft Application Platform Event - February 9th 2010

noreply@blogger.com (Yossi Dahan) — Sun, 13 Dec 2009 18:20:00 +0000

Microsoft UK have published the details of this event taking place in the MS Campus in Reading, see the details and register here

The event has a very promising agenda -

I had the pleasure of meeting Robert Hogg several times through the MVP program, and he is probably one of the most suitable guys I know to be talking about pragmatic approach to projects with a track record of successful “rescue projects” under his belt.

I also have the pleasure (although I’m not sure it’s mutual) to be working closely with Paul Saggar and Jon Simpson, and if anybody can talk about implementing complex business processes in an ever changing, and highly competitive, environment it’s these two.

And of course, as many know, I work closely with PNMsoft, and so its only natural that I’d say it’s always very interesting to hear from their vast experience in the BPM field in which they’ve been working for many years.

See you there!

What not to do – avoid reading entire message’s to memory unnecessarily

noreply@blogger.com (Yossi Dahan) — Fri, 11 Dec 2009 14:31:00 +0000

This one is fairly common, I suspect, and I can certainly see why – the temptation is simply to big – but too many pipeline components start by reading the message into memory, when, with a little bit more effort this could have been avoided.

One pipeline component I’ve seen, for example, receives a flat file, and needs to remove records already processed (duplication elimination) – quite a good thing to do in a pipeline, and I also liked the approach of doing so before the disassembler, to make the xml produced smaller.

V1 of the component used a memory stream – the incoming stream was read line by line, each line was assessed, and – if was not a duplicate – it would get written to the memory stream.

When the component had finished going through the entire incoming stream, the memory stream would be assigned to the message, replacing the original stream, and the message would get returned to the pipeline.

There are two downsides to this approach – the first is memory consumption – the component will always consume at least as much memory as the size of the (outgoing) message; done properly BizTalk would then clean this memory, but only after completing the processing of the message; the second downside is potentially unnecessary delay in further processing of the message – one of the huge benefits of the pipeline, in my view, is its streaming fashion, where subsequent components, if developed in the correct manner, can start working on parts of the message before preceding components completed their processing; basically each component passes back to the pipeline the portion of the message it already processed, whilst working on the next portion.

It appears that the customer in question encounter memory issues as the component’s code was changed to use virtual stream instead of memory stream; a virtual stream is effectively a stream that uses disk for storage instead of memory.

This solves the memory consumption issue, but merely replaces it with IO operations which may have an even bigger impact on the server’s overall performance (and does not address the processing delay point at all).

What would have been the correct way to implement this in my view?

The component should have create a custom stream, wrapping the original stream from the message; It would then replace the message’s stream with the custom stream immediately returning the message back to the pipeline. Note that so far the component hadn’t touched the message stream – zero bytes have been read.

As BizTalk (and not the component!) would read the message (for instance when persisting it to the message box), the custom stream’s read function would be called which would contain that reads the underlying stream (the original stream received by the component), probably buffering reads until the end of a line for simplicity (although in many cases this is not necessary) and assessing whether the record is a duplicate or not; if it is a duplicate the function will simply read the next line and so on until a non-duplicate record is found, at which point the line would be returned as the output byte stream from the read function.

This effectively means that the next component, or the message box, will receive the message line by line, duplicate records removed, without having to wait for the component to process the entire message, and with only a maximum of one line ever loaded into memory.

LOB Adapter SDK

noreply@blogger.com (Yossi Dahan) — Fri, 04 Dec 2009 08:56:00 +0000

I’ve been doing some work recently with the LOB adapter SDK.

The adapter I’ve been creating is effectively a wrapper around a web service we’re using extensively, to which we often need to make several consecutive calls.

Doing so from an orchestration, using a send port, is inefficient, as each request involves a pub/sub cycle; on the other hand – I’m not a big fan of ‘inline sends’, which was one of the options considered.

Building a custom adapter would give us the best of both worlds, if you like – the process would create a message containing all the requests to be sent and publish it; a send port, configured with the custom adapter, would pick this message and deliver it to the adapter which would, in turn, debatch the requests, send them to the service, aggregate the responses back and return the aggregated message.

This way we still have all the tracking and management BizTalk provides but only one pub/sub cycle.

The LOB adpater SDK does a very good job in lowering the barrier of creating adapters (I’ve created ‘native’ adapters before) which is a good thing.

Several points I’ve been left with from this, very initial, exercise -

Implementing the metadata browser, resolver and search handler was a bit tedious; I ended up creating an xml file with my metadata, embedded this in my project and wrote code to generate the metadata off that. I wish this was part of the SDK.

Once the metadata was set I needed to write the outbound handler (in my case), for which I needed to know how the incoming messages and outgoing responses should look like; with no generated samples this was a bit too painful to my liking – figuring out the request format is easy although tedious again – I’ve ended up consuming my adapter from a client, making a call, setting a breakpoint and going over the message at runtime.

This is, of course, not possible for the response – where I need to create the message in the adapter, so trial and error was the only way (ok – and reading the docs, and applying some common sense) – in my case I actually made a mistake in my metadata configuration which took me a while to figure out (and Rupert’s help, thanks!)

Bottom line is that if there was a tool to look at the metadata and generate request/response samples for each operation this work would have been much easier. I might be tempted to create one if I get the time.

The last thing that phased me was working with metadata from the client.

I needed the client to provide some information outside the message; I knew the mechanism in BizTalk would be to use a context property, for which I will need to deploy, with the adapter itself, the property schema, but I wasn’t sure if and how the adapter gets access to that or how I could consume this from a .net client.

Figuring out the adapter access was easy enough, turns out all the BizTalk message’s context properties are added as properties to the request WCF message received from the framework, so

token = message.Properties[PROPERTY].ToString();

where PROPERTY is the context property’s namespace#node would return the value of the property requested.

I was a bit stumbled with how to provide this property outside BizTalk, this time Manas Garg came to the rescue in this thread

Turns out you can simply do -

ConcurrentPrograms_ARClient client1 = new ConcurrentPrograms_ARClient(binding, address);

using (new OperationContextScope(client1.InnerChannel))

{

OperationContext.Current.OutgoingMessageProperties.Add("Property Name", "Property Value");

client1.OPERATION(params...);

}

What not to do – ‘bundle’ unrelated components together’

noreply@blogger.com (Yossi Dahan) — Tue, 01 Dec 2009 08:38:00 +0000

So – here’s another real-world they shouldn’t have done this –really! item; somewhat related to my previous post about projects, but a different slant -

MyP had used pipeline components fairly extensively, a very good idea in my view, especially in this case.

But, as discussed in that previous post all pipeline components, regardless of where they were used, were bundled together in a single assembly.

To make matters worse, a single pipeline component often served more than one purpose, for more than one client.

So – for example – a pipeline initially created to remove an unwanted trailer from a message from a particular sender, ended up also converting the message to xml, and then extended to support another format, from another sender, only that the two don’t share any code – the execute method of the pipeline component has a switch statement on the sender name, and runs two separate functions.

Now – considering all the components are in the same assembly already, how can this make matters worse?

Well – single responsibility principle is one that I generally like – I’m a new developer working on this project, I see a component called TrailerRemover, used in a pipeline called <someSender>_Receive I assume this is processing messages from <someSender> and that it removes a trailer.

I eventually discover it does a lot more, and processes messages from another sender as well.

One of the side effects of this is time wasting – it is much more difficult, in my experience, to maintain systems that don’t follow the single responsibility principle.

This is aggrevated by the fact that this is usually a symptom, if not a cause, for bad architecture – I shouldn’t be able to mix logic for two different senders, not unless specified common logic is factored out and reused properly, from a shared assembly.

Now when I come to change some code I find it difficult to know what the impact may be – where exactly is this thing used?

What not to do – projects

noreply@blogger.com (Yossi Dahan) — Tue, 24 Nov 2009 20:17:00 +0000

So – what not to do?

Here’s one – don’t take 15 different interfaces, from 15 different providers, and the canonical format, and your internal process and bundle it into the same set of projects.

My predecessor (MyP in short) in this project had followed the best practice and ensured (for the most part) that schemas, maps, pipelines and processes each have their own project.

This is good – to start with, in the old days, mixing them used to cause all sorts of build issues, but – although I haven’t tried in several years now – I assume that’s all in the past, but – more importantly – this is bad practice because different artifacts have different ‘resilience to change’ – if you need to add a small shape to a small process that gets instantiated as a result of a receive shape (and not being ‘started’ or ‘called’) you can usually un-deploy it fairly easily.

Change a schema, on the other hand, and usually there’s a whole raft of artifacts you now have to un-deploy with it.

For that reason – mixing two artifact types in the same assembly, whilst not technically problematic, usually suggests you will have maintenance nightmare in the near future (unless you don’t mind down time, and regression tests, that is).

Anyway, as I was saying, MyP did separate the different artifact types to different assemblies, but equally MyP only had one assembly of each; so – when we are processing a message arriving from partner A, for which we have a schema, a pipeline, a map or two and an orchestration – these were split across four assemblies; equally - when we are processing a message arriving from partner B, for which we also have a schema, a pipeline, a map or two and an orchestration, all different – these were also split across THE SAME four assemblies.

On the surface not much wrong with this, the problem is what happens when you want to change a small thing in, say the schema for partner B, used from within a map, which in turn gets used within an orchestration – you now have to un-deploy the orchestration assembly, so that you can un-deploy the maps assembly, so that you can un-deploy the schema assembly, so that you can redeploy the new version of the schema assembly followed by the rest. (ok – so I’m assuming versioning and side-by-side deployment are not being used – which I have to – for my story);

So – all that is the general pain that is part of the BizTalk developer’s day, why am I bringing this here? well because you’ve taken a BizTalk ‘challenge’ and doubled it - for one – you took process A down due to a change in process B; why? had they existed in separate assembly sets you wouldn’t need to… and two – as you’ve released new code – you have to test new code; only that now you have to test two sets of code, including one you haven’t intended to change (but may have, by mistake or otherwise); again – had there been two sets of assemblies, you could probably get away with testing just the scenarios related to the ones you’ve changed.

Now multiply this by about 15 partners, and you see how it can be quite wrong. I hope.

What not to do - premier

noreply@blogger.com (Yossi Dahan) — Tue, 24 Nov 2009 17:14:00 +0000

Again I find my self having to apologise for not writing for a while (it’s been one month since my last post, two since my last ‘real’ post), the usual suspects are to blame – too much work, too much bureaucracy, kids, new xbox games….

Work wise I’ve taken on a few more small engagement recently, one of which was to enhance/fix/maintain a small-ish solution someone else built.

I could not resist the temptation to take some notes of all the stuff I would have done differently, and am in the process of compiling them as a proper report for the client.

As I was not posting for a while, partly due to this work, I thought its only fitting that I publish some of these ‘recommendations’ here.

For obvious reasons, I will not name names and will try to generalise any samples/explanations provided; I can’t resist mentioning, though, that this other person, I learnt as I looked up his/her name on the web, had also nicked the entire text off my web site and used it on his/her own, which I found rather annoying (I believe the pages have since been removed, after a polite nudge…)

Anyway….keep in mind that many of my notes may come down to style, or MY best practice; they don’t necessarily mean that the other approach is completely wrong - I don’t presume I know better than anyone else (oh well….) but that there’s value in another point of view – in this case- mine.

So – a few posts coming, likely to be very short and to the point, hopefully someone will findthem useful.

Yossi

Another nice online SBUG meeting

noreply@blogger.com (Yossi Dahan) — Fri, 23 Oct 2009 08:18:00 +0000

This time Global 360 will introduce themselves to the group members, if you’re curious – check the details here

Online SBUG meeting

noreply@blogger.com (Yossi Dahan) — Mon, 12 Oct 2009 14:21:00 +0000

Check out tomorrow’s ONLINE SUBG meeting in which Jean-Pierre Auconie, the engineer behind the famous MessageBoxViewer tool talk us through it; details here

Don’t be scared of custom disassemblers

noreply@blogger.com (Yossi Dahan) — Sun, 27 Sep 2009 09:53:00 +0000

During the last few weeks I’ve been asked to review two separate projects, for two separate companies, developed – naturally – by two separate teams.

The two things both projects had in common were that they both had to deal with legacy “flat files” and they both chose to process these files outside BizTalk using custom code.

In both these cases I completely agree with the decision to use custom code to parse the incoming files – as good as the out-of-the-box flat file support in BizTalk is (made significantly better with the introduction of the flat file wizard, once one gets the hang of using it) – there’s no avoiding from writing custom code to parse flat files every now and then – some file formats are pretty challenging with different records types, conditional records, interleaving segments etc.

I do not agree with, however, the decision to perform this custom parsing outside BizTalk.

I’m pretty sure I would not even bother posting this point had I not seen two of these in the same month, but the fact that I did suggests it may be worth posting a quite note.

One of the projects had the code in a console app, called from a windows scheduled task; the application would pick up files from a folder, parse them, and drop the xml representation in another folder, for BizTalk to consume.

The other had a windows service monitoring a folder and pick up any files, parse them to a different, simplified, flat file format (!), and drop them in another folder for BizTalk to consume.

Both of these introduce another component to the mix; such component needs it’s own error handling, it’s own monitoring, deployment strategy, operations manual etc. similarly it includes a fair bit of re-inventing the wheel – writing code to monitor folders, read files, and write files – stuff that BizTalk is doing out of the box.

What would have been the correct approach then? quite simple – a custom disassembler in the receive pipeline -

Writing a custom disassembler it quite simple - at the end of the day, it boils down to developing a class library, which implements a few simple interfaces, the main one – IDisassemblerComponent defines two methods - Disassmeble and GetNext (the other interface are even simpler, almost insignificant in terms of effort)

Disassemble gets the source message and potentially parses it up front, GetNext is called repeatedly by the pipeline to receive 0 or more parsed messages, until it returns null. simple.

In one of the projects I’ve since taken their existing code (console app), refactored it into a class library, and wrapped it in a custom disassembler class that calls it; converting the scenario to a BizTalk pipeline and performing the key “developer testing” took less than a day.

Why did they not do it to begin with? whilst sometimes there are valid reasons, technical or otherwise, I suspect that in this case it was just unfamiliarity with with BizTalk and some lack of confidence in the development team’s ability to learn and implement (or their belief in themselves); these are valid concerns to any project manager, but I would suggest that a better course of action would have been to spend some time looking at what it takes to implement a custom disassembler, seeing that’s its not at all that scary, and by doing so learning more about a product used in the solution (BizTalk) and achieving a better architecture, and more maintainable approach.

Paolo Salvatori on ways to create messages in orchestration helper classes

noreply@blogger.com (Yossi Dahan) — Sun, 13 Sep 2009 10:23:00 +0000

I’ve been toying with message creation a few times in the past, and recently turned to Paolo for help with a question; Paolo has an amazing blog and he has now posted posted some of his wisdom around ways to create messages from a helper class to an orchestration on his blog; well worth reading (as any entry on this fantastic blob really!)

Loosely Coupled….Part II

noreply@blogger.com (Yossi Dahan) — Fri, 11 Sep 2009 13:33:00 +0000

In a previous post I’ve mentioned our constant attempt to strike the right balance when it comes to loosely coupled services; I’ve mentioned that we were looking at two different scenarios – loosely coupling calls to services outside BizTalk and loosely coupling calls to services inside BizTalk (once implemented within the BizTalk group)

I’ve also mentioned that our solution is composed of a few distinct ‘areas’ (each one generally encapsulated as a BizTalk Application), which we consider, in most cases, to be service boundaries, and – within one ‘flow’ of an incoming request message, we will often have to cross one or more of these boundaries to achieve our end goal.

In most cases, in our solution, the ‘subscriber’ service would use the schema of the ‘publisher’ service for its incoming message; this roughly follows the principle of a service’s proxy, albeit a bit upside down - for practical reasons. Only that - and that’s a much bigger difference - we don’t create a copy of the schema as a service proxy would, but rather reference the schema of the publisher directly (through a shared assembly); this, of course, creates a strong dependency between the two and - over time – this has caused us a lot of headache around deployments as whenever we wanted to update the publisher, we’d have to remove the subscriber too.

Recently we have experimented with following more closely the service proxy approach and instead of using the same referenced schema (using a shared assembly), we’ve created an identical copy of it – same root node and namespace - in the ‘subscriber’ side.

The assumption was that we will be publishing a message using the copy of the schema the publisher holds and be receiving it using the copy of the schema the subscriber holds, but as the message itself will look exactly the same, and will inevitably have exactly the same message type, and so it would be picked up by the subscriber successfully.

Had it worked, we would be have been able to avoid the dependency between the subscriber and the publisher, which would help us gain much needed flexibility in the publisher to support, and change for, multiple subscribers.

Theoretically - if the publisher schema had to change (say – to support functionality required by other subscribers), as long as the change is backwards compatible such as added elements, we could replace the publisher copy of the schema, but leave the subscriber copy as is, until such a point that we need/want to update the subscriber process.

Well - in BizTalk 2006 – this would have worked just fine; unfortunately – from R2 onward, it no longer works – when an orchestration receives a message, it often does so based on a subscription that included the ‘messaging message-type’ (root node and namespace); however – starting with R2 – an additional check has been introduced – to compare the full .net type name of the schema used by the publisher message with the full .net type name of the schema used by the subscriber, assembly, version and all.

This check obviously fails in our scenario, and our fancy loosely coupling solution no longer works (in R2 or 2009).

I think this check is actually a result of code introduced as a hotfix for BizTalk 2004, which – for on reason or another did not make its way into BizTalk 2006 but did into later versions, but I’m not sure, either way – it’s important to note the workaround described at the bottom of the hotfix description, as it appears this behaviour can be turned off, but one would have to check carefully the potential impact.

What else could we do?

Well – one pattern we know that works fairly well is the broker pattern – there’s the publisher, with it’s own schema, there’s the subscriber, with its own – completely different schema, and there’s a broker – a third process that has dependencies on both and contains a map to convert one to the other; on the plus side – this gives us all the flexibility we need – at any one point we only need to deploy two entities – the publisher or the subscriber and the broker, which is good enough; having the process, with a map, allows us to use multi-part messages if we deem them suitable, and all the complexity we need in the mapping; on the down side there are more artifacts to deploy and manage but, more crucially, one extra message box hop which, in a low latency scenario as ours, is not a small price to pay.

Another option is to simply expose the subscriber as a service and call it as such – there are big benefits to doing that – including the fact that we can now have a copy of the schema, in the form of a proxy or without one, and we have also decoupled the services in terms of BizTalk groups –the other service can be anywhere, although this was never a requirements for us; however – we’re paying in more pub/sub again, as well as more IO and quite possibly more complexity.

Theoretically we could have also use XmlDocument (or any other generic wrapper, for that matter) to convey the message, but a) I don’t like typeless intechanges and, b) this does not work well in cases where correlation is required, as the following receive tend to short-cut the subscriber and pick up the request as a response, that is unless you’re willing to introduce two wrappers – one for the request and another one for the response.

Debugging Expression shape code

noreply@blogger.com (Yossi Dahan) — Tue, 08 Sep 2009 07:20:00 +0000

Sandro Pereira has posted a question, and answer, in the BizTalk newsgroup (he also described his answer, in detail, on his blog) about debugging expression code in Visual Studio

He wasn’t referring to debugging code in helper classes, but code in expression and assignment shapes.

My answer was that this was not possible, but Sandro quickly proved me wrong, as he demonstrates in his answer and blog post, and this got me thinking –firstly – despite knowing about the option to use the generated code (and actually using it on very rare cases to understand a certain BizTalk behaviour) I had never thought of using it for debugging, and that is an interesting thought.

However – I had to wonder – how come I never came across the need to? in all those years of BizTalk development, not even once can I remember thinking – oh! I could solve this if only I could debug the piece of code in this shape..

The reason, I think, is two fold -

1. I rarely have more than 2-3 lines of code in an expression shape of any kind; if it’s not straight assignments, its going into a helper method; it’s cleaner, it’s more reusable, and it’s easier to debug.

2. I use trace. a lot. and so every few shapes or so, and certainly in expression shapes, I will have a trace line that outputs to a log file important information about the state, and the flow of the process; this proves to be invaluable when troubleshooting issues on the live environments, but also really helpful in development.

On ASMX, WCF, namespaces and generated schemas (in BizTalk 2006)

noreply@blogger.com (Yossi Dahan) — Mon, 07 Sep 2009 21:18:00 +0000

Recently we’ve started to consume a new version of a web service we’ve been using for a while.
We’ve known that, as a whole, not much had changed, only that they have now moved to WCF; they would have migrated their classes to VS 2008 but would expose pretty much the same functions, using pretty much the same parameters.

Still – it appears that BizTalk now insists on generating multiple schemas for the web reference, and as more of the service is moved across more schemas are introduced.

This caused Oleg a fair amount of pain as, when new schemas would be introduced, they would re-order the existing schemas, so reference1.xsd (in the web reference) would suddenly become reference2.xsd, which in turn break out maps.

The process of finding out the logic behind which schemas are created was fairly short and simple, but as I’ve documented it I thought I might as well share it -

Initial observation revealed that whilst the ASMX services’ WSDL file contains all the schemas needed, the WCF services using import statements in the WSDL file; the schemas exist in separate ‘files’.

The ASMX services always uses the XmlSerializer, WCF services use the DataContractSerializer by default, but can be configured to use the XmlSerializer if required.

Here’s a walk thorugh of the scenarios we’ve compared (using BizTalk 2006) -

Standard WCF project, DataContractFormat
We’ll start by comparing the standard WCF sample project generated when you create a new WCF service application in Visual Studio 2008 -

[ServiceContract]
public interface IService1
{
    [OperationContract]
    string GetData(int value);

    [OperationContract]
    CompositeType GetDataUsingDataContract(CompositeType composite);
}

[DataContract]
public class CompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    [DataMember]
    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    [DataMember]
    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Looking at the WSDL generated, 3 schemas are imported –

1.       The usual generic types

2.       The definition of the compositeType type

3.       The definition of the service’s messages (GetData, GetDataResposne, GetDataUusingDataContarct, GetDataUsing DataContractResponse)

Adding a web reference to this service from a BizTalk 2006 project we can see it represents this fairly accurately -

We can see all 3 schemas downloaded from the service, but within the reference.map generated code a single reference.odx defined the methods in the form of ports and web-messages, and reference.xsd defineds the compositeType schema.

Equivalent project in an ASMX service

I’ve created an equivalent ASMX service, which looks like this –

[WebService(Namespace = "http://tempuri.org/")]
public class Service1 : System.Web.Services.WebService
{

    [WebMethod]
    public CompositeType HelloWorld(CompositeType composite)
    {
        CompositeType response = new CompositeType();
        response.StringValue = "Hello World";
        return response;
    }
}

public class CompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Publishing this service I can see it’s WSDL contains (does not use import, but that proved to be insignificant) a single schema that represents the service’s messages and the compositeType definition.

Consume this service from a BizTalk 2006 project and only the WSDL file is downloaded (there are no ‘external schemas’ to worry about) but within the reference.map pretty much the same odx and xsd files are generated, no real difference between ASMX and WCF here.

Next I’ve looked at changing the serializer the WCF service works with from DataContract to XmlSerializer –

Standard WCF project, XmlSerializerFormatter

Now we will change the serializer to XmlSerializer by adding XmlSerializerFormatAttributre to both the service and the data contracts

   [ServiceContract]
    [XmlSerializerFormat]
    public interface IService1
    {

…and

[DataContract]
  [XmlSerializerFormat]
  public class CompositeType
  {

The WSDL in this case includes only one import, for a single schema representing both the service messages and the compositeType schema (basic types are not exposed) and BizTalk now only has one schema downloaded, but again – the reference.map code remained identical (one ODX, one schema)

How will adding a second namespace affect these behaviours? Lets investigate -

WCF project, two namespaces DataContractFormat

To demonstrate this I’ll add another data contract - AnotherCompositeType, specify an explicit namespace for it and include it as a second parameter to the GetDataUsingDataContract operation -

[DataContract(Namespace="HttpL://SomeNamespace")]
    public class AnotherCompositeType
    {
        bool boolValue = true;
        string stringValue = "Hello ";
 
        [DataMember]
        public bool BoolValue
        {
            get { return boolValue; }
            set { boolValue = value; }
        }
 
        [DataMember]
        public string StringValue
        {
            get { return stringValue; }
            set { stringValue = value; }
        }
    }
 
        [OperationContract]
        CompositeType GetDataUsingDataContract(CompositeType composite, AnotherCompositeType anotherComposite);

Using DataContractFormat again, but with two classes, representing two different namespaces, we’re now getting yet another schema - the fourth one - representing the added data contract (if the namespaces of both data contracts were the same, the DataContractFormat would have included them in the same schema)

On the BizTalk side, the reference.map code now also contains a second schema, one describes the original CompositeType, and a second describes the second type – AnotherCompositeType and here as well – were the two types in the same namespace, a single schema would exist, describing both.

Let’s look at the same again, using the XmlSerializerFormat

WCF project, two namespaces XmlSerializerFormat

Adding the XmlSerializerFormat, I also have to remember to include the XmlRoot attribute to set the namespace, as the serializer does not look at the DataContract attribute -

[DataContract(Namespace = "http://SomeNamesapce")]
[XmlSerializerFormat]
[XmlRoot(Namespace = "http://SomeNamesapce")]
public class AnotherCompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    [DataMember]
    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    [DataMember]
    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

Now the WSDL for this service, using the XmlSerialiserFormat imports two schemas – one for the service messages, and the CompositeType schema, all reside in the same namespace, and a second for AnotherCompositeType which is defined in a separate namespace

Consuming this from BizTalk and again I’m getting two schemas – one for each namespace.

So far – switching between DataContractFormat and XmlSerializerFormat made no difference to the generated code under reference.map, but it did change the way the WSDL is constructed (import vs. embededed schemas) and therefore the downloaded components (wsdl and schemas, vs. wsdl only)

Note - another thing I’ve noticed is that when a new schema needs to be generated under the reference.map code, as a result of a change to the service, updating web reference does not seem to do so; I had to delete the web reference and re-add it to see the newly added schema.

Last - let’s look at how the ASMX service behaves with two namespaces –

ASMX service with two namespaces

I’ve added the second class, and added it as a parameter to my web method

[XmlRoot(Namespace = "http://AontherNamespace")]
public class AnotherCompositeType
{
    bool boolValue = true;
    string stringValue = "Hello ";

    public bool BoolValue
    {
        get { return boolValue; }
        set { boolValue = value; }
    }

    public string StringValue
    {
        get { return stringValue; }
        set { stringValue = value; }
    }
}

And, still consistent – when consumed from BizTalk 2006 I’m getting only the WSDL downloaded (two schemas are embedded) but the reference.map code contains two schemas – one for each namespace.

To summarise -

Using the DataContractFormat you will always get one schema for generic types, one schema for the service’s messages, and then one schema for each namespace any other types are declared in (0..n)

Using the XmlSerializerFormat schemas are embedded in the WSDL file, and you would get one per xml namespace used.

As far as BizTalk generated code is concerned, however, there’s no difference between the two.

What this meant to us – well – we understand better, but there’s still not much we can do.

In our case – we control the service, and – in fact - we know that the only reason we encounter multiple xml namespaces in the service contract is because the various classes exist in several .net namespaces, and they have not supplied the DataContract attribute on any class, they have certainly no supplied the namespace parameter to that attribute, which meant the .net namespace was used as the xml namespace, resulting with multiple namespaces and therefore multiple schemas.

One that team added the attribute, and used a consistent xml namespace throughout, our immediate problem was solved; however - had it been a third party’s service we would not have that luxury and we would have had to update our code whenever we update our web reference, even if only new types were added in a backwards compatible way (as the schema ordering may have changed)

On that – it’s probably easier to simply rename the schemas (and the underlying .net types) under the reference.map generated code rather than the referencing maps and messages.

Pro Business Activity Monitoring in BizTalk Server 2009

noreply@blogger.com (Yossi Dahan) — Thu, 03 Sep 2009 06:42:00 +0000

Richard Seroter wrote a review of this book on his blog

Whilst I haven’t yet finished reading the book, I completely agree with Richard – this book is very well written, and is doing a fantastic job explaining this fascinating, and often misunderstood, if not completely overlooked, capability of BizTalk.

The book also does a very good job at looking at scenarios outside BIzTalk server, making it well worth reading for any enterprise solution architect.

Highly recommended.