Developing Streaming Pipeline Components

A couple of months ago I published this post suggesting that pipeline component developers should look to design those in a streaming fashion which avoids loading the message into memory.

Quite a few people contacted me asking for an example so eventually I sat down to create one.

Creating the sample warranted writing down an explanation for it, which turned out rather long and (hopefully) elaborate, so i turned that into an article and, I’m proud to say, has now been published on MSDN.

In the article I wanted I wanted to demonstrate a realistic case, which doesn’t necessarily take the “easy path” and yet is clear enough to demonstrate the principles without overwhelming the reader; primarily I felt it was important to show a sample that modifies the stream (and not just using information from it) as these tend to be more complex.

Hopefully I have managed to provide a good demonstration of the principle. Let me know…

I guess I could consider this a compliment, but I’m not happy!

A few months ago I found out that a chap called Jason Hyland, who is another BizTalk consultant here in the UK, has nicked the contents of my web site to his own. word for word.

Despite the hidden compliment (I guess it means I write reasonably well) I was not very happy and so I thought I’d drop Jason a polite note.

I never received a response for my email, but several days later his web site was down, so I assumed he accepted this wasn’t nice and is working on his own wording. I guess I was wrong as today I was reminded of that and decided to check what he has done, and what do you know - http://www.needlecast.co.uk/services still uses, word for word, the wording from my web site.

How bad is that?!

 

Jason – at the off chance that you’re reading this – here’s some free publicity on my blog (currently averaging around 160 unique visitors per day) – write your own stuff, I’ve done it, so it can’t be that difficult, and it will be the decent thing to do.

 

Yossi

Wrong impressions with MaxConnection

Recently we’ve done some performance tuning on aspects of our solution, and as part of that we’ve looked at configuring the system.net maxCnnections setting (see Darren Jeffords post on this subject) to increase the number of concurrent calls our application can make to various services it uses.

We’ve looked at this for our BizTalk solution but also for a complex, multi-threaded smart client we’re building.

Initially some have been somewhat sceptical about the effects of this, despite all the information on the web, but when I tried to figure out why the reason became apparent – their observations did not match the expectation – no matter they set in the MaxConnections setting, the amount of time it took, and the number of concurrent calls to the external service remained the same.

Trying to figure out why, we’ve used two scenarios – in one we ran our smart client application and monitored using Fiddler we examined how many concurrent requests are sent to the service; on the other we ran a test application we created specifically for this purpose. Confusingly there were two separate reasons for why the observations did not match our expectations -

In the test scenario, the application and the service we used to test were all done on the local machine. Of course in this case the request to the service does not actually go through the network, and so the system.net MaxConnections setting does not play a part; as soon as we’ve moved the service to another machine the results were as expected.

The smart client application, however, was already pointing at a service on a remote machine, and so there must have been something else there; as timing was a little bit more tricky in this case the guys used Fiddler to look into the requests made where they saw all the requests were getting sent out at the same time, despite setting the MaxConnections to 1.

It turns out, and that I did not know, that Fiddler was the cause of the unexpected behaviour.

I don’t know exactly how Fiddler works, but I suspect it hijacks the requests on the local machine so that from our client point of view the requests never actually went through the network and so the MaxConnections setting has been ignored. Fiddler itself does not adhere to this (I’m guessing it is not a .net application) and so multiple concurrent requests have been forwarded by it to the remote service.

As soon as we’ve removed Fiddler the behaviour was as expected.

Lesson learnt.