The lost property

You've created your pipeline component, added a few public properties for good measure (to make it a bit more flexible of course).

You went on to create a pipeline with your component, set all the properties you've introduced and deploy it to BizTalk Server.

You then configure a port to use the pipeline (and the contained component) and test your scenario. all is sweet.

But then - you decide to add one more property to your component.

You quickly go and change the component, adding the public property you've missed, and, if you're anything like me, you decide to skip the whole: update pipeline, remove from port, undeploy, redeploy, re-configure port etc. and simply GAC the updated component planning to set it's value through the admin console to save the pain (and time).

However, when you open the port in the admin console and going to set the newly added property in the pipeline configuration you are into a surprise - your new property does not appear in the UI.

You double check everything - check your code, re-build, re-gac and re-open the admin console, but it's all the same - the property is simply not there. you bang your head against the wall. twice.

And then it hits you - it's not the component! it's the pipeline!

A pipeline "source code" is essentially an XML document. pipeline components, and their properties, are XML fragments within this XML.

When you add a component to the pipeline, its information, including all known properties (and their values) are added to the pipeline's XML.

And yes - you've guessed it right - when you're editing a pipeline's configuration through the admin console, the source of the generated UI you see is that XML, not the actual components' assemblies in the GAC.

If, like me, you have just GAC-ed a new version of the component and went to the admin console to configure it, you're up for a disappointment as your newly added property will simply not be there.

you will be forced to re-deploy the pipeline (unless you are happy to change xmls in the management database, that is - and you shouldn't be).

Throttling in full action

Here's another one from the archives (=the list of things I have waiting to be blogged)

At some point we had a sudden peak in system load on our BizTalk processes and, as a result, our BizTalk solution that was running so nicely seem to have gotten "stuck".

In "stuck" I mean - we ended up with lots of processes in "Active" state, but they did not seem to be active at all; a closer inspection (of trace that should have been emitted) showed that although the instances status says "Active" they were all very passive indeed - nothing was executing on the server - close to 0% CPU and no trace whatsoever.

This is where you might expect me to describe the long hours we've spent investigating the issue, the sleepless nights and empty cartons of pizza... - but really what happened is that, not being able to afford any more down time, we called out premier support which turned out to be a great thing because the first thing they did (well, not literally, but anyway) was to ask us to check the state of the server using the MsgBoxViewer which in turn pointed out that we have simply "max-ed out" our memory consumption throttling level.

You see - we use a lot of caching of data in our processes; mostly because we access a lot of reference data frequently - data that does not change very often; this is by design. what we forgot to do is estimate the amount of memory this caching will require when many different clients use the system and adjust the throttling level accordingly.

As you can see from the image below - out of the box the BizTalk hosts are configured to throttle at 25% of the server's physical memory. the idea is to prevent the BizTalk processes from taking up too much memory and killing the server, and the assumption is that if throttling kicks in, and stops processing instances, memory consumption will slowly reduce until the server gets back to a more healthy state. however - from it's very nature - caching does not really release memory that often and so instances have stopped progressing but no memory was released as a result and so we got "stuck".

In our case, the solution was straight forward - as we know our memory consumption will be high, and we know there's nothing else running on the server to compete with that memory consumption (more or less) we could increase the threshold to 50%, which is enough to grant BizTalk Server enough memory for the caching and all the processing requirements.

In the process we monitored the situation by investigating two BizTalk performance counters - "Process memory usage threshold" (here shows as 500MB) compared to "process memory usage" (here showing around 130MB).

As long as there was large enough gap between the two we knew our processes are going to be just fine; it is always important, of course, to monitor these over time to ensure there's no memory leak in the processes, which we have done, on top of peak load tests - which we have not.

Now, while all of this is down to a test or two we may have neglected on our side, there are a couple of interesting points at the back of this from a product perspective -

1. We were confused by what we saw mostly because of the "active" state of all instances (and we had quite a few); we would have diagnosed the problem much quicker, and on our own, had the admin console indicated that the server is not actually processing anything due to it's throttling state.
   
   
2. I can't help but wondering whether the throttling mechanism couldn't be a bit more clever and identify it has reached a dead end and is not actually helping in improving the situation. following on our case the engine realised memory usage has gone too high and has stopped processing instances. wouldn't it be great if after, say, 10 minutes it realised that memory is not actually reducing and so it will never exit the throttling state and would write something to the event log?

Again - not trying to make any excuses, just thoughts with the power of hind sight...

Logging application blog configuration error

We're using the logging application block as a basis to a tracing and logging service across our solution.

As always the application block is driven by configuration sections in config files.

Today I got an error similar to this one -

"The configuration for TraceListener named Trace Listener A is missing from configuration."

I've briefly checked the configuration file and didn't find a problem in my source or listener.

What I did find strange was that I did not used to so called Trace Listener A in the bit of code I was running but another trace listener.

A closer look at the config file (through the skilled eyes of Ben here) revealed the problem - another source had that listener configured but had a leading space in its name.

I did not expect the application block to check the validity of the entire config section, but assumed it only looked at the bits required for the current execution i.e. the trace source and corresponding trace listener, but that is not the case.