The message box as a service boundary

For the last 18 months or so I've been working on a very exciting, and quite large, BizTalk implementation here in the UK, I'll leave the full details of it for now, but I can tell you that it involves all the nice buzzwords we keep hearing about SOA, SaaS, S+S, ESB at least to some degree (and with various level of quality, if we're honest)

Anyway, as you can imagine we're using web services quite extensively - we expose a lot of them, and consume even more; some are internal to the company (but cross teams, although not so much platforms) and many are external (which do cross platform as well)

The reasons to use service oriented architecture should be very clear to everyone by now, as are the famous four tenants of SOA.

In out implementation we've abstracted the calls to all the internal web services through utility orchestrations which would take a message in our canonical format , convert it to the service's format, call the web service and transform the response to the canonical format before returning it to the calling process; this way we can re-use those transformations, and have a central place to deal with each request, apply error handling, etc.

From the parent process we then use call orchestration to initiate these utility orchestrations passing in the request cannonical message and receiving the response canonical message as an out param, which is quite efficient (when initiating orchestrations through the call orchestration shape the request does not go through the message box)

As we're doing the transformations in these utility processes, we consider them to be in the boundary of our process, and not, obviously, within the boundary of the called service, for this reason we call the web service from the process rather than the actual assembly behind the web service.

When we, within the utility process call the web service, what actually happens is that the request message (now in the WS' format) gets published to the message box, being picked up by the send port which would pass it to the SOAP adpater which, in turn will serialises it and transmits it over the wire to the service; the service then deseralises the message on the other end before executing whatever code needs to be executed and the entire process now repeats in the opposite direction.

In this case the service boundary is the web service endpoint.

A few weeks ago I had what I thought was a brilliant idea - why not treat the message box as the service boundary!?

If I had a process that takes in the service's format of the message using a directly bound receive shpae and a filter, execute the code internally (as we're now inside the service boundary we can use the service code directly from expression shape, no need to go through a web service) and when finished publish the response back to the message box (in it's own format), I could have simply published a request message for that service, and get the response published back for me; correlation should be used, but this can be handled using self-correlating ports or a correlation set.

The client process would do pretty much the same - it would use a utility process to transform the canonical format to the service's format and publish the request. it would then use correlation to receive the response and transform it back to the canonical format before retunrning it to the calling process(synchronously).

What would we save? - following this approach for at least some of our internal services can save us the need to serialise the messages over the network; in the web service case we have to go through the message box from the process to the send port anyway, so going through the message box from one process to aonther would not make a difference, but all the network traffic and the work by the SOAP adapter (which is far from being efficient) can be saved.

This was a good idea (I thought anyway), but I suspect it won't work, as it has two main flaws (and I will be extremely happy to get some ideas around those) -

Firsly - both subsystems will need to exist on the same BizTalk group so that they share the same message box and so we could use pub/sub to exchange messages between them (on it's own this is not necessarily a problem, but it is the main cause for the next one, which is the big one)

Secondly - the schemas will have to be shared -

When you're adding a web reference to a web service from a standard .net project a proxy gets generated for you; that proxy will include a local version of all the classes used by the web service (these will be in YOUR code namespace rather then ther service's but will serialize to the same XML).

Equally - when you add a web reference in a BizTalk project, you get schemas generated so you can create messages to send and receive to/from the web service; these will be in the service's XML namespace as they have to represent the XML supported by it, and here lies the problem.

If both the service implementation and the client implementation are on the same BizTalk group, the schemas will have to be shared as there's no way to deploy two schemas using the same root node and namespace and we all know that sharing schemas is a bad idea as it strongly couples the implementation together and that pretty much renders the idea useless (this, confusingly I suspect, means we're sharing a class and not a contract).

Of course one could play around with the idea of having two BizTalk groups and communicating between them, and although you can choose better transports than SOAP for that internal communication I suspects that brings us closer to simply calling the web service and so I'd rather stay with that standrad approach.

Anybody still thinks SOA is just a buzz-word?

If there was any doubt in anyone's mind that SOA is here to stay, that services are the way forward for many applications, and that Microsoft is investing a lot in enabling us to deliver even better solutions based on SOA, SaaS, S+S etc, yesterday's announcement about Oslo (also check out this issue of research directions) shows that the level of commitment in this field by Microsoft is only increasing, and that we're about to see a lot more in this area over the next couple of years.

Everyone says it, and it's true - these are very exciting times to be in software development, especially in the enterprise and BPM/EAI in particular perhaps, and I can't help the feeling we've really only scratched the surface on this one, a lot more is still to come.

I'm sure we're all going to hear, read and write a lot in the next few weeks; I know I've got tons of new stuff to read on my desk now, and hopefully will get to see some nice stuff coming out of this in a few months time.

BTW, don't forget to check out the newly launched SOA web site as mentioned by Marjan Kalantar

Can project code name "Astoria" align with SOA?

I've recently had the pleasure of seeing Guy Smith-Ferrier presenting Microsoft project codename "Astoria".

I've been doing some reading on Astoria (and generally on the REST architecture style) recently, and although I have one serious reservation I'm about to discuss I think this is a very interesting area to look at;

What is my reservation regarding "Astoria" then? well, I'll start by saying, of course, that it could very well be simply my lack of understanding, and of course this is all based on a very first impression of "Astoria", and I'm quite certain a lot is still about to change in this area (which is why many "smaller" questions can simply be left aside for now I think), but anyhow - what I'm not sure about at this point is whether, and how, can "Astoria" and SOA fit together, as to me they seem to conflict.

I will not explain Astoria here, as there is enough information out there, but if you just take one sentence from the link provided above - "The goal of Microsoft Codename Astoria is to enable applications to expose data as a data service that can be consumed by web clients within a corporate network and across the internet. "

Also, from playing around a little bit with Astoria, and from Guy's demonstrations , I see "Astoria" as an easy way to publish REST like data services to allow a client to perform CRUD operations on data entities; currently SQL Server is supported, but it is quite obvious this is going to evolve to other platforms.

So - my problem is, quite naturally I should think, with the CRUD bit; I like services, and I am learning to like REST (although have not used it in practice anywhere yet), but over the last couple of years I've been hearing all around (not to say thinking on my own right) that CRUD services are bad, and I do think there are lots of reasons as to why they are bad (some are very nicely explained here)

Do we really want a service that simply provides create/read/update/delete operations on a table, or view?

As I've mentioned before - this raises a lot of questions around security, and transactions etc. some can be solved simply by the fact that "Astoria" is built on WCF which provides support for some of these, but will it be enough? Time will tell.

Either way, from a design perspective, it seems to me that CRUD services like that are only useful, at least in my (narrow?) world, when they actually form a part of a bigger service, one that would actually provide business logic on top of the data logic, and that will "speak" in business terms rather the database entities terms.

For example, if I had a service that maintains inventory in a warehouse, and a product was to be shipped to a customer, using a CRUD-like service you would update a row with the new inventory value (so your message would look something like-[Update inventory for product 'x' with '15']);

Using a business service, the message you would send will probably look something like ['2' items of product 'x' need to be shipped to customer 'y']; the service can then update the stored inventory with the correct value (without worrying about concurrency, I might add), as well as do whatever logic is required around that, for example initiating the packaging and shipment of the product to the customer, updating other systems, etc.

Now, it is true that "Astoria" does provide extensibility points, and one can write code around that data retrieval logic provided to perform some actions before, or after the data was retrieved, but is that a good way to implement business logic? this is a very data centric, data entity aligned, way of thinking, which does not sit well with all that has been said and done with SOA to my understanding.

I like the REST approach for urls and am hoping to see more of it used, but I don't like the idea that logic is treated as essentially pre or post-processing stage on top of a data retrieval logic, nor do I think that business processes should think in data entities terms.

It would be extremely cool to expose REST endpoints to processes in BizTalk or WF, but I feel that is the only way to align REST and SOA and in my mind "Astoria" is a deviation from what I think of a good implementation of SOA.