New ‘Oslo’ CTP released, and so there’s a new version of BTSDF

Somewhat quietly Microsoft released another CTP of ‘Oslo’, which you can find on the Oslo dev centre; it is great to see the team release early and frequent drops of ‘Oslo’, especially given the impact ‘Oslo’ is likely to have on how we build software.

Having the chance to play with the bits so early, and provide feedback, which they seem to be very keen on receiving, is pretty awesome!

I’ve only had a brief play with it and I guess that, to me, the greatest news are that – all the potential benefits aside – my existing code (BTSDF) still worked as is; of course this is only temporary, as the team have made significant changes (read: improvements) to the API, but they have kept backwards compatibility TEMPORARILY as they work to align all their existing code to the new model as the rest of us worry about ours.

So, next I needed to spend some time looking at the new release in detail and align my code with it, so I reap some of the benefits from the improvements made, but that’s a much better position to be in than – “it’s all broken now and I need to figure out how to fix it”m kudos guys!

Paul Arundel was kind enough to give me a gentle nudge to start taking a look at these changes, and it certainly took me longer than I would have wanted to get around to these things thanks to other commitments (a repeating theme here, recently), but – slowly but surely – I went through my code and am happy to say I’ve now published the necessary changes to codeplex.

I was going to write a post on the changes required when moving to the new code, but Paul had done so already, and so there’s little point in me saying pretty much the same things (Paul is only focused on the aspects that are relevant to his project, but as we both focus on pretty much the same area my words would have been pretty much identical).

There only one point I think is worth mentioning from my end – it’s probably only me – but being able to walk to graph in an easier and more convenient way, and being able to easily access nodes by label (or by checking their brand) motivated me to work more on the grammar itself, or – to be more specific – on the projection of the grammar –  where previously i would just jump through whatever hoops were needed to parse whatever projection I got.
This is a very good thing I think (although it is the grammar itself that really matters, as this is what users will see); the other half of this, though, is that it is still not quite possible to get the projection just right – I can’t get a projection that would work well with any M model for my domain, for instance, but I suspect the smart guys in Redmond are hard at work sorting this out for us…

Oslo based solution for deploying BizTalk applications – published

Unfortunately life is a bit hectic for us at the moment, and it’s been a while since I’ve posted anything, or was able to do anything other than family or work.

As part of this general “neglect” I was unable to spend the time required to complete, and publish, my Oslo based solution for deployment BizTalk applications (given the somewhat ugly, but suitably short, name – BTSDF), but as many waited (ok – more than 5), and some kept asking (2), I thought I’d do my best to get something out of the door, so I have.

 

in Codeplex  you can now find an initial version published, which includes – the language definition, the core runtime I have as well as two “executors” in various stages of [non] completeness – my MsBuild generator is already quite useful for simple apps, it generates a set of MSbuild files you can use to deploy the application on any machine as well as the required SDC tasks dlls and targets file; the API executor deploys the app on the current machine, and is quite basic, but a good sample (I think) and a reasonable starting point.

I’m happy to entertain requests for changes/additions, and even more happy to add any one who’s willing and able to put the time as a contributor; Oslo knowledge is optional! :-)

 

You can download the source, build and run it locally – but you will have to remember to copy the two executors dlls from their folders to the main bin\debug folder as there is no compiled reference to them.

 

I’ve also uploaded an “Alpha release” which includes the compiled assemblies and the supporting files required.

There’s still more work to do, but it’s getting shape now, and I’m using the MSbuild executor today; it certainly needs a bit more documentation, which I hope to get around to…

 

I hope this works for you, let me know what you think (the good and the bad), and – if you have some time and will – drop me a line and I’ll add you to the team!

 

Yossi

Oslo based solution for deploying BizTalk applications

Since PDC I’ve been working on and off on an “Oslo” based solution for deploying a BizTalk application; unfortunately I couldn’t get a good chunk of time to play with this, so it’s been dragging a bit, but I’m getting close, so here are some details -

I’m a big advocate of automated builds; it’s a topic that probably deserves a post of its own, so I won’t get started on this here, but the idea is that one must have a way to be confident that, when its time to [re-]deploy the app, it will get deployed successfully time after time, without a hitch;

From my experience, deploying a BizTalk application of a decent size is often not trivial, and often it does not go smoothly if done manually (things are being missed, done out of order, wrong versions being picked up, etc), which does not help to boost the confidence in the solution (or BizTalk as a whole) in the organisation; automating the process can save a lot of time (can be done unattended, while in a meeting, out for lunch or overnight), save a lot of head scratching, boost the confidence in the solution and set the ground for proper automated builds, auto testing etc.

To that extent, I have previously built an MSBuild based framework for deploying BizTalk applications.

While I’m sure it does not suit all purposes as it was developed to support those scenarios I had, I think it’s quite comprehensive and had served me well over the past couple of years.

It allows one to provide an MSBuild-formatted file with some parameters and lists and, using the pre-created framework, would do things like remove the existing application (after terminating current instances and un-enlisting services), build the new solution(s), deploy assemblies to BizTalk, add/import binding files, start ports etc.

This is working great for us, and we’ve been using it extensively for quite a while now, but there’s one major downside – it requires one to maintain those MSBuild scripts.

Now -  MSBuild is cool, I do like it very much, but as it’s generic – it does not speak in BizTalk terms, and as it’s XML – it’s quite verbose, and I wanted an easier format for all of us to work with.

So – inspired after seeing Oslo and even more driven after visiting PDC - I’ve decided to come up with an MGrammar based language for describing a BizTalk application to be deployed.

So far I’ve come up with “version 1.0” of my grammar that allows to describe the major artifacts in a BizTalk application and a basic runtime to process source files written in this language, the language allows you to describe things like -

• References to other applications
• Solutions and projects to build
• BizTalk assemblies to deploy
• Orchestrations contained in those (these can then be terminated before attempting to remove the application prior to deployment)
• .net assemblies to deploy
• Binding file to import
• Binding files to add (and the name of the environment to attach to those)

As Oslo hasn’t RTM-ed yet, I can’t quite rely on it yet, and so I cannot use it in production in any shape or form, or can I?
I found a good middle ground for us which allows us to gain from the benefits I’m hoping to get by using my language, while not exposing ourselves too much to the risks of using early technology –

At the moment the runtime I’ve built for this is used to generate the MSBuild scripts we’re already using out of the source files; in this way - if Oslo disappears tomorrow, or significantly changes (not that I think that’s going to be the case), we’re safe – we still have the MSBuild scripts as “checked-in artifacts” and so we have lost nothing.

So – how does an instance of my language looks like? here’s an example:

Application MyApp
{
    //add references to other applications
    add reference SomeOtherApp;
    add reference AndAnother,andAThird;
    
    //build the solution
    build "SomeFolder\SomeOtherFolder\SomeSolution.sln";
    build "SomeFolder\SomeOtherFolder\SomeProject.btproj";
    
    //add required biztalk assemblies
    add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Schemas.dll"
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;

    add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Maps.dll"
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;

    add biztalk assembly "SomeFolder\SomeOtherFolder\bin\deployment\Orchestrations.dll" 
    with orchestrations
        {
            MyApp.SomeOrchestration,
            MyApp.AnottherOrchestration,
            MyApp.AndAnotherOrchestration
        }        
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;
    
    //and .net helpers assembly
    add assembly "SomeFolder\SomeOtherFolder\bin\release\HRG.FareWatch.Hotel.Helpers.dll" 
        version=1.0.0.0 
        culture=neutral 
        publicKeyToken=314bd7037656ea65;
    //import dev bindings
    import binding "SomeFolder\SomeOtherFolder\Bindings.xml";

    //add various environment's bindings.
    add binding "SomeFolder\SomeOtherFolder\Bindings.Dev.xml" "development";
    add binding "SomeFolder\SomeOtherFolder\Bindings.Stage.xml" "stage";
    add binding "SomeFolder\SomeOtherFolder\Bindings.Prod.xml" "production";
}

as I’ve said – I already have a runtime that generates the MSBuild scripts required to deploy these - the runtime outputs several files to the temp folder - 

• The “framework” MSBuild script that contains all the targets I’m using



• The Microsoft.Sdc.Tasks.BizTalk.dll and the Microsoft.Sdc.Tasks.dll which contains many useful custom tasks.



• The relevant import file for the SDC tasks



• The generated MSBuild file that contains all the various artifacts based on the source code

That latter file, which is the equivalent of my source code, and what we had to maintain so far, looks like -

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="DeployAll">
  <Import Project="Deploy.Framework" />
  <ItemGroup>
    <MyApp Include="SomeOtherApp" />
    <MyApp Include="AndAnother" />
    <MyApp Include="andAThird" />
  </ItemGroup>
  <ItemGroup>
    <Bindings Include="General">
      <path>SomeFolder\SomeOtherFolder\Bindings.xml</path>
    </Bindings>
  </ItemGroup>
  <ItemGroup>
    <EnvironmentBindings Include="General">
      <path>"SomeFolder\SomeOtherFolder\Bindings.Dev.xml"</path>
      <environment>development</environment>
    </EnvironmentBindings>
    <EnvironmentBindings Include="General">
      <path>"SomeFolder\SomeOtherFolder\Bindings.Stage.xml"</path>
      <environment>stage</environment>
    </EnvironmentBindings>
    <EnvironmentBindings Include="General">
      <path>"SomeFolder\SomeOtherFolder\Bindings.Prod.xml"</path>
      <environment>production</environment>
    </EnvironmentBindings>
  </ItemGroup>
  <ItemGroup>
    <ExternNetAssembly Include="HRG.FareWatch.Hotel.Helpers">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\release</path>
    </ExternNetAssembly>
  </ItemGroup>
  <ItemGroup>
    <Solution Include="SomeSolution">
      <path>SomeFolder\SomeOtherFolder</path>
    </Solution>
    <Solution Include="SomeProject">
      <path>SomeFolder\SomeOtherFolder</path>
    </Solution>
  </ItemGroup>
  <ItemGroup>
    <BTSProject Include="Schemas">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\deployment</path>
    </BTSProject>
    <BTSProject Include="Maps">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\deployment</path>
    </BTSProject>
    <BTSProject Include="Orchestrations">
      <Version>1.0.0.0</Version>
      <Culture>neutral</Culture>
      <PublicKeyToken>314bd7037656ea65</PublicKeyToken>
      <ProcessorArchitecture>MSIL</ProcessorArchitecture>
      <path>SomeFolder\SomeOtherFolder\bin\deployment</path>
    </BTSProject>
  </ItemGroup>
  <PropertyGroup>
    <BTApplicationName>MyApp</BTApplicationName>
    <!-- Set for a remote deployment -->
    <!-- Deploying BizTalk Server name - leave blank if local-->
    <!-- not currently supported by runtime-->
    <BTServerName>
    </BTServerName>
    <!-- Deploying BizTalk Server database - leave blank if BizTalkMsgBoxDb-->
    <BTServerDatabase>
    </BTServerDatabase>
    <!-- Deploying BizTalk Server SQL user name - leave blank if local-->
    <BTServerUserName>
    </BTServerUserName>
    <!-- Deploying BizTalk Server SQL password - leave blank if local-->
    <BTServerPassword>
    </BTServerPassword>
  </PropertyGroup>
</Project>

Would you agree that the former source code is easier to maintain?

 

note: one thing you would notice is that none of the paths contains a root; I assume that this would be used by different developers/IT pros which may have the code in a different path; however, as the assumption is that the code will come from source control, my framework script expects you to provide the root path to your source code and assumes the specified paths start there.

So – what’s next?

From a coding perspective - I plan to add support in my runtime to perform the actual deployment; this is I would like it to do once Oslo is in production so I’ll add it as an option through a command line switch.

This option would tell the runtime to deploy the artifacts to a specified BizTalk server using BTSTask or the Object model instead of generated the MSBuild script.

I also want to make some modification to my language definition to make the MGraph produced cleaner (and better geared towards using the repository and Quadrant at a later stage) as well as, obviously, add support for more features, after which I plan to publish both my language and the runtime somewhere (here, codeplex, DevCente..we’ll see)

I’ll post some more notes on all of these here in the near future hopefully…

Lots more information is now published on Oslo

Now that the main announcements in PDC are happening MS are starting to release a lot more information about it all.

check out http://modelsremixed.com/ as well as http://msdn.microsoft.com/en-us/oslo/default.aspx

BizTalk get's a new logo (oh - and a new version too!)

Microsoft has announced the next major version of BizTalk - BizTalk Server 2009.

Planned to be released 'H1 CY09', the new version, previously referred to in some circles as BizTalk "R3", adds support for the latest Microsoft stack - Windows Server 2008, SQL Server 2008, Visual Studio 2008, .net framework 3.5 and TFS 2008 - including automated builds and continuous integration as well as virtualisation using Hyper-V.

There are also further investments in B2B (EDI, SWIFT), RFID, enhanced LOB adapters and support for UDDI 3.0

Next CTP is planned for sometime in the last quarter of this year, while previously Microsoft announced that BizTalk '6' will be part of the 'Oslo' wave this version released ahead of 'Oslo', which is great news for all of us longing to move to the latest platform with BizTalk Server.

See the announcement here

BizTalk's Pub/Sub

The publish/subscribe mechanism in BizTalk is one of the key features of the product and is very useful and powerful.

I guess there's some learning curve around it, and that most first implementations of any BizTalk developer do not make much use of it (as they often start with all ports being directly bound) and that it takes a while befor a BizTalk developer and the organization involved establish a good architecture and move more towards using true publish-and-subscribe, losely-coupled, approach to implementation.

However, the existing model is not perfrect; in my view (and I suspect it is shared by many) it has two main weak points -

The pub/sub is implemented on top of MS-SQL which introduces a significant performance overhead

The orchestration subscriptions are 'compiled' and cannot be configured withouth a build-and-deploy cycle

The first point is quite an obvious one - there would be a latency associated with any implementation of publish/ubscribribe mechanism;. in the BizTalk case it involved writing the message and it's meta data (context) to the message box (a SQL database) and having a separate process locate newly published messages, figuring out which subscribers need to receive a copy of the message and manage the activation/correlationthe of message-process interaction (as well as keeping a list of references for house keeping etc).

Reading and writing to the database, the the polling interval of the subscription evaluation process, etc. all introduce latency, which, in certain scenarios, can be crucial.

If to believe the fractions of information floating around regarding Oslo then we might see an in-memory pub/sub mechanism in future version of BizTalk (in addition, not as a replacment to the existing model I suspect) which, while will no-doubt come with a price (persistance, and therefore scalabiltiy and durability to some extent), will no-doubt make supporting low-latency scenarios much easier.

As for the second point -

At first look the pub/sub in BizTalk is very flexible; in all the BizTalk demonstrations I can remember from the past the presenter would create a recieve port and a couple of send ports and will edit the subscriptions of those ports in the administration console to show how easy it is to create content-based routing in BizTalk server and configure it at runtime.

In BizTalk 2006 you even did not have to restart the host to speed things up (as you did in demos with 2004), it happens pretty much instantly.

However, the case with orchestrations is not that simple...

The subscription for orchestrations is specified as a a filter in the properties of the initalizing receive shape in the process; this gets compiled into your assembly together with the process, and will be used to create the subscription when you deploy the orchestration.

As far as I know, short of manipulating the management database yourself (which would not be supported) there's no way to change those subscription at runtime.

If you want to change the subscription you have to change the filter in the orchestration, build, undeploy the old version and deploy the new one (or version the process and perform the side-by-side deployment)

This is, in my view, an un-necessary pain, in dynamic organizations (aren't they all?) that require changes often; and to that extend developers had to find a solution to the "I need to be able to change that subscription from outside the process" requirement.

That solution is often adding some routing metadata to messages in the form of context properties ('nextProcess', 'Operation', etc.) which would be set by publishing processes and/or pipeline components and use these in the filters (rather than the actual content data).

So you could often see a pipeline component, often driven by some external configuraion, that would check for certain bits in the message or it's meta-data and set these context properties based on the values it found; the premise is that pipeline components are easier to replace, but also - thesee components often use database or a rules engine in one form or another to decide what goes into the message context and by doing so introducing real flexibility as is advertised.

What all of this means is that we, developers, end up developing a pub/sub mechanism on top of the existing pub/sub simply because we need flexibility the product does not provide.

I don't like this apprach, but I end up doing this myself occasionally, simply because I have to.

I could possibly understand why MS has decided to do so - there are benefits to editing the subscription expression within the orchestration (known types would be one thing), and also - one could argue that the process subscription is part of the process design and so changing it is likely to involve code changes as well which will require a re-build, but really - I think we would all have benefited from the ability to edit the orchestration subscription in the same way we can edit send port subscriptions - through the admin console.

Anybody still thinks SOA is just a buzz-word?

If there was any doubt in anyone's mind that SOA is here to stay, that services are the way forward for many applications, and that Microsoft is investing a lot in enabling us to deliver even better solutions based on SOA, SaaS, S+S etc, yesterday's announcement about Oslo (also check out this issue of research directions) shows that the level of commitment in this field by Microsoft is only increasing, and that we're about to see a lot more in this area over the next couple of years.

Everyone says it, and it's true - these are very exciting times to be in software development, especially in the enterprise and BPM/EAI in particular perhaps, and I can't help the feeling we've really only scratched the surface on this one, a lot more is still to come.

I'm sure we're all going to hear, read and write a lot in the next few weeks; I know I've got tons of new stuff to read on my desk now, and hopefully will get to see some nice stuff coming out of this in a few months time.

BTW, don't forget to check out the newly launched SOA web site as mentioned by Marjan Kalantar