Geeks With Blogs

News View Michael Stephenson's profile on BizTalk Blog Doc View Michael Stephenson's profile on LinkedIn
Michael Stephenson keeping your feet on premise while your heads in the cloud

I've just been looking over Bizmonade which is a testing tool for testing BizTalk orchestrations in isolation from the BizTalk execution environment. It's similar in concept to BizMock however the implementation is different. The simple way to describe how BizMock is implemented is that it uses a custom adapter which allows you to manipulate the messaging around an orchestration which is executing within BizTalk. Bizmonade is different in that it creates a custom version of the XLANG Engine which it then allows you to load an orchestration completely outside of BizTalk for testing.

Both projects allow you to test the orchestration from NUnit or MsTest with a syntax similar to most mocking frameworks.

As I've wrote a bunch of stuff in the past about testing, I'll be keeping an eye on these projects. What I'm really interested in is their positioning within the testing you would normally do. Both projects position themselves in that gap between grey-box testing with BizUnit and the component testing of schemas/maps/etc. There is clearly a gap there but how big of a deal it is I'm not sure.

My current questions and thoughts about where I would use the above two components are as follows:

  1. Would I really want to extensively test all orchestration paths with Bizmonade and probably have an amount of duplication in my BizUnit testing
  2. Would I be looking to use Bizmonade for edge or obscure test cases which would be a pain to test with BizUnit
  3. Would I be looking to trust Bizmonade enough to reduce the amount of BizUnit testing I do
  4. Could a key place for Bizmonade be where I usually have a chain of orchestrations which could be difficult to test with BizUnit, and I could test the orchestrations individually with Bizmonade
  5. What risk does running my orchestration outside of BizTalk pose. Might I end up dealing with issues which wouldn't happen in XLANG or missing conditions that would happen in XLANG
  6. Would Bizmonade offer a better way to ensure good test coverage for orchestrations
  7. Would I lose the ability to debug orchestrations
  8. Am I going to have to pay for Bizmonade (longer term)

Hopefully the answers to a lot of these questions will become clear as I get more experience with the component and with future releases (remember this is only a preview release).

As mentioned earlier there is a gap in the testing capabilities between BizUnit and the ability to unit test orchestrations, however this hasn't really caused me too much pain in the past. I think to be successful Bizmonade needs to be able to save me time in my development cycle and improve the quality of any deliverable maybe one of the best things Bizmonade could look to do would be to interpret an orchestration in the same way that Visual Studio Unit Tests can for C# and then generate some code which has tests for most of the common conditions and branches within my orchestration. Now that id really like.

Posted on Monday, October 26, 2009 7:43 PM BizTalk | Back to top

Comments on this post: Bizmonade & Orchestration Testing

# re: Bizmonade & Orchestration Testing
Requesting Gravatar...
Hi Michael,

I've been looking over Bizmonade for the last few days and I have to say I like what I see. Purely on a technical level I think it's quite an achievement and shows what could be possible. There are still some functional gaps, but I think, with some community support, these can be overcome (and as you say, it's very much a preview release)

My biggest single complaint with BizTalk development (and there are a few!) is the inability to do test driven design (or BDD as I've started using more and more). The main problem is the feedback loop of build, deploy, test is simply too long to allow TDD. Both BizMocks and BizUnit suffer in this regard. I've already managed to use Bizmonade to simulate driving an orchestration design without deploying, and it was a liberating experience. I can now start with a set of user stories, and drive out the design of interfaces (schemas) and processes (messaging/orchestration/mapping). There a still a few areas around mocking internal C# component usage which I’m unclear about, but I'm investigating if a windsor container can provide this functionality.

Of course BizUnit will allow you test most execution paths and with a little creativity I’m sure you could have pretty much 100% coverage. But in my experience, most of these tests are written either at the end of development, or used just as a harness to inject messaging. I’m not knocking BizUnit, it’s an absolutely essential part of the arsenal of tools required – I just don’t get much of any rhythm from the develop/test process with it.

I do have some concerns. It really needs to be open-sourced. I gather this might be an option in the future, but I think it's required for two reasons. Firstly, to ensure that if you invest time in the product, if it does "die on the vine", it can be forked. Secondly, imagine where this could go with community contributions / patches – pretty important to me. It doesn't replace BizUnit integration tests. This is clearly identified by Bizmonades author who positions the tool firmly in the unit tests are, but the danger is that some developers could miss this and that would be dangerous. I’m also concerned about how accurate it can be in simulating the run time. But to be honest this has to be caught by integration tests, and so I don't see this as a major issue unless something really ugly surfaces

Onto your questions over licensing, I asked the same from the author and he assures me that “We will pick a license that allows using it in closed source/non-free projects (such as a BSD or Ms-PL license), so you can use it freely in commercial/non-open/non-free projects”. He also hinted that they may go open-source

I think you're 100% on your conclusion of Bizmonade chances of success – it’ll have to allow you to develop faster and better. Frankly, Bizmonade had me on “you don’t need to deploy” :)
Left by Jason Hyland on Oct 27, 2009 4:41 AM

# re: Bizmonade & Orchestration Testing
Requesting Gravatar...
Hi Jason,

Cheers for the good and detailed comment.

I agree with pretty much all of what you say and agree it is a very impressive achievement to mock XLANG.

I must admit im not 100% on Bizmonade but it certainly will be in my toolkit. Where im not convinced is if i will bizmonade test everything or just use it for awkward or obscure tests where it would be more effective to cover a particular case without having to worry about the rest of the biztalk plumbing.

I also think the RAD bit you mention is a big win too to get more towards a TDD approach. At the minute I am used to working probably as close as i can get to using a TDD approach with BizTalk as is and I fine that because we have good build processes the pain of running a build balances out with the fact that i know the environment is good.

The bit im really interested in is how people would position the overlap of tests that would be done with bizunit and bizmonade. Did you find you did more testing overall?, did you do less bizunit testing? was the quality level higher? were the timescales increased or decreased?

If your doing real stuff with this (i know your UK based) do you fancy doing a webcast at somepoint for UK SOA/BPM User Group to show people what your doing with this?

Drop me an email if you fancy it

Cheers for the comments
Left by Mike on Oct 27, 2009 9:44 AM

# re: Bizmonade & Orchestration Testing
Requesting Gravatar...
Hi Michael,
Thank you for your feedback, I have posted some answers here:

One important thing is the licensing, as Jason mentionned the tool will remain free and eventually be open source (once the code is stable enough and an appropriate license has been chosen)

About your question: "2. Would I be looking to use Bizmonade for edge or obscure test cases which would be a pain to test with BizUnit " : I'm not sure I correctly understood what kind of obscure edge cases you had in mind, do you have any examples? (it could give me some ideas on missing features to implement)

I also talk about the issues of trust in Bizmonade results, test duplication, and of my guiding objective when developing the tool: accelerating development by providing faster feedback.
Left by Christos Karras on Oct 28, 2009 10:11 PM

# re: Bizmonade & Orchestration Testing
Requesting Gravatar...
I think Christos has replied to a few of these points, so I won’t repeat them. BizUnit has been all I’ve had on my tool belt when it came to testing BizTalk, and I don’t that changing very much with bringing in Bizmonade. I do think however that Bizmonade will improve the quality and speed of delivery. Let me give you a simple example

Imagine a (common) scenario where you are looping around a repeating node in a message in an orchestration (assume debatching isn’t appropriate, indulge me <g>), and generating and sending an individual message out to another service for each loop element. A pure TDD approach might be to write a test that, for a given input message asserts a loop count is correctly returned. I’d love to do this Bizmonade, but it’s not possible without a bit of shenanigans (I have some ideas about how we could though), but what is possible is to write a test that expects 5 messages dispatched. Now of course this is possible with BizUnit, but if I get the xpath wrong to pull out the count of loop items (not that we’d do that!), the iteration cycle for BizUnit has involved a deploy of the artefacts, and maybe a restart of the host instances. Not a massive overhead (unless it’s a real world project with, say, 4 or 5 applications pointing at it), but say we get it wrong again, then we the xpath that pulls out the separate message out wrong, etc. Multiply them all up, and you see what I mean. Developers sat around waiting for a deploy to see if a small change has worked is very poor.

I’m not yet able to use it extensively in my production projects because it doesn’t yes support a couple of shape we use in our design (suspend and terminate), but Christos is working towards it. As for a webcast, I’m sure Christos would do more justice to it, but failing that, once I’m rocking and rolling I’d be happy to.

Regards, Jason
Left by Jason Hyland on Oct 29, 2009 1:27 PM

# re: Bizmonade & Orchestration Testing
Requesting Gravatar...
Hi Jason

Yep see where your coming from. I think also on bigger teams the ability to test without deployment probably is a big win too.

In terms of the webcast ive spoken to christos about something after xmas. Why dont we get you guys to team up? Maybe we could get christos to give us an overview and then jason could talk about his practical experience with bizmonade. Would be a very rounded session and im sure very popular

Shall we hook up in december and have a chat about this?

Think you both have my details but if not ill pass them over

Left by Mike on Oct 30, 2009 8:23 AM

Your comment:
 (will show your gravatar)

Copyright © Michael Stephenson | Powered by: