Geeks With Blogs


Dylan Smith ALM / Architecture / TFS

In a previous post I mentioned that I had been trying to use Team Build to get a Continuous Integration environment setup.  I recently had the time to return to that task and have managed to get most of what I wanted up and running.  I had to jump through a few hoops to get everything working so I'm going to try and describe what I had to do to make everything work right.

My solution consists of a WinForms UI, ASP.Net Web Service, 2 Class Libraries, Database project (VSTS DBPro CTP7), Testing project, Setup project for UI, and a Setup project for web service.

My testing project contains just over 250 tests.  About 150 of these are what I call "unit tests" that directly invoke methods in one of the DLL's and mock out the database (they run very fast; about 5 secs for all 150).  The other 100 are what I call "acceptance tests" that operate at a higher level, and invoke calls against the web service (I have no tests against the UI project, or directly against the database at this point).  The acceptance tests require the web server to be up and running, and require the database to be wiped out and re-initialized on every test.  It takes about 20-30 mins to run the full suite of 150 acceptance tests.

The first thing I did was create a Team Build using the built-in wizard.  I didn't specify any tests to be run initially.  I had to do a little tinkering around to get the Team Build to work.  There were a few dependencies that I needed to either add to our source control, or install on the build machine.  I had some problems with the setup projects in there, and the database project (the problems with the database project were back in DataDude CTP3 I believe they are fixed in the latest CTP).  What I did to work around these issues was just create a 2nd solution that only contained the WinForms, Class Libraries, and Testing projects.  I had some custom code in the testing project that would auto-deploy the database with the tests, but I had to make sure the build output (the .sql file) from the datbase project was included in source control. 

I did notice that the Team Build didn't seem to like to build my Web Service project.  After a bit of research I found this post by Joel Semeniuk about this issue and how to resolve it.  I had to modify a section of the TfsBuild.proj file to look like this:

<ConfigurationToBuild Include="Debug|Mixed Platforms">
  <PlatformToBuild>Mixed Platforms</PlatformToBuild>

Not a big deal once I knew how to do it.  Now the Team Build ran and correctly built all 4 of my projects.

At this point I had a Team Build that I could run on demand from within VS and it appeared to work great.  Now I just had to get it to run my tests as part of the build, and to run automatically on every check-in.

I knew from before that there would be issues with trying to use and maintain Test Lists in our environment since we mostly just use VSTS Dev.  This post by Buck Hodges gave me hope that there was a nice workaround available.  First things first though, I just wanted to get the Team Build running with my tests.  The easiest way to do this was to just fire up Team Suite and create a Test List to use temporarily.  I created a Test List and added all my tests to it.  Next step was to modify my Team Build to run the tests as part of the build.  In my TfsBuild.proj file I had to update the <RunTest> element to True, and add the following section in there:

<MetaDataFile Include="$(SolutionRoot)\ShopFloorTeamBuild\ShopFloorTeamBuild.vsmdi">

This tells Team Build to run the Test List BVTList as part of the build.  I fired up my build and noticed that the Unit Tests were all working correctly, but all my acceptance tests were failing.  I had expected this to happen, as there is no Web Server running on the build machine for the tests to hit against.  To fix this I had to add the <AspNetDevelopmentServer> attribute to all of my acceptance tests and add a line of code that performs a little magic on the client-side to redirect the invocation to the proper address.  More info on how this works can be found here.  This is how my code looks for one of my acceptance tests:

<TestMethod()> _
<AspNetDevelopmentServer("MyWebService", "%PathToWebRoot%\MyWebService")> _
Public Sub MyAcceptanceTest()
    Dim Target As MyWebService = New MyWebService
    WebServiceHelper.TryUrlRedirection(Target, Me.TestContext, "MyWebService")
End Sub

I ran into a little trouble getting this working correctly.  It seemed that I could get it to work either on my local machine, or on the build server, but not both.  It seemed that they were interpreting the %PathToWebRoot% variable differently.  After a little more research I discovered that I can set it up so that it works on the build server, then for the local machine there is a local setting I can change to override where %PathToWebRoot% will point to locally.  It can be overridden in Tools->Options->Test Tools->Test Execution, the Web Application root directory property.

At this point my Team Build is building on demand, and properly executing all of my tests.  I'm getting close now, I can almost taste it.  Time to try and make it run without the test list.

My first attempt was to follow the directions given by Buck Hodges in this post.  I downloaded the task and targets file and set them up on my build server.  Then I modified my TfsBuild.proj file like so:

<MetaDataFile Include="$(SolutionRoot)\ShopFloorTeamBuild\ShopFloorTeamBuild.vsmdi">
<TestContainerInOutput Include="ShopFloorTests.dll" />

I tried running the build and all the tests run, however all my acceptance tests are now failing.  After a little investigation I discover that a config file that all the acceptance tests rely on is not getting copied to the tests directory when the tests are run.  I had previously used the .testrunconfig file to specify that this file should be deployed along with the tests.  It appeared that the <TestContainerInOutput> wasn't using my .testrunconfig file (this was confirmed in the comments from Buck's Post).  I was doing some fiddling around with the TfsBuild.proj file, and I had commented out the <TestContainerInOutput> tag, and uncommented the <MetaDataFile> tag but deleted the <TestList> part of it.  It appeared that it was still running all my tests, even though I wasn't specifying what test list to use.  So now my TfsBuild.proj looks like this:

<MetaDataFile Include="$(SolutionRoot)\ShopFloorTeamBuild\ShopFloorTeamBuild.vsmdi">
<!--<TestContainerInOutput Include="ShopFloorTests.dll" />-->

All the tests run and pass.  I tried removing a test from the project (not touching the vsmdi file) and re-running the build, and sure enough it didn't try to run the deleted test.  I also tried adding a new test, and the build picked that up automatically also (without me needing to modify any test lists).  I'm not sure what I did, but I could swear it never used to work like this.  Maybe it's a result of installing Buck's new task/targets files.

The next step is to get this build running automatically on every check-in.  I followed the approach used by Jeff Atwood here.  That turned out to be pretty straightforward and easy.  Now I just need to setup something so that we can be notified when the build is complete.  After a little reading I discovered the Alerts system built into team foundation server (I've never really noticed it before).  All I needed to do was go Team->Project Alerts...  check off the "A build completes" alert and type in my email.  Every time the team build completes, I now get an email with a link to a build results webpage.

So far so good.  I have my team build running automatically on every check-in.  It compiles all my code and runs all my tests.  It sends me a notification email when it's complete.  That's all the important stuff working.  I still have to play around with getting Code Coverage working in the automated build.  And I would really like to get a build report that just shows me the differences between that build and the previous build as I talked about here.  The most important features of a CI system though are all up and running.

I know the TFS Build team at MS is working hard on the next version, and hopes to make alot of improvements.  I'm hoping that this process will be a little more straightforward in the next version of TFS, and that there will be some added features (such as diff reports between builds, and CI out of the box) that enable us to be more productive and effective at what we do.

Posted on Wednesday, November 22, 2006 3:08 PM | Back to top

Comments on this post: Continuous Integration with Team Build - Up and Running

# re: Continuous Integration with Team Build - Up and Running
Requesting Gravatar...
Oh, and if you like taskbar notifications for CI builds, you might want to check out the following:

Left by Buck Hodges on Nov 22, 2006 10:04 PM

Your comment:
 (will show your gravatar)

Copyright © Dylan Smith | Powered by: