Geeks With Blogs
John Conwell: aka Turbo Research in the visual exploration of data

I've spent a lot of time lately thinking about instrumentation and how to integrate it into software projects.


As a performance engineer I tend to think about instrumentation from the point of view of someone who wants to record the details of what a system is doing, and then dig through the data and use it to figure out what is wrong.


But I’ve been talking to people the past few months about instrumentation, I’ve come to realize that instrumentation means different things to different people.  Some people think of instrumentation as a high level, light weight set of metrics that are easy to consume, understand, and extrapolate performance deltas; a management point of view.  Other people, like me, think of it as recording low level details of what’s going on in the call stacks and sql engine; a trouble shooter point of view. And then others think its somewhere in between; everyone else.


Well, I think everyone is correct.  There are different levels of instrumentation that are useful at different points in validating system health.  There should be easy to consume and understand metrics to validate day to day health checks, there is medium level detail instrumentation that is used to figure out where a problem is, but takes a bit more effort to analyze.  And if that isn’t enough to find and fix the problem, there is the dump everything to file model that gives you all the data you need to understand what is going on in the system, but requires internal knowledge of the system and time to analyze the data.  Also, each level builds upon the other, so there is as little duplicated effort as possible.


So I’ve tried to create an instrumentation model demonstrate these different levels, the answers each level tries to answer, and when you move onto the next level


The first level will provide you with the most early bang for your buck, and it’s a easy way to tell if you have a problem, with as little dev effort as possible.  Then as you get the high level metrics in, you can start building in the mid level metrics, and so on.  The main thing is to not try and build the entire instrumentation framework up front before you put anything it.  Start putting high level metrics in early and use then in your automated testing

Instrumentation Model Image

Posted on Wednesday, March 26, 2008 10:02 PM Code Performance | Back to top

Comments on this post: The Instrumentation Model

No comments posted yet.
Your comment:
 (will show your gravatar)

Copyright © John Conwell | Powered by: