Skip to content

The 12 Steps of Continuous Integration

March 19, 2009

I recently gave a talk on Continuous Integration with Hudson. I’m planning on posting a number of how-to guides covering everything that I did, but to start with, here’s the core of the talk – The Twelve Steps

  1. Maintain a code repository
  2. Automate the build
  3. Make your build self-testing
  4. Everyone commits early & often
  5. Everyone expects their commits to succeed
  6. Every commit (to mainline) should be built
  7. Keep the build fast
  8. Test in a clone of the production environment
  9. Make it easy to get the latest deliverables
  10. Everyone can see the results of the latest build
  11. Keep the build “successful” – if you break the build, your first priority is fixing it
  12. Automate Deployment

Obviously, I’ve borrowed heavily from Martin Fowler, but as every recovering alcoholic knows, he’s two steps short. My additional steps are…

5. Everyone expects their commits to succeed


11. Keep the build “successful” – if you break the build, your first priority is fixing it

…and I also modified…

4. Everyone commits early & often

…which is probably of greater importance than both 5 and 11, even though it’s only half a line.


In his original, Martin says “Everyone commits every day” (but you already knew that, because you followed the link right?).

The problem is, I’ve worked with people who thought that “every day” meant “at the end of every day, regardless of the state of your code”. Sorry, but no. The whole point of CI is that you have an up-to-date, working system at all times. Once it becomes acceptable to check in broken code, then the whole thing falls apart.

Which also explains why I’ve added both 5 and 11 – whilst, in his prose, Martin extols the virtues of keeping the build working, none of his 10 points explicitly tell you to do that.

And who’s going to read the whole article when you’ve a list of bullet-points right at the beginning to scan through?

10 Comments leave one →
  1. March 20, 2009 1:35 pm

    Firstly, I agree with you.

    But agreeing is boring so here’s the opinion an ex-colleague of mine (even though I disagree) just to get a discussion going…

    “Why should I verify that the code passes the tests before I check it in? Our tests are slow and I can’t be bothered, I’m sure it’ll pass. Anyway, that’s what CI is for, to run our tests for us, if they fail I’ll check in a fix and it can run the tests again”

    To be honest, I can see where he’s coming from (computers are our slaves right?), but a broken build is the thing to avoid at all costs (the needs of the many are greater than the needs of the few).

    Currently, we do this:

    developer writes code
    developer runs tests
    developer checks in to VCS
    CI server builds code from VCS
    CI server runs tests

    I quite like the model proposed by people like TeamCity:

    developer writes code
    developer sends changes to CI server
    CI server builds source with changes
    CI server runs tests
    CI server commits your code IFF tests pass
    That way, you know the code is source control is not broken.


  2. March 21, 2009 10:50 pm

    I think that, yet again, Mr Burns has blessed us with two comments in one…

    So, “Our tests are slow“… You read #7, right?

    Damn, gotta go. I’ll make point number two tomorrow…

  3. March 22, 2009 12:43 pm

    So, with CI, you make sure that each piece of work is in and working before you move onto the next task. But people are stupid and lazy, so that’s only gonna happen if CI is both the carrot and the stick.

    CI, as a carrot, is easy and fast and fun to do. You want to check your code in. You want to see your changes in the log/timeline and the “Success” tag next to them. You want to dry-run the tests locally before check-in, to make sure that you keep the build successful. And all of that is easy and fast, because the build is fast, and you only run Unit Tests that test one object at a time, and you decompose your software into coherent modules, each of which compiles to its own jar, so that when you touch one bit you only have to verify one bit, and the machine only has to build and test one bit, and the whole thing just works.

    CI, as a stick, sends out messages to all of your team, telling them how you broke the build, and how they won’t be able to get the warm glow of Build Success for any of their work until you’ve cleaned up you own mess. People are stupid and lazy, but they don’t like other people knowing just how stupid and lazy they are. The more promptly, visibly and accurately CI can beat them with the “don’t be stupid and lazy” stick, the more success you’ll have in beating them into the CI way. Once again, the faster the CI cycle (i.e. the faster the project is to build and test), the more prompt, accurate and visible CI can be.

    Unfortunately, for some (many?)  people, it’s not that CI of itself is sufficiently carrot or stick-like, it’s just that they have the experience of working on projects where integration was long and hard and painful and so CI is really just an “absence of stick” – what motivates them is the fact that they’re not going through all the pain that they did back on Project X. That’s fine, and great, and it works… for them. But what happens when Little Johnny joins the project fresh from college, and doesn’t have all those experiences? How do you explain to Little Johnny the pain and horror and misery and keyboard-face-mashing in such a way that makes him understand? How do you motivate Little Johnny into writing good, comprehensive Unit Tests when “all the tests pass all the time anyway, and I’ve only made a little change”?

    I don’t know about the last one (or, at least, I have some ideas but this comment is already long enough), but if you’re going to have any hope with either of the first two, you’ve got to keep the build fast.

    The 12 Steps of CI are in a vague order of implementation, but if I had to put them in order of importance, that one’s got to be somewhere near the top of the list.

  4. March 22, 2009 1:01 pm

    So, with CI, you make sure that each piece of work is in and working before you move onto the next task. But people are stupid and lazy, so you’ve made your build fast and easy to dry-run prior to check-in.

    But why bother with the dry run? Why not have CI “run our tests for us, [and]  if they fail I’ll check in a fix and it can run the tests again“? In fact, why not have the “CI server run the tests, [and] commit your code if the tests pass”.

    Well, I can think of at least two good reasons.

    Remember that you make sure that each piece of work is in and working before you move onto the next task.

    It’s never going to be faster to check the code in and let the CI server run your tests than it is to dry-run the tests locally. If it is, then you’re not investing in your developers, and that’s just wrong (and a whole other post). If (check-in + check-out + server compilation + server tests + email result) < (developer dry-run) then you need to buy your developer a new machine.

    And if it’s slower (and it will be), then that’s even more opportunity for the developer to start the next task before verifying that the changes for the previous task are in and working.

    Even worse, letting the CI server do it actually frees up the developer to start the next task. Which he will, because people are stupid and lazy.

  5. March 22, 2009 2:37 pm

    So, having got all that off my chest, it’s time to see what TeamCity have got to say for themselves. You’ll be pleased to know that, up until about 5 minutes ago, I hadn’t followed that link. Unless you’re Matt, you probably haven’t either, in which case I’ll give you a minute to check out their diagram.

    All done?


    Where do I start with this? How about…

    “Verify Remotely – This is handled by TeamCity so you can continue coding”


    “Work never stops”


    “Typically, your team submits code to Version Control BEFORE verifying that it works, and risks breaking the build every single time”

    That’ll do.

    No. Typically, my team is motivated either by the carrot of good coding, or the stick of looking like an idiot in front of everyone else. Typically, my team submits working code to version control, that they’ve tested locally and expect to integrate successfully. Don’t forget:

    5. Everyone expects their commits to succeed

    So that’s one areas where TeamCity and I (/CI best practice) differ. What about remote verification? Well, I’ve already told you what’s going to happen if people are free to continue coding whilst their latest check-in gets verified – they’re going to continue coding whilst their latest check-in gets verified! More specifically, they’re going to continue coding something else, and so think about something else, so that when their original task proves unfinished (and TeamCity spits their check-in back at them) they’re going to have to throw away the something else they were thinking about, and switch back to the something they should’ve finished in the first place.

    Bleugh. Give me…

    • Think about somthing.
    • Do something.
    • Finish something.
    • Have a cup of tea.
    • Repeat as required.

    …any day of the week.

    And finally, as for “Work never stops”… Gee, thanks TeamCity. My manager may love you for it (time at desk == value of work, right?) but I certainly won’t…

  6. September 16, 2009 9:33 am

    Matt pointed me at this article, which talks about “The Coffee Cup Metric for Continuous Integration”. I mean, tea is clearly much better than coffee, but it’s nice to know that I’m not alone…

  7. Ben Lidgey permalink
    November 30, 2010 11:53 am

    At my last place we installed the Hudson Continual Integration Game plugin ( which was quite a fun way of quantifying the use of Hudson.

    It started off encouraging people to check their code before committing it (so they gained points rather than losing them when the build broke). Some people who did the bigger, chunkier refactoring of the builds lost points. People then realised if they did things like “fixing” stuff like Checkstyle errors (by “fixing” in quotes I mean not necessarily doing a proper job) they could get loads of points and it all started to get a bit silly.

    If people can use the game as a bit of a fun way to focus them on producing stable, tested code before they check in then it can work well.

  8. November 30, 2010 9:43 pm

    Yes, we installed that plug-in too, and with similar results. At the time I felt that, given a re-weighting of the points system, it might be possible to make the game sustainable/useful over the long term. Since then, I’ve used information radiators, and I think that perhaps these give a better means to a similar end (I’m not sure that the long-term nature of the game is necessary, just a short you-broke-the-build kick up the bum). Ultimately though, I’m not sure that there’s any substitute for bright, enthusiastic people who are willing to take these tools and use them in the right way.

  9. December 6, 2010 2:57 pm

    i didn’t like that game. It was fun at first, but like you say, it made people do odd things (/be scared to Do The Right Thing).
    In my head, I chuck that game alongside other code metrics like emma and test coverage. It seems like a good idea but it changes people’s behaviour. It may change a bad programmer’s behaviour (like mine) for the better, but bad programmers are a different (and bigger) problem. I think good programmers _shouldn’t_ use them. Pair programming and peer review should be enough to keep you focussed on doing the right thing.

  10. December 8, 2010 1:18 pm

    Good programmers are like good drivers…

    • Everyone thinks that they are one.
    • People tend to concentrate on comparing one driver to another, without considering how a driver’s ability varies over time.

    Yes, on a good day, I think I’m a good driver. But if I’m tired, or angry, or distracted, or… then I can be just as poor a driver as the next guy (if not worse).

    Code metrics aren’t there for the bad programmers, to be thrown away once everyone on the team is awesome – they’re there for the bad days, when you’ve got a stinking cold, or your daughter woke you up at 4am and refused to go back to sleep. Those are the days when you want your automated tests and metrics to say “whoa, no unit tests?” or “hang on, what about all these checkstyle errors?”.

    Yes, there’s a balance; teams should aim for a simple, automated set of metrics that are both easy to apply and easy to understand. If (collectively) you don’t think that its appropriate to mandate a certain level of code coverage then fine, don’t use emma (or rather, don’t let emma fail your build).

    It’s wrong to approach this from the angle of “code analysis tools are cool, let’s grab as many as possible and apply them to our build”. The drive needs to come from a code-quality perspective first – What problems are we having (with maintenance, deployment, testing, security etc)? What coding mistakes are causing these problems? Can we detect these errors and highlight them, whilst keeping false-positives low?

    It’s likely that as part of such a code quality improvement drive, you’ll try out some tools and see what they find – espeically if your concerns are security-related. But you should always try to distill it back to the simpliest set of tools possible, which give you the quick feedback you need to keep you code on-track.

    Programming is hard. To write something that’s extensible and maintainable, that fulfils the customers requirements (which almost certainly won’t be what they asked for) whilst being readable and intuitive is no easy task. Tools can’t tell you if you’re doing to right thing, any more than the compiler can – they can only tell you if you’re wrong, or doing something that is likely to fail. But I’d still rather have them as part of my process.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: