Thoughts on Continuous Deployment

Eric Ries posts about continuous deployment at O’Reilly Radar. This is the second post on this approach I’ve seen lately, and I find it to be something interesting to think about, if not to be adopted.

Point five in his five step plan for migrating to continuous deployment is root cause analysis, or “the five whys”. He has a whole post just on that topic that’s definitely worth reading. The case for using root cause analysis is self-evident, the rest of the continuous deployment process is more interesting to discuss.

The basic idea is simple: deploy changes to the application as rapidly as possible, where the definition of “as possible” means “without breaking things”. This is a technique that clearly works for big sites. What I’d really like to do is spend a day or a week watching this process unfold, because it leads me to all sorts of questions.

My first question is, how does quality assurance play into this process? The process Eric describes involves lots of automated testing, but most shops have testers as well. Do they test on the production system as changes go live?

The second question is, what kind of version control strategy is being used to back up this process? I’d expect that each developer has their own branch, and that they merge from the production branch to their branch constantly, and only merge back to the production branch when they have a change they are ready to publish. That way developers can commit code frequently without worrying about it being deployed by mistake.

My third question is whether this process is more suited to some kinds of applications than others? Let’s take World of Warcraft for example — they have a big, heavy release process that involves lots of internal testing and lots of customer testing before important releases go out the door. Even with exhaustive testing, after every release there’s a round of tweaking to deal with all of the unintended consequences of their changes. A change to one constant in World of Warcraft (say, the amount of attack power a warrior gets per point of strength) has changes that ripple through the entire game. It also seems like they’d have to group changes to see how those changes interact with one another without inflicting them on players first.

They could try continuous deployment to the Public Test Realm, but everything that happens there is reported to and dissected by the player community. It’s risky.

I’d love to read a whole lot more about this kind of process and how it scales. It’s pretty much how everybody works on small applications that they maintain by themselves, but I’m very curious to hear more about how it works for big teams.

4 thoughts on “Thoughts on Continuous Deployment

  1. Some people from BioWare gave a talk last week (at GDC), about how they’re using an incremental approach in developing Mass Effect 2. It’s a game that will take over two years to make; I think they could release it in smaller chunks if they wished, but nothing like the continuous deployment that the traditional agile literature talks about. I was very impressed by their intelligent approach to resolving that tension; I wrote it up at http://malvasiabianca.org/archives/2009/03/gdc-2009-friday-bioware-talk/ if you’re interested.

  2. I briefly talked to Eric on Wednesday after a workshop he ran at web 2.0, and he seemed pretty cognizant of the fact that “continuous release” is difficult for non-hosted, non-web applications where deployment and installation costs are high. I think his point is really that minimizing release cycles even to the point of making them daily (if you can) is a good idea. Reducing the costs of deployment to make that easier is a good idea too.

  3. Oh, hey, he mentions the switching symlinks trick for rapidly rolling back new releases. That was something we did at SFI back in … 1999-2000 or so. I’m sure it wasn’t new then. But I guess it’s still not all that well-known even though it’s very simple.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>