Getting bug fixes into the pipeline

I don’t work at a company that does continuous deployment like Flickr, or Etsy, or Facebook. We are not close to doing continuous deployment. I’m not sure anybody at the company besides me is interested in continuous deployment. Generally speaking, when we create new features, the product team creates a spec, then developers write a short tech spec, then quality assurance comes up with a test plan, then we write the feature, test it, launch it, and then test it again.

There are many reasons why we do things this way, some of them good, some of them bad. For one thing, our product is aimed at businesses, and so our features are driven by feedback from customers and from people who work with our customers every day. As a developer, I don’t really know what sorts of features our customers would pay for. The second is that our product is a key piece of infrastructure for our customers. When we make changes that affect how it works, there’s pretty substantial risk of making our existing customers unhappy, so we have to be very careful about the changes we make to existing functionality in our product. Finally, our product and processes have been around since before continuous deployment or even automated tests were industry practices, and it’s tough to fight against history.

Anyway, one persistent problem we have is putting bug fixes for minor issues or general maintenance of the code base onto the schedule. The product people decide which features get implemented, and nobody wants to add testing time by approving additional development work that isn’t essential to whatever business requirement is the top priority at the time.

The trick is to find a way to hijack our process so that we can get more developer priorities onto the release schedule. What I realized recently is that there are gaps in our process — times when most of the team is focused on preparing for an upcoming release or we’re designing features and waiting between steps in the review process. We should be filling that time with work on other bug fixes and back end improvements.

So after discussing it with the team, we decided to create a new process. Generally we check all of the features and fixes for the release to come into trunk. Now we’ve created a new branch for each developer. They’ll claim bugs that they want to see fixed and check the fixes along with automated tests in them into their own branch. Then, as we review the bugs and fixes and our testers find time to verify those fixes, they will be merged into the trunk for the subsequent release.

If we were using Git instead of Subversion, I’d probably recommend creating a new branch for every bug fix or feature, but one branch per developer is the least painful approach with Subversion. It’ll be interesting to see how this new approach works. I get an email every night with a list of all the errors that are logged on our production servers on a daily basis, and I’m looking forward to making those email messages much shorter.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>