O’Reilly Radar’s best and worst tech of the decade inspired me to think about trends in software development. In the past decade, one prevailing trend has been the migration of process in software development from management to individual developers. Traditionally, process has been imposed on the development team, who usually ignore the process or actively resist it. Or in many cases there just isn’t any process at all.
Traditionally, “process” has been associated with project managers. Most processes had to be applied at the team level in order to work. Whether it was old school process like Waterfall or newer, agile processes like Scrum or Extreme Programming, everybody on the team has to participate for the processes to work. These processes are heavy, difficult to implement, and in many cases, completely ineffective.
The most powerful contribution of Extreme Programming, though, was that it was much more developer-friendly than the other approaches, and it really arrived as a bundle of specific practices. XP made it easy for developers to pick the practices that they liked best from the whole collection and start applying them, whether or not management bought into XP as an approach. So we’ve seen specific XP components like test-driven development, refactoring, continuous integration, and small releases really find democratic support among programmers.
A second major factor in encouraging developers to implement their own processes has been the growth of open source software. For one thing, it’s proven to be a laboratory for a lot of different approaches to building software, and for another, all of those approaches were implemented and used by developers themselves, and so they all work for at least some developers. The growth of open source software has also contributed a powerful set of programmer-friendly tools for implementing a development process, including version control, issue tracking, unit testing libraries, automated testing tools, continuous integration servers, project management applications, and more.
The third major change we’ve seen in the past decade or so has been the growth of blogging. People talk a lot about the effect of blogging on other fields, but its effect on software development has been profound. There were ever many magazines that were aimed at working software developers. Back in the day, there were magazines like Information Week, InfoWorld, and PC World that were for IT management — not for the people doing the real work. Developers had Usenet and mailing lists, but blogging led to an explosion in the amount of information that was written by programmers for programmers.
All of these factors contributed to developers stepping up and taking ownership of the process of developing software. These days, you see lone developers implementing a somewhat formal process for building software. Ten years ago, it was tough to find programmers who wrote unit test. Now there are a lot of programmers out there writing unit tests even when people aren’t looking. Programmers are taking ownership of the full life cycle of software development like never before as well. Take a look at some of the more recent writing on continuous deployment — it covers everything from requirements to maintenance to monitoring of running applications. More work done in public, public discussion of that work, and the introduction of new best practices have defined the trend of the decade — developers owning the processes under which they work. Hopefully it will continue in the next decade.
December 26, 2009 at 11:08 pm
Great article – I totally agree on all points, especially the role that blogging has taken.
December 28, 2009 at 2:01 pm
12 years ago the Software Engineering courses in my CS degree were concerned mostly with process, but the semi-official message of the classes was “This is an annoying waste of time you will be forced to participate in that will help neither you nor your managers”.
The problem is that cargo-cult imitation of engineering practices is fine IF you have the time to do it – software developed that way will be extremely reliable but it will take forever. So, for instance, the software for the Space Shuttle is famous as an example of something developed that way.
XP said hey wait a minute, we’re computer programmers and what we’re testing is software, it is a mathematical object and it is malleable, we can reconstruct it using mathematical transforms that do not change the functioning of it, just the structure; it is fast to run and we can test it without physically constructing anything; we’re smart, we can communicate concepts among ourselves without long meetings; we know that without regular releases we can get far off track.
Actually XP didn’t really tell me or the people I talked with many of those things. We already knew them. But it did codify them.
Religious adherence to process in situations where innovation is the goal is never going to work. What XP (etc) did was find some things that actually do work in that situation and put a stamp of approval on it.
In general the world of software is far too attached to formality, cargo-cult imitation of either formal mathematics or mechanical engineering. Those of us exposed to Perl and shell scripting and subsequent developments understand that it doesn’t have to be like that, but conventional wisdom about what is or is not acceptable still prevails.
I’ve been working mostly on my own for years, and the most valuable thing for me has been the arrival of refactoring tools. (These are especially good if you’re working on your own!) I do a fair amount of automated testing but the majority of what I’ve been doing has been what you might call exploratory, working out ideas into code and then refining them, so test-first doesn’t really fit. I use a ticket tracker and aim for (don’t always achieve) continuous releasing.
As much as the rise of lightweight process has been the apparent demise of heavyweight process at least in the world I deal with. If the choice is running code or your own weight in paper specifications, well, ever tried selling a specification? Right.