There’s a big gaping hole in my experience when it comes to software engineering, and that’s coping with scalability issues. I suspect that this is true for many software engineers. I’ve written lots of applications and seen them deployed, but only rarely have I run into problems with scalability, and most of those were trivial to fix (bad SQL is a common culprit).
The thing is, you can only get experience with this class of problems by being out in the real world and building applications that outstrip their capacity in terms of processor usage, or bandwidth, or data storage. On the application I’m currently working on, the database is getting fairly large, and I expect it to get a whole lot bigger over the rest of the year.
I’m already running into problems with latency on some queries. The problems initially started showing up when I was running certain queries with Hibernate, and I’m reading Hibernate in Action to make sure that I’ve configureed it optimally (and to bone up on my Hibernate knowledge in general). Now some reports are getting slower, and it looks like I’m going to have to turn myself into a MySQL DBA. Off I go to order High Performance MySQL so that I can make sure our database is running like it should.
Even though these kinds of problems are a pain, it’s also pretty exhilerating to deal with this sort of thing. For one thing, it means that the application is seeing lots of use, and for another, these are the kinds of problems that are interesting to solve, because they’re like detective work. Right now, I’m in the phase of problem solving where I’m trying to figure out the best way to instrument the application. There’s no sense in turning the knobs until I get an idea of exactly what I’m dealing with.