Being that we live in a DevOps world and all, the application I work on sends me texts when it looks like things are going wrong in production, even though I am a developer and not a systems administrator. For months, I have been mildly annoyed that these texts originate from a variety of phone numbers preventing my phone from categorizing them nicely and forcing me to delete them individually to keep my texts organized.
The texts have been sent through the email-to-text gateway provided by AT&T. My scripts email a message to my_number@txt.att.net and it arrives as a text message on my phone. Unfortunately, there’s no way to control the originating phone number when you use that approach.
It occurred to me that I may be able to use Google Voice to get around this problem. You can send texts from a Google Voice account, and I figured they had some kind of API that would let you do it. I went and set up a Google Voice account associated with my work email and then quickly discovered that there is no Web service that you can use to access Google Voice.
However, some enterprising person has created a Python library that accesses Google Voice through the regular Web interface. I did some testing and found that I could send the texts using the library. Unfortunately, I’m not allowed to install random Python libraries on our production servers. I had been hoping I could just use the curl utility to send the texts so that I could avoid having to ask our systems administrator to install anything but the Python library was the only option.
Fortunately, I have my own virtual server, so I wrote a simple Python CGI script that sends the texts and installed it on that server. Here’s the source:
#!/usr/bin/env python
import cgitb, cgi
from googlevoice import Voice
from googlevoice.util import input
cgitb.enable()
form = cgi.FieldStorage()
phoneNumber = form.getfirst("phone", "")
text = form.getfirst("text", "")
if len(phoneNumber) == 10 and len(text) > 0:
voice = Voice()
voice.login("USERNAME", "PASSWORD")
voice.send_sms(phoneNumber, text)
print "Content-Type: text/html"
print
print "<html><body>SMS Sent</body></html>"
So now my monitoring script sends the alert using curl to access the CGI script on my personal server, which in turn uses pygooglevoice to access Google Voice via screen scraping, which then sends me a text from a specific phone number. I don’t rate this approach very highly in terms of robustness, but at least my texts are now properly organized.
Tim Bray laments the feeling of productivity you get when you work with tool set that you’re intimately familiar with. He talks about constantly experimenting with the next great thing as part of his job, specifically the downside:
But it means that I’ve been sort of a perpetual newbie. There’s another big piece of the software experience, one I haven’t shared in for years: where you’re concentrating on some problem and you know the codebase and tools, so your time goes more into doing and less into learning. In fact, that kind of work, in particular maintaining a large running production system, is where most of my professional colleagues spend most of their years of service.
That captures what I always liked least when I worked as a consultant. Moving from one project to the next, often on a different technology stack, meant never really feeling comfortable with what you’re working on.
On the other hand, working on maintaining a large, established system is often frustrating in its own way. You’re saddled with a code base that’s tough to change, with libraries that are often out of date, and with processes that aren’t cutting edge.
I have a friend who has worked at the same small company for nearly 10 years, and they seem to rarely update anything that currently works. I describe his skill set as being trapped in amber. He is a complete master of the way Java applications were built ten years ago, but the industry has changed a lot in that time, and he’s missed out on benefitting from those changes because the company he works for is hidebound.
Keeping an existing application up to date is difficult. I’d love to see some statistics on how many applications that were built in Rails 1 have been upgraded to Rails 2 or Rails 3.
In the end, this is one of the toughest parts of a software development career to manage. Walking the line between having the sorts of deep skills that enable you to be spectacularly productive and having a wide skill set that enables you to contribute in more situations is tough. Personally, I’ve always envied developers who are hyper-specialized. I know a guy who’s an expert on C compilers and always has a job with whatever company is trying to push gcc forward. If nothing else, he doesn’t have to sit around wondering whether he’s making a big mistake by not picking up Scala.
I just wanted to point you to a post I really enjoyed, under the innocuous name Safari Keychain Woes. In it, developer Daniel Jalkut explains how he used OS X development tools to figure out why Safari 5.1 wasn’t filling in passwords properly from his Keychain. Interesting from beginning to end and a great illustration of what you can do when you understand how things really work.
Kellan Elliott-McCrea has a great post about the high cost of false positives when it comes to building software that detects fraud, spam, abuse, or whatever. The cost of false positives is explained by the base rate fallacy. The BBC explains the base rate fallacy very well. Here’s a snippet:
If 3,000 people are tested, and the test is 90% accurate, it is also 10% wrong. So it will probably identify 301 terrorists – about 300 by mistake and 1 correctly. You won’t know from the test which is the real terrorist. So the chance that our man in the mac is the real thing is 1 in 301.
Anybody who wants to talk about screening systems without an understanding of the base rate fallacy needs to do more homework.
At work, we’re switching things to encrypt a lot of information in our databases for security reasons. The project has been time consuming and painful, and in the end, our database is far less usable from a developer’s standpoint than it was before. Soon the days when I can quickly diagnose issues on the production system with a few well-placed SELECT statements will be a thing of the past.
As far as the implementation goes, I’ll tell Hibernate users who want to implement an encryption system that there’s only one way to go — UserTypes. Don’t bother with anything else.
What this project really has me thinking about, though, is the high cost of security. It ties into something from the Bill James interview that I linked to the other day. Here was his response to the question of whether we overestimate or underestimate the importance of crime:
We underestimate it, because it’s our intent to underestimate it. We only deal with it indirectly. We all do so many things to avoid being the victims of crime that we no longer see those things, so we don’t see the cost of it. Just finding a safe place for us to have this conversation, for example — we needed a quiet place, but before that, we needed to find a safe place. A hotel lobby is what it is because of the level of security. I’ve checked out of this hotel, but I’m still sitting here in the third-floor lobby, because it’s safe. When you buy something, it’s wrapped in seven layers of packaging in order to make it harder to steal.
I think that people are generally excessively afraid of crime but underestimate the day to day costs that crime imposes. In software engineering, we spend a lot of time and effort on security. If everyone were honest, we wouldn’t need passwords, encryption, or any of the other stuff that occupies a lot of time on every project. We’d still need to take precautions against damage caused by user error, but most of the hours we spend on security could be spent on other things.
The other cost of security, beyond implementation time, is the ongoing cost related to the inconvenience of security. Whether it’s the time we take to unlock our screen or set up SSH tunnels or deal with the fact that we have to decrypt data in the database in order to see it, it all counts. Security is almost always a form of technical debt.
In many cases security precautions are necessary (or even mandated by law), but it’s important to be vigilant and not add more of it than is necessary, because it’s almost always painful in the moment and forever thereafter.
I am a huge fan of Ruby on Rails, but I was not terribly surprised to read that as Twitter’s code base has grown, they’ve found it more amenable to move to JVM-based languages for reasons having mostly to do with encapsulation. InfoQ interviews Twitter engineer Evan Weaver about how the company’s stack is evolving as they get bigger.
On one hand, they’re moving more services to the JVM for performance reasons. When they extract components from their main code base to optimize them, they generally migrate them to the JVM:
The primary driver is honestly encapsulation, so we can iterate faster as a company. Having a single, monolithic application codebase is not amenable to quick movement on a per-team basis. So when we decide to encapsulate something, then because of our performance concerns, its better to rewrite it in the JVM for most systems, than to write a new Ruby system.
They’re also finding the rigidity of the JVM useful for productivity reasons:
And the other half is that, as search has moved into a Service Oriented Architecture and exposes various APIs, static typing becomes a big convenience in enforcing coherency across all the systems. You can guarantee that your dataflow is more or less going to work, and focus on the functional aspects.
Those are the kinds of reasons why I could not imagine rewriting the main application I work on as a Ruby on Rails application. I’m surprised he doesn’t also mention the maturity and power of the tools on the Java side. Text editors like TextMate and Vim are great, but when it comes to navigating through a large, complex code base, you cannot beat the state of the art Java IDEs like Eclipse and IDEA.
Update: Edd Dumbill lists seven reasons you should use Java again.
I don’t work at a company that does continuous deployment like Flickr, or Etsy, or Facebook. We are not close to doing continuous deployment. I’m not sure anybody at the company besides me is interested in continuous deployment. Generally speaking, when we create new features, the product team creates a spec, then developers write a short tech spec, then quality assurance comes up with a test plan, then we write the feature, test it, launch it, and then test it again.
There are many reasons why we do things this way, some of them good, some of them bad. For one thing, our product is aimed at businesses, and so our features are driven by feedback from customers and from people who work with our customers every day. As a developer, I don’t really know what sorts of features our customers would pay for. The second is that our product is a key piece of infrastructure for our customers. When we make changes that affect how it works, there’s pretty substantial risk of making our existing customers unhappy, so we have to be very careful about the changes we make to existing functionality in our product. Finally, our product and processes have been around since before continuous deployment or even automated tests were industry practices, and it’s tough to fight against history.
Anyway, one persistent problem we have is putting bug fixes for minor issues or general maintenance of the code base onto the schedule. The product people decide which features get implemented, and nobody wants to add testing time by approving additional development work that isn’t essential to whatever business requirement is the top priority at the time.
The trick is to find a way to hijack our process so that we can get more developer priorities onto the release schedule. What I realized recently is that there are gaps in our process — times when most of the team is focused on preparing for an upcoming release or we’re designing features and waiting between steps in the review process. We should be filling that time with work on other bug fixes and back end improvements.
So after discussing it with the team, we decided to create a new process. Generally we check all of the features and fixes for the release to come into trunk. Now we’ve created a new branch for each developer. They’ll claim bugs that they want to see fixed and check the fixes along with automated tests in them into their own branch. Then, as we review the bugs and fixes and our testers find time to verify those fixes, they will be merged into the trunk for the subsequent release.
If we were using Git instead of Subversion, I’d probably recommend creating a new branch for every bug fix or feature, but one branch per developer is the least painful approach with Subversion. It’ll be interesting to see how this new approach works. I get an email every night with a list of all the errors that are logged on our production servers on a daily basis, and I’m looking forward to making those email messages much shorter.
Nelson Minar suggests the proper way to handle login names in Web applications:
The right way to do logins right now on the Web is use email address as the login name and let the user choose their own display name which does not need to be unique. That’s not ideal (email addresses can change) but it works pretty well. If you absolutely have to not use email as the login name, please at least let my login name have a space in it.
Seems reasonable to me.
Since February, I’ve been working on a pro bono project that launched last week. What’s in it for them is they got a brand new application that replaces an unwieldy bunch of paper and email-based processes. Before, they had a Web page that was protected by basic authentication and led to a set of form-to-mail scripts and downloadable Word documents. Now they have an application with real user accounts that can be carried over from year to year. There’s still a lot of work to do on the administrative side of the application, but I’m quite proud of the results so far.
What’s in it for me is I got to revisit Ruby on Rails after a couple of years away working primarily with Java and PHP. I also used Git, GitHub, Pivotal Tracker, and switched from TextMate to Vim for this project. The project has been very entertaining for me, aside from some stress as the release date moved closer to the early bird deadlines on some of the forms we were replacing.
I found it really easy to jump back into Rails, despite the fact that a lot of things have changed since the last time I used it. Most of the utility scripts that used to be run individually are now bundled under a single rails script, and Rails is geared much more strongly toward using resource-oriented controllers rather than developers structuring controllers in their own way, but beyond that, things are basically the same. I also found that most of the things that are different are, in fact, improved. Rails 3 handles dependencies far, far more elegantly than its predecessors did, which makes deployment much better. It’s easier than it has ever been to set up deployment using Capistrano. The testing framework is more flexible and powerful.
For some time I wondered whether Rails was going to be an evolutionary dead end in terms of Web applications platforms, but I’m very impressed at where it is as a platform right now.
I used two plugins that were a huge help in terms of getting the application built. The first is the Devise plugin, which provides user registration and authentication functionality. I’ve been very impressed with its capabilities and ease of use. The second is CanCan, which provides a role-based authorization system that integrates nicely with Devise. The two of them saved me weeks of work.
The one big mistake I made was not taking a test-first approach to development. I had intended to, but I was in a rush and I ran into an inflector bug that caused me some grief. That bug prevented my tests from working properly, and rather than tracking down the issue, I did a bunch of development without writing accompanying tests. Now I’m backfilling the tests, which is never fun.
The organization I’m building this application for uses DreamHost for hosting, and I assumed I’d be able to use their existing hosting account to deploy this application. Unfortunately, while DreamHost does support Rails, they do not yet support Rails 3. I wound up having to deploy the application on my own slice running at Linode. I considered sites like Heroku, but they were just too expensive. I had thought we’d be closer to turnkey hosting for Rails by now, but that still appears not to be the case. On the other hand, getting the Rails application up and running from scratch on my own Linux server was simpler than it has ever been.
Rails is no longer the hot young thing that all of the developers are swooning over, but I’m finding it to be more excellent than ever. I still can’t imagine building an application in PHP if Ruby on Rails is also an option. The other takeaway is that developers need to be on GitHub, period. I’m using a private repository to host this project and it’s working beautifully.
Industry analyst James Governor takes a shot at creating a taxonomy of developers, with, I assume, people who work in developer relations in mind. I started thinking about how I classify developers, and boiled it down to positioning them on two scales.
The first is vocation versus avocation. The second is inward focus versus outward focus.
The first scale has to do with motivation. Does the programmer program because it’s their job, or because they enjoy developing software for its own sake? It’s helpful to know where your colleagues and potential hires lie on this scale, because it matters a great deal in terms of managing them. It can be difficult to motivate developers who are at the vocation end of the spectrum to learn new things or be experimental if you can’t show them in a tangible way how doing so will be better for their career. On the other hand, with developers who are at the avocation end of the spectrum to quit writing code and ship. When they pick an approach to solving problems, it’s often hard to determine whether they’ve chosen the best solution for the business or they’ve chosen the solution that intrigues them the most.
Inward focus versus outward focus has to do with how developers prefer to solve problems. When an outwardly focused developer runs into a problem, they’ll use Google, they’ll ask a coworker, they’ll post a question on Stack Overflow or on the appropriate forum. When they’re assigned a task, they’ll look for open source libraries that satisfy the requirement or they’ll look for blog posts from people who’ve attacked the same problem in the past. They’re not afraid to get other developers on the team to stand in front of the whiteboard and puzzle out the problem with them. On the downside, these are the developers who create Web sites that use jQuery and MooTools and wind up loading 25 jQuery plugins on every page of a site. They copy and paste code they find in blog posts even if they don’t actually know how it works.
Inwardly focused developers generally prefer to rely on their own brainpower as much as possible. They often times exhibit “not invented here” syndrome but on a personal level. When they are working on a tough problem they often seem to disappear completely until they’ve figured it out. It often takes them longer to solve simple problems because they don’t tap into the community to see how other people problems. On the other hand, the further you are toward this end of the scale, the more likely you are to be able to solve deep problems at all. These developers are never stuck when Google doesn’t return any interesting results related to their problem. They’re also often the only developers on the team who actually know how the entire system works. They’re the people who actually invent stuff.
Both of these scales are value neutral. A good team will have developers who fall all over the graph. Teams that are too inwardly focused often fail to incorporate industry progress into their own code and practices. Teams that are too outwardly focused have a hard time gaining a competitive advantage in terms of technology, although they can often deliver very quickly. Teams with too many developers who program for its own sake often frustrate the rest of the company for a variety of reasons. Teams with too many career-focused developers often lack creativity and usually fail to achieve excellence.
The other scale that matters is good versus bad. Falling to one side or another on either of the previously mentioned scales doesn’t make you good or bad at software development, but goodness and badness manifest themselves in different ways based on the type of developer. Identifying good and bad developers is a separate discipline, one I’d like to get better at.
© rc3.org. Powered by WordPress using the DePo Skinny Theme.