software development – Page 5

Tag: software development (page 5 of 16)

Security is a cost

At work, we’re switching things to encrypt a lot of information in our databases for security reasons. The project has been time consuming and painful, and in the end, our database is far less usable from a developer’s standpoint than it was before. Soon the days when I can quickly diagnose issues on the production system with a few well-placed SELECT statements will be a thing of the past.

As far as the implementation goes, I’ll tell Hibernate users who want to implement an encryption system that there’s only one way to go — UserTypes. Don’t bother with anything else.

What this project really has me thinking about, though, is the high cost of security. It ties into something from the Bill James interview that I linked to the other day. Here was his response to the question of whether we overestimate or underestimate the importance of crime:

We underestimate it, because it’s our intent to underestimate it. We only deal with it indirectly. We all do so many things to avoid being the victims of crime that we no longer see those things, so we don’t see the cost of it. Just finding a safe place for us to have this conversation, for example — we needed a quiet place, but before that, we needed to find a safe place. A hotel lobby is what it is because of the level of security. I’ve checked out of this hotel, but I’m still sitting here in the third-floor lobby, because it’s safe. When you buy something, it’s wrapped in seven layers of packaging in order to make it harder to steal.

I think that people are generally excessively afraid of crime but underestimate the day to day costs that crime imposes. In software engineering, we spend a lot of time and effort on security. If everyone were honest, we wouldn’t need passwords, encryption, or any of the other stuff that occupies a lot of time on every project. We’d still need to take precautions against damage caused by user error, but most of the hours we spend on security could be spent on other things.

The other cost of security, beyond implementation time, is the ongoing cost related to the inconvenience of security. Whether it’s the time we take to unlock our screen or set up SSH tunnels or deal with the fact that we have to decrypt data in the database in order to see it, it all counts. Security is almost always a form of technical debt.

In many cases security precautions are necessary (or even mandated by law), but it’s important to be vigilant and not add more of it than is necessary, because it’s almost always painful in the moment and forever thereafter.

Why people are returning to Java

July 7, 2011 / Rafe / 7 Comments

I am a huge fan of Ruby on Rails, but I was not terribly surprised to read that as Twitter’s code base has grown, they’ve found it more amenable to move to JVM-based languages for reasons having mostly to do with encapsulation. InfoQ interviews Twitter engineer Evan Weaver about how the company’s stack is evolving as they get bigger.

On one hand, they’re moving more services to the JVM for performance reasons. When they extract components from their main code base to optimize them, they generally migrate them to the JVM:

The primary driver is honestly encapsulation, so we can iterate faster as a company. Having a single, monolithic application codebase is not amenable to quick movement on a per-team basis. So when we decide to encapsulate something, then because of our performance concerns, its better to rewrite it in the JVM for most systems, than to write a new Ruby system.

They’re also finding the rigidity of the JVM useful for productivity reasons:

And the other half is that, as search has moved into a Service Oriented Architecture and exposes various APIs, static typing becomes a big convenience in enforcing coherency across all the systems. You can guarantee that your dataflow is more or less going to work, and focus on the functional aspects.

Those are the kinds of reasons why I could not imagine rewriting the main application I work on as a Ruby on Rails application. I’m surprised he doesn’t also mention the maturity and power of the tools on the Java side. Text editors like TextMate and Vim are great, but when it comes to navigating through a large, complex code base, you cannot beat the state of the art Java IDEs like Eclipse and IDEA.

Update: Edd Dumbill lists seven reasons you should use Java again.

Getting bug fixes into the pipeline

June 9, 2011 / Rafe / 0 Comments

I don’t work at a company that does continuous deployment like Flickr, or Etsy, or Facebook. We are not close to doing continuous deployment. I’m not sure anybody at the company besides me is interested in continuous deployment. Generally speaking, when we create new features, the product team creates a spec, then developers write a short tech spec, then quality assurance comes up with a test plan, then we write the feature, test it, launch it, and then test it again.

There are many reasons why we do things this way, some of them good, some of them bad. For one thing, our product is aimed at businesses, and so our features are driven by feedback from customers and from people who work with our customers every day. As a developer, I don’t really know what sorts of features our customers would pay for. The second is that our product is a key piece of infrastructure for our customers. When we make changes that affect how it works, there’s pretty substantial risk of making our existing customers unhappy, so we have to be very careful about the changes we make to existing functionality in our product. Finally, our product and processes have been around since before continuous deployment or even automated tests were industry practices, and it’s tough to fight against history.

Anyway, one persistent problem we have is putting bug fixes for minor issues or general maintenance of the code base onto the schedule. The product people decide which features get implemented, and nobody wants to add testing time by approving additional development work that isn’t essential to whatever business requirement is the top priority at the time.

The trick is to find a way to hijack our process so that we can get more developer priorities onto the release schedule. What I realized recently is that there are gaps in our process — times when most of the team is focused on preparing for an upcoming release or we’re designing features and waiting between steps in the review process. We should be filling that time with work on other bug fixes and back end improvements.

So after discussing it with the team, we decided to create a new process. Generally we check all of the features and fixes for the release to come into trunk. Now we’ve created a new branch for each developer. They’ll claim bugs that they want to see fixed and check the fixes along with automated tests in them into their own branch. Then, as we review the bugs and fixes and our testers find time to verify those fixes, they will be merged into the trunk for the subsequent release.

If we were using Git instead of Subversion, I’d probably recommend creating a new branch for every bug fix or feature, but one branch per developer is the least painful approach with Subversion. It’ll be interesting to see how this new approach works. I get an email every night with a list of all the errors that are logged on our production servers on a daily basis, and I’m looking forward to making those email messages much shorter.

How to handle logins in 2011

June 5, 2011 / Rafe / 0 Comments

Nelson Minar suggests the proper way to handle login names in Web applications:

The right way to do logins right now on the Web is use email address as the login name and let the user choose their own display name which does not need to be unique. That’s not ideal (email addresses can change) but it works pretty well. If you absolutely have to not use email as the login name, please at least let my login name have a space in it.

Seems reasonable to me.

Jumping back into Rails with both feet

May 10, 2011 / Rafe / 2 Comments

Since February, I’ve been working on a pro bono project that launched last week. What’s in it for them is they got a brand new application that replaces an unwieldy bunch of paper and email-based processes. Before, they had a Web page that was protected by basic authentication and led to a set of form-to-mail scripts and downloadable Word documents. Now they have an application with real user accounts that can be carried over from year to year. There’s still a lot of work to do on the administrative side of the application, but I’m quite proud of the results so far.

What’s in it for me is I got to revisit Ruby on Rails after a couple of years away working primarily with Java and PHP. I also used Git, GitHub, Pivotal Tracker, and switched from TextMate to Vim for this project. The project has been very entertaining for me, aside from some stress as the release date moved closer to the early bird deadlines on some of the forms we were replacing.

I found it really easy to jump back into Rails, despite the fact that a lot of things have changed since the last time I used it. Most of the utility scripts that used to be run individually are now bundled under a single rails script, and Rails is geared much more strongly toward using resource-oriented controllers rather than developers structuring controllers in their own way, but beyond that, things are basically the same. I also found that most of the things that are different are, in fact, improved. Rails 3 handles dependencies far, far more elegantly than its predecessors did, which makes deployment much better. It’s easier than it has ever been to set up deployment using Capistrano. The testing framework is more flexible and powerful.

For some time I wondered whether Rails was going to be an evolutionary dead end in terms of Web applications platforms, but I’m very impressed at where it is as a platform right now.

I used two plugins that were a huge help in terms of getting the application built. The first is the Devise plugin, which provides user registration and authentication functionality. I’ve been very impressed with its capabilities and ease of use. The second is CanCan, which provides a role-based authorization system that integrates nicely with Devise. The two of them saved me weeks of work.

The one big mistake I made was not taking a test-first approach to development. I had intended to, but I was in a rush and I ran into an inflector bug that caused me some grief. That bug prevented my tests from working properly, and rather than tracking down the issue, I did a bunch of development without writing accompanying tests. Now I’m backfilling the tests, which is never fun.

The organization I’m building this application for uses DreamHost for hosting, and I assumed I’d be able to use their existing hosting account to deploy this application. Unfortunately, while DreamHost does support Rails, they do not yet support Rails 3. I wound up having to deploy the application on my own slice running at Linode. I considered sites like Heroku, but they were just too expensive. I had thought we’d be closer to turnkey hosting for Rails by now, but that still appears not to be the case. On the other hand, getting the Rails application up and running from scratch on my own Linux server was simpler than it has ever been.

Rails is no longer the hot young thing that all of the developers are swooning over, but I’m finding it to be more excellent than ever. I still can’t imagine building an application in PHP if Ruby on Rails is also an option. The other takeaway is that developers need to be on GitHub, period. I’m using a private repository to host this project and it’s working beautifully.

Classifying software developers

April 28, 2011 / Rafe / 3 Comments

Industry analyst James Governor takes a shot at creating a taxonomy of developers, with, I assume, people who work in developer relations in mind. I started thinking about how I classify developers, and boiled it down to positioning them on two scales.

The first is vocation versus avocation. The second is inward focus versus outward focus.

The first scale has to do with motivation. Does the programmer program because it’s their job, or because they enjoy developing software for its own sake? It’s helpful to know where your colleagues and potential hires lie on this scale, because it matters a great deal in terms of managing them. It can be difficult to motivate developers who are at the vocation end of the spectrum to learn new things or be experimental if you can’t show them in a tangible way how doing so will be better for their career. On the other hand, with developers who are at the avocation end of the spectrum to quit writing code and ship. When they pick an approach to solving problems, it’s often hard to determine whether they’ve chosen the best solution for the business or they’ve chosen the solution that intrigues them the most.

Inward focus versus outward focus has to do with how developers prefer to solve problems. When an outwardly focused developer runs into a problem, they’ll use Google, they’ll ask a coworker, they’ll post a question on Stack Overflow or on the appropriate forum. When they’re assigned a task, they’ll look for open source libraries that satisfy the requirement or they’ll look for blog posts from people who’ve attacked the same problem in the past. They’re not afraid to get other developers on the team to stand in front of the whiteboard and puzzle out the problem with them. On the downside, these are the developers who create Web sites that use jQuery and MooTools and wind up loading 25 jQuery plugins on every page of a site. They copy and paste code they find in blog posts even if they don’t actually know how it works.

Inwardly focused developers generally prefer to rely on their own brainpower as much as possible. They often times exhibit “not invented here” syndrome but on a personal level. When they are working on a tough problem they often seem to disappear completely until they’ve figured it out. It often takes them longer to solve simple problems because they don’t tap into the community to see how other people problems. On the other hand, the further you are toward this end of the scale, the more likely you are to be able to solve deep problems at all. These developers are never stuck when Google doesn’t return any interesting results related to their problem. They’re also often the only developers on the team who actually know how the entire system works. They’re the people who actually invent stuff.

Both of these scales are value neutral. A good team will have developers who fall all over the graph. Teams that are too inwardly focused often fail to incorporate industry progress into their own code and practices. Teams that are too outwardly focused have a hard time gaining a competitive advantage in terms of technology, although they can often deliver very quickly. Teams with too many developers who program for its own sake often frustrate the rest of the company for a variety of reasons. Teams with too many career-focused developers often lack creativity and usually fail to achieve excellence.

The other scale that matters is good versus bad. Falling to one side or another on either of the previously mentioned scales doesn’t make you good or bad at software development, but goodness and badness manifest themselves in different ways based on the type of developer. Identifying good and bad developers is a separate discipline, one I’d like to get better at.

EveryBlock’s response to the AWS outage

April 25, 2011 / Rafe / 0 Comments

EveryBlock’s post-mortem on the Amazon Web Services outage that took down their site sets a standard of accountability and transparency that every engineering team should aspire to. Rather than blaming AWS for their downtime, Paul Smith explains that had they followed the architectural guidelines provided by AWS, they would have been fine. Nearly all potential outage scenarios can be mitigated given sufficient resources, but it rarely makes sense to build the infrastructure to avoid all of the known outage possibilities. It’s just too expensive and time consuming. What I respect is Paul Smith’s acknowledgement that it was those choices that resulted in the site’s downtime, rather than the problems with the AWS data center.

Update: This Webmonkey post on the outage is worth a read as well.

Software engineers and computer programmers

April 22, 2011 / Rafe / 6 Comments

Here’s how the Boston Globe describes the difference between a software engineer and a computer programmer:

While software engineers develop the software, computer programmers convert the design into code that the computer can follow.

That is, I believe, the definition of a distinction without a difference. The good news is that both jobs are on the paper’s list of ten least stressful careers.

The etymology of +1

March 31, 2011 / Rafe / 2 Comments

When Google announced their new +1 feature, which enables end users to recommend things, I was mainly struck by the incredible geekiness of it. Everybody understands “Like,” whereas +1 was taken from the Apache open source community and its history probably isn’t known to even 5% of Google’s users. Kellan at Laughing Meme digs into the history and finds what is, perhaps, the first mention of +1 style voting. I love it.

The dangerous allure of one size fits all

March 7, 2011 / Rafe / 5 Comments

It seems like almost everyone with a blog is captivated by the debate over Apple’s new policies related to in-app purchases. I read at least one good post on the subject every day. The money issues are important but not really interesting. Apple is leveraging its absolute control over which applications can be installed under iOS to pry away a big chunk of the revenue from application vendors.

The one rationale for the new policy I best understand is that application vendors will modify their pricing model so as to pay Apple the smallest amount possible. So if Apple charges 30% on direct purchases through iTunes and 15% for in-app purchases, many developers will distribute their application for free and then unlock the good features through an in-app purchase. If the percentage is different for subscriptions and for standalone in-app purchases, developers will try to switch to subscription-based pricing. In that sense, Apple has a strong incentive to charge the same price across the board.

What really interests me, though, is Apple’s false confidence in the idea that one payment system will actually work for everyone. Chris Adamson explains why this won’t work:

A client of a client of mine is likely to get caught up in this I-AP drama, and in a meeting this week, we laid out exactly how I-AP works, and what they have to do in order to implement it, including entering every product into the iTunes Connect web interface, a nightmarish prospect when you have thousands of SKUs. When we finished, there was a long silence on the phone, followed by a colleague saying “you can probably imagine the look on everyone’s faces here.”

I’m sure that the iOS team at Apple feels that they have designed an elegant and powerful payment system, maybe the best that anyone has ever created. But it’s apparent that not only is such a system insufficient for any application that might be conceived in the future for iOS, it’s also insufficient for many applications that already exist today.

It strikes me that the core error was when Apple allowed itself to be convinced that a one size fits all payment system would work for the full iOS ecosystem. I do wonder whether it was an executive decision that was passed on to engineers to implement, or the product team came up with a solution that the executives decided could work for everyone.

rc3.org

Strong opinions, weakly held