Amazon’s misplaced faith automated anti-fraud algorithms

Amazon has gotten a lot of bad publicity today because they canceled the account of a customer named Linn and deleted all of the content on her Kindle because her account was flagged by a fraud detection algorithm that linked her account to an account associated with fraudulent activity. Let’s look at what went wrong.

First, a lot of the coverage is focused on DRM. This is the risk of purchasing DRM-protected content. Amazon was able to revoke her access to material that she previously purchased because of the DRM. That’s bad. DRM is bad. Don’t buy books protected by DRM.

What interests me as a software engineer, though, is the fraud-detection part of the equation. Using algorithms to identify related accounts is pretty standard stuff. Amazon is closing fraud-related accounts, and then apparently running an algorithm that finds related accounts and closing them as well. The problem with any algorithm like this is that false positives are inevitable. Some number of accounts identified as being related will actually be unrelated.

Given that this is a foreseeable outcome of any algorithm that performs this sort of categorization, Amazon’s business policies should reflect this. For one thing, they shouldn’t be automatically suspending accounts based on the results of this check alone. It’s incredibly hostile to customers. Furthermore, the responses from customer service reflect an absolute faith in an algorithm that is certain to be imperfect. That’s bad business.

If a business is going to use an algorithm-based approach to fraud problems like this, there’s got to be an understanding of the limitations of such a system. When you ignore that fact, you run into public relations disasters like the one Amazon encountered today.