Amazon has gotten a lot of bad publicity today because they canceled the account of a customer named Linn and deleted all of the content on her Kindle because her account was flagged by a fraud detection algorithm that linked her account to an account associated with fraudulent activity. Let’s look at what went wrong.
First, a lot of the coverage is focused on DRM. This is the risk of purchasing DRM-protected content. Amazon was able to revoke her access to material that she previously purchased because of the DRM. That’s bad. DRM is bad. Don’t buy books protected by DRM.
What interests me as a software engineer, though, is the fraud-detection part of the equation. Using algorithms to identify related accounts is pretty standard stuff. Amazon is closing fraud-related accounts, and then apparently running an algorithm that finds related accounts and closing them as well. The problem with any algorithm like this is that false positives are inevitable. Some number of accounts identified as being related will actually be unrelated.
Given that this is a foreseeable outcome of any algorithm that performs this sort of categorization, Amazon’s business policies should reflect this. For one thing, they shouldn’t be automatically suspending accounts based on the results of this check alone. It’s incredibly hostile to customers. Furthermore, the responses from customer service reflect an absolute faith in an algorithm that is certain to be imperfect. That’s bad business.
If a business is going to use an algorithm-based approach to fraud problems like this, there’s got to be an understanding of the limitations of such a system. When you ignore that fact, you run into public relations disasters like the one Amazon encountered today.
The fraud ratchet
I want to write a bit about businesses that make their money through fraud, inspired by Jon Bell’s post The Graph That Changed Me. In it, he talks about RealNetworks. RealNetworks was one of the first companies that provided streaming media infrastructure. They created proprietary streaming audio and video prodocols. They offered a free version of their client, and tried to make money by selling licenses for premium versions of the client and their streaming server. More importantly, they were pioneers in bundling unwanted software with their client downloads in exchange for cash.
As Bell’s post points out, the money they made this way was a substantial part of Real’s business. While people at Real hated the shady business, they were in, their jobs were also dependent on it. Bell’s manager showed him a graph with a big dip in the middle and then explained the implications:
The ratchet effect is one of my favorite metaphors, and it applies perfectly to companies that make fraud part of their business model. Bell’s manager went on to inadvertently explain how the ratchet effect prevented RealNetworks from abandoning their shady practices. What’s particularly depressing is that RealNetworks was in many ways an innovator and influencer in teaching the rest of the industry how to exploit people’s need to download your software to earn money through fraud. This fraud-based business model is alive and well today.
Scott Hanselman wrote last week about Download.com’s “download wrapper,” a piece of malware that they attempt to foist on every unsuspecting user who uses the site for its intended purpose. Similarly, there’s the Dark Patterns site, which catalogs the practices Bell and Hanselman wrote about, along with many others. As much as the “app store” model of distributing software depresses me, it remains an infinitely superior alternative to “free” distribution funded through deceptive business practices.
The main thing I’d suggest is that if you work for (or run) a company that engages in these practices, it’s already too late. The ratchet effect all but insures that once a company goes down this road, it is nearly impossible to reverse course. If this sort of thing bothers you (and it should), you might want to seek other work.
I’d also recommend not using software from any company who engages in these practices. Awareness of these practices makes it likely that you can make your way through the minefield when you install the software, but you’re being subsidized by the portion of the user base that is being defrauded. You can also assume that companies that engage in these practices will eventually sell out completely and just install malware on your computer without asking you.
We should be exposing and shaming companies that engage in these practices to the extent that we can stand to. Sites that review software should always take care to mention when the installers attempt to foist unwanted crap upon the user, and mark them down accordingly. This business model isn’t going away, but those of us who are familiar with it should not be enablers.