Creating tools that facilitate online A/B experiments is a big part of my job. My team makes sure that we’re collecting data as accurately as possible, and we also created a tool that aggregates the results of experiments and performs statistical analysis of them to insure that our analysis is valid. Needless to say, the controversy over experiments run by Facebook and OkCupid has been interesting to watch from a distance.
For some background on my involvement with Web experiments, you can read this post a member of my team wrote experiments at Etsy back in 2012. I think it holds up pretty well.
Last week Christian Rudder wrote about OkCupid’s experiments on the OkTrends blog, in a provocative post entitled We Experiment On Human Beings! It was written in his inimitable style, with a pugilistic tone. OkCupid ran some pretty radical experiments, and Rudder isn’t apologizing for any of them. He was the interviewed on NPR and refused to apologize for anything OkCupid did.
I am a big believer in iterating on products through experimentation. As I wrote a couple of years ago, quantifying user behavior and analyzing it is what liberates us to some degree from the realm of anecdote and opinion. That said, there’s a reason why there are so many ethical guidelines in academia for experiments on human beings.
Writing at Kottke.org, Tim Carmody has the best argument I’ve read for why OkCupid’s experiments were problematic. I think that everyone who’s responsible for experimenting on the Web ought to read it and think about how it bears on the kinds of experiments they’re running.
Experimentation is a singularly powerful tool for refining ideas and testing the viability of features on the Web, but it’s also easy to abuse, especially in a social context. Fortunately, in the world of e-commerce, experiments are usually about making it easier to check out or testing out changes to search that hopefully make it easier for customers to find items they want to buy.
We’ve seen how a cavalier attitude toward user privacy on the part of Web companies has led to restrictions on cookies that make it more difficult to track user activity. These regulations restrict many kinds of bad behavior, but they also make it more difficult to do legitimate analsysis as well. I worry that a cavalier attitude about the ethics of experimentation will lead to regulations in that area that make it problematic to run any kind of Web experiment.
Many people are already suspicious of any kind of data-driven approach to problem solving. I’m as cynical as anyone about industry self-regulation. While it makes sense not to publicize experiments, we should discuss the kinds of experiments we run, and the role that ethical considerations play in expeirment design.
The ethics of Web experiments
Creating tools that facilitate online A/B experiments is a big part of my job. My team makes sure that we’re collecting data as accurately as possible, and we also created a tool that aggregates the results of experiments and performs statistical analysis of them to insure that our analysis is valid. Needless to say, the controversy over experiments run by Facebook and OkCupid has been interesting to watch from a distance.
For some background on my involvement with Web experiments, you can read this post a member of my team wrote experiments at Etsy back in 2012. I think it holds up pretty well.
Last week Christian Rudder wrote about OkCupid’s experiments on the OkTrends blog, in a provocative post entitled We Experiment On Human Beings! It was written in his inimitable style, with a pugilistic tone. OkCupid ran some pretty radical experiments, and Rudder isn’t apologizing for any of them. He was the interviewed on NPR and refused to apologize for anything OkCupid did.
I am a big believer in iterating on products through experimentation. As I wrote a couple of years ago, quantifying user behavior and analyzing it is what liberates us to some degree from the realm of anecdote and opinion. That said, there’s a reason why there are so many ethical guidelines in academia for experiments on human beings.
Writing at Kottke.org, Tim Carmody has the best argument I’ve read for why OkCupid’s experiments were problematic. I think that everyone who’s responsible for experimenting on the Web ought to read it and think about how it bears on the kinds of experiments they’re running.
Experimentation is a singularly powerful tool for refining ideas and testing the viability of features on the Web, but it’s also easy to abuse, especially in a social context. Fortunately, in the world of e-commerce, experiments are usually about making it easier to check out or testing out changes to search that hopefully make it easier for customers to find items they want to buy.
We’ve seen how a cavalier attitude toward user privacy on the part of Web companies has led to restrictions on cookies that make it more difficult to track user activity. These regulations restrict many kinds of bad behavior, but they also make it more difficult to do legitimate analsysis as well. I worry that a cavalier attitude about the ethics of experimentation will lead to regulations in that area that make it problematic to run any kind of Web experiment.
Many people are already suspicious of any kind of data-driven approach to problem solving. I’m as cynical as anyone about industry self-regulation. While it makes sense not to publicize experiments, we should discuss the kinds of experiments we run, and the role that ethical considerations play in expeirment design.
Commentary
Previous post
Trigger warnings for misogynistsNext post
Policing based on fear can’t work