OK, so a guy made a Web site called Tab Closed Didn’t Read to post screen shots of site that hide their content behind various kinds of overlays that demand that users take some action before proceeding. He’s written a followup blaming the problem on over-reliance on analytics, mainly because some people justify the use of these intrusive calls to action by citing analytics. Anyone who justifies this sort of thing based on analytics should be sued for malpractice.

You can measure almost anything you like. It’s up to the practitioner to determine which metrics really matter to them. In e-commerce the formula relatively simple. If you’re Amazon.com, you want people to buy more stuff. Cart adds are good, but not as good as purchases. Adding items to your wish list is good, but not as good as putting them in your cart. If Amazon.com added an overlay to the home page urging people to sign up for a newsletter, it may add newsletter subscribers, but it’s quite likely that it would lead to less buying of stuff, less adding of stuff to the shopping cart, and less adding of items to wish lists. That’s why you don’t see annoying overlays on Amazon.com.

Perhaps in publishing, companies are less clear on which metrics really matter, so they optimize for the wrong things. Let’s not blame analytics for that.

As I’ve mentioned previously, currently I’m working in the realm of Web analytics. I don’t have a deep statistics background, and I’m definitely not what anyone would mistake for a data scientist, but I do have a good understanding of how analytics can be applied to business problems.

I gained most of that understanding by way of being a baseball fan. I was hanging out with baseball nerds on the Internet talking about baseball analytics long before Moneyball was a twinkle in Michael Lewis’ eye.

Around the time most baseball teams started hiring their own analysts, I assumed that baseball analytics was a solved problem. Given all of the money at stake and all of the eyes on the problem, new analytic insights would be less common. That has turned out not to be the case, for interesting reasons.

The aspect of baseball that makes it the perfect subject for statistical analysis is every game is a series of discrete, recordable events that can be aggregated at any number of levels. At the top, you have the score of the game. Below that, there’s the box score, which shows how each batter and pitcher performed in the game as a whole. From there, you go to the scorecard, which is used to record the result every play in a game, in sequence. Most of the early groundbreaking research into baseball was conducted at this level of granularity.

What happened in baseball is that the instrumentation got a lot better, and the new data created the opportunity for new insights. For example, pitch-by-pitch records from every game became available, enabling a number of interesting new findings.

Now baseball analytics is being fed by superior physical observation of games. To go back in time, one of the greatest breakthroughs in baseball instrumentation was the radar gun, which enabled scouts to measure the velocity of pitches. That enabled analysts to determine how pitch velocity affects the success of a pitcher, and to more accurately value pitching prospects.

More recently, a new system called PITCHf/x has been installed at every major league ball park. It measures the speed and movement of pitches, as well as where, exactly, they cross the strike zone. With it, you can measure how well umpires perform, as well as how good a pitcher’s various pitches really are. You can also measure how well batters can distinguish between balls and strikes and whether they’re swinging at the wrong pitches. This data enabled the New York Times to create the visualization in How Mariano Rivera Dominates Hitters back in 2010.

If you’re working on analytics and you find it’s difficult to glean new insights, it may be time to see if you can add further instrumentation. More granular data will always provide the opportunity for deeper analysis.

Yesterday I saw an interesting link on Daring Fireball to a study that reported the results of searching for 2028 cities and towns in Ontario in the new iOS 6 Maps app for which Apple has apologized. Unsurprisingly, the results of the searches were not very good.

The first question that sprang to my mind when I read the piece, though, was, “How good are the Google Maps results for these searches?” Not because I thought Google’s results would be just as bad, but because looking at statistics in isolation is not particularly helpful when it comes to doing useful analysis. Obviously you can look at the results of the searches and rate the Apple Maps versus reality, but rating them against their competitors is also important. What should our expectations be, really?

Marco Tabini dug into that question, running the same searches under iOS 5.1 (running the Maps app that uses Google’s data). He found that the old Maps app does not outperform the new Maps app by a wide margin, and some interesting differences in how Apple and Google handle location searches.

This isn’t an argument that people shouldn’t be mad about the iOS 6 Maps search capabilities or lack of data, but rather that useful comparisons are critical when it comes to data analysis. That’s why experiments have control groups. Analysis that lacks baseline data is particularly pernicious in cases when people are operating under the assumption that they already know what the baseline is. In these cases, statistics are more likely to actually make people less informed.