Yesterday I saw an interesting link on Daring Fireball to a study that reported the results of searching for 2028 cities and towns in Ontario in the new iOS 6 Maps app for which Apple has apologized. Unsurprisingly, the results of the searches were not very good.

The first question that sprang to my mind when I read the piece, though, was, “How good are the Google Maps results for these searches?” Not because I thought Google’s results would be just as bad, but because looking at statistics in isolation is not particularly helpful when it comes to doing useful analysis. Obviously you can look at the results of the searches and rate the Apple Maps versus reality, but rating them against their competitors is also important. What should our expectations be, really?

Marco Tabini dug into that question, running the same searches under iOS 5.1 (running the Maps app that uses Google’s data). He found that the old Maps app does not outperform the new Maps app by a wide margin, and some interesting differences in how Apple and Google handle location searches.

This isn’t an argument that people shouldn’t be mad about the iOS 6 Maps search capabilities or lack of data, but rather that useful comparisons are critical when it comes to data analysis. That’s why experiments have control groups. Analysis that lacks baseline data is particularly pernicious in cases when people are operating under the assumption that they already know what the baseline is. In these cases, statistics are more likely to actually make people less informed.