Cam Davidson-Pilon talks about 21st Century Problems. Here’s how he describes most of the great technological leaps of the 20th century:

What these technologies have in common is that are all deterministic engineering solutions. By that, I mean they have been created by techniques in mathematics, physics and engineering: often being modeled in a mathematical language, guided by physics’ calculus and constrained and brought to life by engineering. I argue that these types of problems, of modeling deterministically, are problems that our fathers had the luxury of solving.

And here’s the truth behind the hype about Big Data we see so much of these days:

Statistical problems describe the space we haven’t explored yet. Statistical problems are not new: they are likely as old as deterministic problems. What is new is our ability to solve them. Spear-headed by the (constantly increasing) tidal wave of data, practitioners are able to solve new problems otherwise thought impossible.

Alistair Croll’s post about Big Data and civil rights is important, but I don’t think he has it exactly right. The article does a good job of explaining how Big Data differs from data warehousing as it was traditionally done. It also illustrates how Big Data can be misused as a vector of discrimination, violating people’s civil rights.

While Big Data may enable people to discriminate in innovative new ways, or to make better guesses about who to discriminate against. However, I don’t think this is necessarily a new civil rights challenge. Discrimination against protected groups is already illegal, at least in the United States. In fact, at least when it comes to employment law and housing, the doctrine of disparate impact also applies. It holds that you can be liable for discrimination even if your practices are not intentionally discriminatory, if those practices have a disproportionately adverse effect on members of a protected group.

It is not the use of Big Data to implement discriminatory practices but rather the discrimination itself that is the fundamental problem. The challenge in fighting discrimination is what it always was, proving the discrimination in the first place. In the era of Big Data, the data and algorithms provide a more concrete paper trail than the unspoken, unrecorded discrimination that still occurs every day.

Here’s an interesting definition of Big Data, from Douglas Patterson:

One useful definition of big data — for those who, like us, don’t think it’s best to tie it to particular technologies — is that big data is data big enough to raise practical rather than merely theoretical concerns about the effectiveness of anonymization.

Distributed database creator and LinkedIn engineer Jay Kreps pokes a hole in the widespread myth that systems that scale horizontally are inherently reliable. This is an important post, because intuitively it seems like a system that scales horizontally and makes provisions for fault tolerance should be reliable. Indeed, that’s the value proposition for many people. Rather than having to be smart enough to provision big servers intelligently and figure out how to make them fault tolerant, you can just throw commodity hardware at the problem and be ready to swap out systems when you encounter the occasional hardware failure.

If you enjoy that article, move on to Daniel Abadi’s post on Replication and the latency-consistency tradeoff. It’s not about system failure but about the performance characteristics of distributed systems. This is the sort of real-world issue that is glossed over when you talk to people about distributed systems a lot of the time.

The hype about the new distributed database systems is that they make life easy. The truth is that they’re incredibly complex, but they make it possible for small companies to achieve things that were out of reach for all but the largest companies until very recently. I’m just starting to wrestle with some of this stuff and you can expect more posts about this topic.

At work, I’ve been experimenting with Apache Solr to see whether it’s the best choice for searching a very large data set that we need to access. The first step was to just set it up and put a little bit of data into it in order make sure that it meets our current and anticipated future requirements. Once I’d figured that out, the next step was to start loading lots of data into Solr to see how well it performs, and to test out import performance as well.

Before I could do that, though, I generated about 33 million records to import, which take up about 10 gigabytes of disk space. That’s not even 5% of the space that the full data set will take up, but it’s a start.

What I’m quickly learning is that when it comes to dealing with Big Data, knowledge of the Unix shell is a huge advantage. To give an example, I’m currently using Solr’s CSV import feature to import the test data. If we wind up using it in production, writing our own DataInputHandler will certainly be the way to go, but I’m just trying to get things done right now.

Here’s the command the documentation suggests you use to load a CSV file into Solr:

curl http://localhost:8983/solr/update/csv --data-binary @books.csv 
    -H 'Content-type:text/plain; charset=utf-8'

I quickly found out that when you tell curl to post a 10 gigabyte file to a URL, it runs out of memory, at least on my laptop.

These are the kinds of problems for which Unix provides ready solutions. I used the split command to split my single 10 gigabyte file into 33 files, each a million lines long. split helpfully named them things like xaa, xab, etcetera, all the way through xbh. You can use command line arguments to tell split to use more meaningful names. Anyway, then I used a for loop to iterate over each of the files, using curl to submit them:

for file in x* ; do
    curl http://localhost:8983/solr/update/csv --data-binary @$file 
        -H 'Content-type:text/plain; charset=utf-8'
done

That would have worked brilliantly, except that Solr wants you to list the fields in the file on the first row of your CSV file, so only the first file imported successfully. I wound up opening all of the others in vim* and copying the headers over rather than writing a script, proving that I need to brush up on my shell skills as well, because prepending a line to a file is easy if not elegant.

Once the files were updated, I used the loop above to import the data.

When it comes to working with big data sets, there are many, many tasks like these. Just being able to use pipes to make sure that your very large data files are always compressed can be a life-saver. Understanding shell scripting is the difference between accomplishing a lot in a day through automation or doing lots of manual work that makes you hate your job.

* I should add that MacVim gets extra credit for opening 33 252 megabyte files at once without complaining. I just typed mvim x* and up popped a MacVim window with 33 buffers. Unix is heavy duty.