Paul Querna has written an interesting post arguing that developers should rethink how they handle logging — using a robust, machine-readable format (like JSON) rather than human-readable strings that are formatted so that each log entry is a single line long.
The big change is not in how you create log messages but rather in how you consume them. Right now, when a user notices an error, I tend to immediately log into the server and start looking for the log messages associated with the transaction in question using grep or my favorite pager.
Were I to log everything in a machine-readable format, it would make sense to have a more robust tool to parse the logs. Finding or building such a tool is doable, but it becomes yet another project. You need management to sign off on it, the systems administrators to agree to the infrastructure change, and someone to actually choose, test, and deploy the new tool for dealing with logs. Then you have to teach everyone who’s used to finding things in the old logs how to find things in the new logs.
That’s how things that seem like a great idea find a way to never become reality.
December 27, 2011 at 1:22 am
We do normal line based logging from Apache and Python. We ALSO log machine readable exception messages originally in zilch written specifically for our systems, but have recently moved to using sentry/raven. The combined system works great, but I admit I still mostly just grep in the error logs.
December 27, 2011 at 1:18 pm
If you haven’t seen it before, one of the Disqus developers has been working on https://github.com/dcramer/sentry which has some of these concepts. It integrates well with logging implementations which allow you to provide structured data so human-readable strings can be generated when necessary but your aggregator can e.g. group entries based on the original format string (e.g. “error opening file %s”). There’s an HTTP transport so you can use things other than Python to report logs and still use the Django app for reporting.
December 27, 2011 at 3:07 pm
Machine parsing of log files is one reason tools like Splunk are popular. Out of the box it knows how to consume common log formats, and I can go to the web console and search for events filtering by the client IP address, the response code, the request URL, etc.
December 31, 2011 at 10:33 am
Lennart has RH going full speed ahead /w “Journal,” a structured logging system.