DTrace

Technical Beauty — Episode 30 In 2001, Bryan Cantrill was debugging a production system at Sun Microsystems. The system was misbehaving. The logs said nothing useful. The metrics showed nothing abnormal. By every observable measure, the system was fine. Except it was not fine. It was doing something wrong, and nobody could see what. Cantrill had helped build an entirely synthetic system: every instruction, every data structure, every byte was placed there by human beings. And yet the system could not answer the most basic question: what are you doing right now? That bothered him rather a lot. The Problem Debugging production systems has always offered two options, both terrible. Option one: add logging statements, rebuild, redeploy, wait for the issue to recur. This works in development. In production, it means restarting a database serving ten thousand connections to add a printf. The cure is worse than the disease. And if you guessed wrong about where to log, you restart again. Optio