Using “git bisect” to find the GOOD patch
I’ve been using git as my Subversion client at work lately. I’ve promised myself that I will blog more about this, but I still need to work out some kinks in my work flow and bootstrapping the process.
In any case, git bisect has turned out to be an invaluable tool for isolating the cause of reproducible failures when you are starting with a known good build and a known bad build. I’ve isolated three failures in the last couple weeks. All were subtle race condition or other threading issues which appear apparently at random – the worst kind!
Prior to this, we were doing what my coworker Jeff Messner calls “debugging by epiphany” – stare at the logs, then the code, then the logs, then the code, then the logs… until in a Zen koan moment you are enlightened.
Perhaps this is why the Linux kernel is so reliable. Linus advocating the identification of failures by bisection since the very first kernel documentation I ever read. Curiously, Linus himself authored the “git bisect” command.
In any case, I encountered a scenario yesterday where a recent build of a component resolved a customer’s issue; however, it wasn’t obvious which change was the actual resolution. I’m documenting this information these days in machine-readable format so that our program which builds customer installs will flag bad combinations of components before we ship to QA.
I tried to reverse the “good” and “bad” patches with bisect, hoping it would be smart enough to deduce that I was looking for the patch which introduced the fixing change, rather than the usual breaking change. It was no go – git tells me there is a problem with the patch ancestry; it absolutely requires the good change to appear before the bad.
It turns out that it wasn’t too difficult to just tell git that the good builds were bad and the bad builds were good in order to isolate the change which resolved the customer issue.