debugging-methodology

Systematic debugging follows a loop: record observed facts, form a falsifiable hypothesis, design the cheapest experiment that could disprove it, and update based on results. Keep a written fact/hypothesis/result log so you never re-test a dead theory hours later.

Run git bisect start, mark the broken commit as bad and a known-working commit as good, then test each midpoint. Automate it with git bisect run and a script whose exit code is 0 for good and non-zero for bad, using exit 125 to skip untestable revisions.

Use an interactive debugger for fast local reproductions with tangled state, structured logging for production systems that cannot pause, and distributed tracing for latency or causality across services. For hangs use thread dumps, and for hard crashes use core dumps with post-mortem debugging.

This is a heisenbug: instrumentation perturbs timing or memory layout, hiding races or uninitialized memory. Observe externally with tracing or packet capture, use sanitizers and race detectors, and increase concurrency pressure rather than masking the bug with sleeps.

Treat it as a controlled experiment: enumerate differences in runtime versions, lockfiles, environment variables, data, config, locale, and network between the passing and failing systems. Then bisect the difference by moving one axis at a time until the working machine breaks.

Avoid changing code to see what happens, making several changes at once, reading only the top line of a stack trace, and adding sleeps to hide races. State a hypothesis before editing, change one variable per experiment, and verify fixes by prediction rather than symptom disappearance.