debugging-and-error-recovery

Reproduce the failure reliably first, then localize which layer is failing, reduce to a minimal case, and fix the root cause rather than the symptom. Finish by adding a regression test that fails without the fix and verifying the full suite passes.

Use git bisect: mark the current commit as bad and a known-working commit as good, then let git checkout midpoint commits while you run the failing test at each step. You can automate it with git bisect run followed by your test command.

Classify it as timing-dependent, environment-dependent, state-dependent, or truly random, then apply the matching strategy: add timestamps and artificial delays, compare environments, check for leaked shared state, or add defensive logging and alerting until it recurs.

Symptom fixes mask the underlying defect, so the bug resurfaces elsewhere and compounds with later changes. Asking why the failure happens until you reach the actual cause produces a durable fix that a regression test can guard.

Add instrumentation when you cannot localize a failure, the issue is intermittent, or multiple components interact. Remove it once the bug is fixed and guarded by tests, and always remove logs containing sensitive data.