Debugging C++ — method over guesswork

The difference between a senior engineer and a junior is not "knows gdb commands" but the ability to approach an unfamiliar bug systematically. Half of bugs are solved in a minute when you ask the right questions in the right order. The other half are unsolvable in a week when you ask them in the wrong order.

Topic map

• Systematic approach — reproduce, binary-search, form hypotheses.
• Breakpoints — HW vs SW, watchpoints, conditional breakpoints.
• Debuggers — gdb, lldb, Visual Studio, remote debugging.
• Sanitizers and Valgrind — ASan, UBSan, TSan, MSan — what each catches and when to use it.
• Production debugging — core dumps, structured logging, perf, rr.

The systematic approach

The standard workflow:

1. Reproduce reliably. A bug that happens "sometimes" cannot be debugged — first find the minimal sequence of steps after which it happens every time.
2. Compare expected vs actual. State clearly what should have happened and what actually happened. Surprisingly often the bug is in your expectation, not in the code.
3. Isolate by binary search. Halve the suspect zone. Comment out half the code, or git bisect to the half of commits, or disable half the features.
4. Form a hypothesis and test it. Not "let me try again" — but "if X were true, then Y would happen; let's check Y".
5. Fix the cause, not the symptom. If a function returns null, find out why it returns null — don't wrap the call site in an if.

Breakpoints

A software breakpoint replaces the instruction at the target address with int 3 (on x86) and restores the original when it fires. Free in number, but requires the code to be writable (not the case in JIT or ROM).

A hardware breakpoint uses CPU debug registers (DR0–DR3 on x86) for breakpoints and watchpoints. Fires on either instruction address or data access. A watchpoint on a variable change is the only practical way to find who corrupted it. The limit is four simultaneously.

A conditional breakpoint has an attached condition (break foo if x > 100). Convenient, but slow: every hit is filtered by the debugger.

Debuggers

gdb — the Linux standard. Commands: b (breakpoint), n / s (step over / into), bt (backtrace), p (print), info locals, watch. TUI mode (Ctrl+x a) shows source alongside commands.

lldb — the macOS / Clang standard. Semantically similar, slightly different syntax.

Visual Studio — the best GUI debugger on Windows, with first-class remote and IntelliTrace support.

⚠️ Debug a debug build or RelWithDebInfo. In an optimized release build the debugger jumps to unexpected lines — that is inlining and reordering, not a bug in the debugger.

Sanitizers and Valgrind

Compile with a flag and get runtime checks:

• AddressSanitizer (-fsanitize=address) — out-of-bounds, use-after-free, double-free. Overhead ~2× memory and time.
• UndefinedBehaviorSanitizer (-fsanitize=undefined) — signed overflow, null dereference, misalignment. Low overhead.
• ThreadSanitizer (-fsanitize=thread) — data races. Incompatible with ASan, ~5–15× overhead.
• MemorySanitizer (-fsanitize=memory) — reads of uninitialized memory. Requires every library you link to be built with MSan.

Valgrind (memcheck) — no rebuild required, but 10–50× slower than ASan. Use when you cannot rebuild with sanitizers.

⚠️ Do not run ASan and TSan together — both intercept allocation and conflict.

Production debugging

Production crash that does not reproduce locally? The standard kit:

• Core dumps. Enable via ulimit -c unlimited and /proc/sys/kernel/core_pattern. Analyze: gdb binary core → bt. In Kubernetes, configure securityContext and a volume for dumps.
• Structured logging — JSON logs with a request-id so traces cross service boundaries.
• perf record — collects a sampling profile without rebuilding.
• rr (Record and Replay) — records execution, then debugs deterministically with reverse-step. A game-changer for intermittent bugs.

Common traps

Interview relevance

Debugging is almost always a middle-level question. The interviewer is not checking "do you know gdb commands" — they are checking the thought process: reproduce → isolate → hypothesis → verify. A strong answer names sanitizers (especially ASan and TSan) and knows when to use which.

Typical wrong answer: "I would add printf everywhere and see where it breaks." That is guessing, not debugging. The right starting point is reproduction and bisection.

Popular question directions:

1. How does a hardware breakpoint differ from a software one? How many HW breakpoints can you have?
2. Which sanitizer finds a data race? Use-after-free?
3. How do you debug a crash that only happens in production?
4. What is a core dump and how do you analyze it?
5. What is rr and when does it beat plain gdb?