Skip to main content
Back to module
Software Engineering Practices · Quality

Debugging production issues

Reading47mLesson 3 of 3 in this module

Turn a stack trace and logs into a hypothesis, a failing test and a fix.

Why it matters

On your own machine you can pause code in a debugger and inspect every variable. In production you usually cannot. A user reports "checkout is broken", and what you have is an error message, a stack trace and some logs. Debugging production issues is the skill of turning those clues into a reproducible bug, a fix and a test that keeps it fixed. It is a large part of what working engineers do, and it rewards a calm, methodical approach over clever guessing.

Reading a stack trace

When a JavaScript error is thrown and not caught, Node prints the error and the chain of function calls that led to it:

TypeError: Cannot read properties of undefined (reading 'price')
    at lineTotal (/app/src/cart.js:14:20)
    at Array.reduce (<anonymous>)
    at cartTotal (/app/src/cart.js:20:22)
    at handleCheckout (/app/src/routes/checkout.js:31:17)
    at Layer.handle (/app/node_modules/express/lib/router/layer.js:95:5)

Read it like this:

  1. The first line is the error type and message. TypeError: Cannot read properties of undefined (reading 'price') means some code did x.price while x was undefined.
  2. The next line is where it was thrown: function lineTotal, file src/cart.js, line 14, column 20.
  3. Each line below is the caller of the one above. cartTotal called lineTotal (through reduce), and the route handler called cartTotal.
  4. Skip frames from libraries (node_modules, <anonymous>, framework internals) and find the first and last frames in your code. The bug is usually in your code, near those frames.

So the question becomes: why did cartTotal pass undefined to lineTotal? Open cart.js at line 20 and look at what it iterates over.

Reading logs

Logs tell you what happened before the error. Good production logs are structured (one JSON object per line) and carry a request id, so you can filter every line from the failing request:

{"level":"info","time":"2026-10-23T09:14:02Z","requestId":"r-7f3a","msg":"checkout started","userId":118,"items":3}
{"level":"warn","time":"2026-10-23T09:14:02Z","requestId":"r-7f3a","msg":"product not found","productId":"sku-404"}
{"level":"error","time":"2026-10-23T09:14:02Z","requestId":"r-7f3a","msg":"checkout failed","err":"Cannot read properties of undefined (reading 'price')"}

The warn line is the clue: a product in the cart no longer exists, the lookup returned undefined, and lineTotal crashed on it. Useful questions while reading logs:

  • When did it start? Compare with the time of the last deploy or configuration change.
  • Who is affected? Every request, one user, one region, one browser?
  • What is different about the failing requests compared with the successful ones?

A method that works

  1. Gather facts. Error message, stack trace, logs for one failing request, time it started, recent changes.
  2. Form a hypothesis that explains all the facts: "carts containing a deleted product crash checkout."
  3. Reproduce it locally, ideally as a failing automated test: a cart with an item whose product is missing.
  4. Fix the cause, not just the symptom. Here the right behaviour might be to drop the missing item and show a message, rather than wrapping everything in try/catch and hiding the error.
  5. Run the test (now green) and the rest of the suite.
  6. Check in production after the deploy: the error rate in the logs or dashboard should fall.

If you cannot form a hypothesis, narrow the search. Bisect: if it worked in last week's release and fails in today's, git bisect can find the commit that introduced the bug in a handful of steps by testing halfway points. Add logging around the suspect area and deploy it, then wait for the next occurrence.

Mitigate first, fix second

When users are affected, the first priority is to stop the damage. Rolling back the last deploy or turning off a feature flag is often faster and safer than writing a fix under pressure. Then debug calmly. The incidents lesson in this course covers this in more depth.

Common mistakes

  • Reading only the error message and skipping the stack trace and logs.
  • Changing several things at once, so you cannot tell which change fixed it.
  • "Fixing" by catching and ignoring the error, so the bug hides and corrupts data later.
  • Not writing a test for the bug, so it comes back.
  • Logging secrets or personal data while debugging. Logs are often kept for months and seen by many people.

Recap

  • A stack trace reads top-down from where the error was thrown to the outermost caller; focus on frames in your own code.
  • Structured logs with a request id show what led up to the error.
  • Facts, hypothesis, reproduce as a test, fix the cause, verify in production.
  • Under pressure, mitigate first (roll back, disable the feature), then fix.

Try it

In the Do tab, reproduce a real bug from a stack trace on your own machine, then fix it under a test.

Also in this lesson: