Observability, Tracing & Debugging
Why 'the bot gave a confusing answer in 14 seconds' is undebuggable from a final log line, and what tracing every span of an agent request actually captures.
Fourteen Seconds, No Idea Why
The customer-ops system is ready for a wider rollout. During the launch readiness review, someone pulls up a support ticket: a customer's refund question took 14 seconds to answer and came back confusing. The only thing in the logs is the final response — no record of which worker agent handled it, which tools got called, how long any individual step took, or how many tokens the whole interaction burned. Every phase up to this point built a system that works; nothing yet built a way to see what it actually did on any specific request, which makes a slow, wrong answer like this one nearly impossible to diagnose after the fact.
Traces and Spans: Structure, Not Just Logs
Observability for an agent system means capturing structured detail at every meaningful step, not just the final output. The standard vocabulary: one full customer request is a trace; each individual unit of work inside it — the orchestrator's classification call, the refund worker's reasoning turn, each tool call and its latency — is a span within that trace, nested to reflect what called what.
With this structure, the 14-second mystery stops being a mystery: check_eligibility alone took 11.4 of the 14.2 seconds, because its underlying API call was silently retried three times against a degraded dependency — a Phase 8 reliability concern that was firing correctly, just invisibly. Without spans, that entire story is invisible; with them, it's a two-minute read.
What Actually Belongs in a Span
Beyond timing, a useful trace captures the prompt sent, the response received, which tools were called with what arguments (redacted or hashed for sensitive fields as needed), the model and parameters used, and token counts per call — because token usage is what the next guide's cost work actually measures against. OpenTelemetry's semantic conventions for GenAI define a standard, vendor-neutral schema for exactly this — the same interoperability instinct as MCP in the last phase, applied to observability data instead of tool calls, so traces captured this way aren't locked into one specific backend. LangSmith is a purpose-built example of a tracing platform for LLM and agent systems specifically, if a general-purpose observability stack feels like the wrong fit for the LLM-specific fields (prompts, token counts, tool call arguments) that matter most here.
Instrument tracing from the start of a project, not after the first undebuggable production incident. Retrofitting spans onto an already-complex multi-agent system is real, tedious work — adding it to each new agent and tool as they're built is nearly free by comparison.
Check yourself
Why does capturing only the final response (not per-step spans) make it nearly impossible to diagnose why a specific refund request took 14 seconds and returned a confusing answer?
What's Next
Tracing shows what a system did on any given request. It doesn't say whether what it did was actually good — that requires a different kind of measurement, run systematically rather than read one trace at a time.
Frequently asked questions
Does adding tracing to every LLM call and tool call slow the system down?
The overhead is typically small — sending trace data is usually asynchronous and doesn't block the actual request — but it isn't literally free, and very high-volume systems do need to think about sampling (tracing a representative subset of requests) rather than capturing every single one in full detail forever.
Should sensitive data (customer names, payment details) be captured in traces?
Generally no, not in raw form — redact or hash sensitive fields before they're written to a trace, the same way you'd handle sensitive data in any other logging system. A trace that makes debugging easy but leaks customer payment details into an observability backend has just traded one production incident for a worse one.
Is OpenTelemetry's GenAI schema required, or can I define my own trace format?
Nothing requires it — a custom schema works fine for a single team using a single observability backend. The standard schema's value is the same as MCP's: it becomes worth adopting once you might switch observability tools, want to compare traces across teams or systems, or want to use tooling built against the standard rather than something bespoke.