Observability and Monitoring: Logs, Metrics, Traces, SLOs, and Alerts

Start With a Payment That Disappeared

A customer places a $250 order. The payment page shows "processing." Then it shows "something went wrong — check your email." No email arrives. The customer tries again. Same thing. They give up and buy from a competitor.

Three days later, the finance team notices a reconciliation discrepancy: two payments were captured by Stripe but no orders were created in the database. The payment provider authorized the charge. The checkout service crashed after receiving the confirmation. Neither the payment nor the order was logged with enough context to understand why.

There were dashboards. CPU was fine. Memory was fine. No alerts fired.

The system was monitored. It was not observable.

What Observability Means

Observability is the ability to understand a system's internal state from its external outputs — logs, metrics, traces, events, and profiles. A system is observable when you can ask a new question about production behavior and answer it without deploying new code.

Modern observability also includes real-user monitoring (RUM), synthetic checks, business metrics (revenue per minute, signup rate), and continuous profiling.

Why Not Just Use a Monitoring Tool?

Beginners often ask: "We have Datadog / Grafana / Prometheus. Isn't that observability?"

Monitoring tools are the infrastructure of observability. Observability is the practice of using those tools to ask and answer questions.

Step 1: Structured Logging

Plain-text logs are human-readable at small scale and useless at production scale. Structured logs emit key-value fields that machines can parse, filter, and aggregate.

The Anatomy of a Good Log Entry

level: error
service: checkout
traceId: 4bf92f
userId: u_123
orderId: ord_456
message: payment authorization failed
provider: stripe
latencyMs: 382
errorCode: card_declined
environment: production

Every log entry should answer: what happened, in which service, for which user, in which trace, and how long did it take?

Six Rules for Production Logging

1. Include correlation IDs. Every log must carry a trace ID so you can reconstruct the full request path.
2. Log domain identifiers, not just stack traces. "Order 456 failed" is more useful than "NullPointerException at line 203" — log the order ID alongside the stack trace.
3. Redact secrets. Tokens, passwords, PII, and payment data must be stripped before the log leaves the service. A structured log accidentally shipping credit card numbers is a compliance incident.
4. Use consistent field names across services. One service calling it userId and another calling it customer_id makes cross-service queries painful. Agree on a schema.
5. Sample noisy success logs. Log every error. Log every slow request (P99+). Log one in every hundred fast success requests to save storage costs.
6. Keep error logs actionable. An error log should contain enough information to reproduce or diagnose the issue without cross-referencing five other systems.

Step 2: Metrics — RED and USE

Metrics are numeric time-series data used for dashboards, alerts, and trend analysis.

RED Method (For Services)

Every request-driven service should track three things:

USE Method (For Resources)

Every infrastructure resource should track:

Why Averages Lie

A service with average latency of 200ms sounds healthy. But if p50 is 50ms and p99 is 5,000ms, half your users get fast responses while 1% wait five seconds. Always use histograms for latency. Monitor p50, p95, and p99 separately.

Step 3: Distributed Tracing

A single request in a microservice system touches the API gateway, the auth service, the checkout service, the payment provider, the database, and the notification service. Without tracing, a 3-second checkout is a black box.

Trace Concepts

Trace Propagation

The trace ID is the single most important piece of observability data. It connects the log from the API gateway to the log from the database to the span from the payment provider call. Without it, you have isolated data points. With it, you have a story.

Sampling

Tracing every request at scale is expensive. A common approach:

• Head-based sampling: Decide at the entry point whether to trace a request (e.g., trace 1% of all requests, 100% of requests from a specific test account).
• Tail-based sampling: Store a buffer of all spans, then decide at the end which traces to keep (e.g., keep all traces with errors, all slow traces, and a random sample of healthy ones).
• Priority sampling: Always trace errors, always trace critical user paths, sample the rest.

OpenTelemetry is the standard for both instrumentation and export. It provides a single API for generating traces, metrics, and logs, and a single exporter for sending them to any backend.

Step 4: SLIs, SLOs, and Error Budgets

Metrics and traces tell you what is happening. SLOs tell you what should be happening.

The Three Terms

Choosing Good SLIs

A good SLI measures something the user cares about. Not "CPU is below 80%." Not "the process is running." Something the user would notice if it broke.

Error Budget Math

An SLO of 99.9% over 30 days means you can afford 43 minutes and 12 seconds of downtime or degraded performance per month.

total seconds in 30 days = 2,592,000
allowed failure = (100 - 99.9) / 100 * 2,592,000 = 2,592 seconds = 43.2 minutes

Step 5: Alerting

Alerts should page someone when user experience is degrading — not when infrastructure metrics change.

Bad Alerts vs Good Alerts

Multi-Window Burn Rate Alerting

A single alert window is either too fast (false positives) or too slow (misses fast-burning incidents). The multi-window approach uses two windows simultaneously:

• Fast burn window (5 minutes): Catches sudden spikes — a deployment that immediately starts erroring, a database that suddenly becomes unavailable.
• Slow burn window (1 hour): Catches gradual degradation — a slow memory leak, a slowly growing queue, a gradual traffic increase.

Page the on-call when both windows indicate burning. This eliminates the false-positive spike that lasts 30 seconds while catching incidents that take an hour to become critical.

Alert Severity Levels

Step 6: Health Checks

Health probes tell the orchestrator whether a service instance should receive traffic or be restarted.

Health check patterns:

• Shallow: Process is running (checks /health returns 200). Detects crashes, misses dependency issues.
• Deep: Process can reach critical dependencies (checks that the database connection pool has available connections, that Redis responds, that the upstream API is reachable). Used for readiness.
• Synthetic: Runs a real business transaction (creates a test order, deletes it). Used for comprehensive end-to-end checks.

Step 7: Dashboards

Dashboards should answer specific operational questions quickly. A dashboard that tries to show everything shows nothing useful.

Dashboard Types by Audience

Dashboard Anti-Patterns

Common Failure Stories

Story 1: The Payment That Vanished

A payment service crashed after receiving the Stripe confirmation webhook. The crash happened so fast that the error log did not flush to disk. The monitoring dashboard showed the service as healthy (it restarted within 2 seconds). Finance noticed the discrepancy three days later during reconciliation.

Root cause: No structured logging with immediate flush before crash. No trace connecting the Stripe webhook to the order creation.

Fix: Added synchronous logging for payment events (flush before acknowledging the webhook). Added a trace ID to every payment webhook that propagated through the entire payment pipeline. Added an SLI for "payment confirmed minus order created" — if the gap exceeds 60 seconds, an alert fires.

Story 2: The Alert That Everyone Ignored

A team set up 300 alert rules: CPU > 70%, memory > 80%, disk > 60%, one pod restarted, request rate change > 10%, error rate > 0. The on-call engineer received 50+ alerts per night. Within two weeks, every alert was silenced or ignored. When the database actually ran out of connections, nobody noticed for 25 minutes.

Root cause: Alert fatigue from low-signal, non-actionable alerts.

Fix: Reduced to 15 alerts total. Every alert had to pass the test: "If this fires, can the on-call engineer take a specific action?" CPU alerts were replaced by SLO burn-rate alerts. The number of pages per week dropped from 40 to 3. Mean time to acknowledge dropped from 25 minutes to 2 minutes.

Story 3: The Missing Trace

A checkout flow involved six microservices. When a customer complained that checkout "sometimes hangs," the team looked at each service's logs individually — no single log showed an error. The issue was a race condition: service C called service D before service B finished writing to the database. Without a trace ID connecting all six services, the race was invisible.

Root cause: No distributed tracing. Each service had perfect individual logs, but nobody could see the full request path.

Fix: Implemented OpenTelemetry with trace ID propagation through all six services. The trace showed exactly where the 3-second gap occurred: a missing index in the database caused a full table scan under specific conditions. The trace contained the exact query, the exact timing, and the exact condition that triggered it.

Story 4: The Self-Fulfilling Liveness Probe

A team configured Kubernetes liveness probes to check database connectivity. During a database failover, every pod's liveness probe failed simultaneously. Kubernetes restarted all pods. The new pods also could not connect to the database (the failover was still in progress). The restarts happened again. Within 60 seconds, every pod was crash-looping. The service was completely down for 12 minutes — far longer than the database failover itself.

Root cause: Liveness probe depended on an external dependency. Readiness should have checked the database. Liveness should have only checked process health.

Fix: Changed liveness probes to check only that the HTTP server was responding (shallow check). Moved database connectivity to readiness probes. During the next database failover, readiness failed gracefully — pods stopped receiving traffic but did not restart. The service recovered in 45 seconds instead of 12 minutes.

Story 5: The Dashboard Nobody Built

A startup scaled from 1 to 20 microservices over 18 months. They had dashboards for the original monolith (CPU, memory, disk). The new services had no dashboards at all. When the recommendation service started returning empty responses, it took three engineers three hours to trace the issue: the embedding model had been redeployed with a different output dimension, and the vector similarity computation was silently producing garbage.

Root cause: No service-level dashboards for the new microservices. No SLI for "recommendations return valid results."

Fix: Every service got a minimum dashboard: RED metrics, dependency health, and one business SLI. A new rule was added: "No service goes to production without a dashboard and an SLO."

Evaluation: How to Audit Observability

Observability Maturity Model

Audit Checklist

End-to-End Observability Pipeline

Interview Prep

Observability and monitoring is a standard system design interview topic, especially for senior and staff-level roles.

Common Interview Questions

1. Design observability for a microservice checkout system. Walk through structured logging (include trace IDs in every log), RED metrics per service, distributed tracing with OpenTelemetry, SLOs (99.9% of checkouts complete in under 5 seconds), error budgets, burn-rate alerts, and dashboards for each service owner.

2. What is the difference between monitoring and observability? Monitoring is known knowns — you define what to watch. Observability is unknown unknowns — you can debug any issue without shipping new code. Monitoring answers "is the system up?" Observability answers "why is the system behaving this way?"

3. How do you set SLOs for a new service? Start with user-facing SLIs: latency, error rate, availability. Set a conservative SLO (99.9%) and adjust as you learn what users actually tolerate. Use the error budget to decide when to invest in reliability vs features.

4. Your pager is going off every night. How do you fix it? Audit every alert against the question "can the on-call take a specific action from this alert?" Replace infrastructure-based alerts (CPU, memory) with SLO burn-rate alerts. Group related alerts. Set appropriate severity levels.

5. How do you trace a request across 20 services? Implement OpenTelemetry with trace ID propagation through HTTP headers (or message queue headers for async services). Each service passes the trace ID to every downstream call. A centralized trace store (Jaeger, Tempo) collects all spans and reconstructs the full trace.

6. What metrics would you track for a database? USE: utilization (CPU, connections, disk IO), saturation (query queue depth, replication lag), errors (deadlocks, replication errors, disk failures). Plus query-specific: slow query count, cache hit ratio, index usage.

Key Takeaways

1. The three pillars complement each other: logs provide event detail, metrics show trends, traces connect the request flow across services. You need all three.
2. SLOs should measure user experience, not infrastructure. An SLO of "CPU < 80%" does not tell you if users are happy. An SLO of "checkout completes in under 5 seconds" does.
3. Error budgets replace emotion with data. When the budget is full, ship features. When it is empty, fix reliability. No arguments needed.
4. Alert on symptoms, page on burns. Infrastructure alerts create noise. SLO burn-rate alerts create action.
5. Health probes have distinct jobs. Liveness checks the process. Readiness checks dependencies. Never mix them.
6. Dashboards serve different audiences. An executive dashboard shows SLO attainment. An incident dashboard shows current errors and deploys. One dashboard cannot do both.