Logs, Metrics, and Traces: The Three Pillars of Observability

Logs, Metrics, and Traces

"Observability" gets thrown around as a single concept, but it is really three distinct data types answering three distinct questions, produced by three distinct pipelines, at three distinct costs. Confusing them leads to the two most common observability failures in production systems: logging so much that you can't afford to keep it (or search it), and having no way to connect a slow request in a dashboard to the specific log lines that explain why. This guide covers each pillar individually and — more importantly — how they connect through correlation and trace IDs, because the connections are where the actual debugging value lives.

1. Three Pillars, Three Questions

2. Structured Logging

Unstructured logs (log.info("Order " + orderId + " failed for user " + userId)) are fine to read in a terminal and nearly useless to query at scale. Structured logging emits each log entry as a set of key-value fields — typically JSON — so a log aggregator (Loki, Elasticsearch, CloudWatch Logs Insights, Datadog) can filter, group, and alert on fields instead of doing regex over free text.

Configuring structured JSON output (Spring Boot 3.4+)

Spring Boot 3.4 introduced first-class structured logging support without needing Logback XML surgery:

# application.properties
logging.structured.format.console=ecs
logging.structured.format.file=ecs

{
  "@timestamp": "2026-08-14T09:32:11.482Z",
  "log.level": "ERROR",
  "message": "Order processing failed",
  "service.name": "order-service",
  "process.thread.name": "http-nio-8080-exec-4",
  "log.logger": "com.acme.orders.OrderService",
  "trace.id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span.id": "00f067aa0ba902b7",
  "error.type": "PaymentDeclinedException",
  "order_id": "ORD-88213",
  "user_id": "usr_4471"
}

For older Spring Boot versions or finer control, configure Logback's logstash-logback-encoder directly:

<!-- logback-spring.xml -->
<configuration>
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>spanId</includeMdcKeyName>
            <includeMdcKeyName>orderId</includeMdcKeyName>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="JSON"/>
    </root>
</configuration>

Adding contextual fields with MDC

The Mapped Diagnostic Context (MDC) attaches fields to every log line emitted on the current thread, without having to pass them into every log call manually.

@Component
public class RequestContextFilter extends OncePerRequestFilter {
 
    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
                                     FilterChain chain) throws IOException, ServletException {
        String requestId = Optional.ofNullable(req.getHeader("X-Request-Id"))
                .orElse(UUID.randomUUID().toString());
        MDC.put("requestId", requestId);
        res.setHeader("X-Request-Id", requestId);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear(); // MUST clear — thread pools reuse threads across requests
        }
    }
}

What to log — and what not to

3. Metrics with Micrometer

Micrometer is Spring Boot's metrics facade — a vendor-neutral API (think SLF4J, but for metrics) that can export to Prometheus, Datadog, CloudWatch, or others without changing your instrumentation code. Spring Boot Actuator auto-configures a MeterRegistry and instruments HTTP requests, JVM memory, GC pauses, thread pools, and DataSource pools out of the box.

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

The three metric types that matter

@Service
public class OrderService {
 
    private final Counter ordersPlaced;
    private final Timer orderProcessingTime;
    private final MeterRegistry registry;
 
    public OrderService(MeterRegistry registry) {
        this.registry = registry;
        this.ordersPlaced = Counter.builder("orders.placed")
                .description("Total orders placed")
                .tag("region", "us-east-1")
                .register(registry);
        this.orderProcessingTime = Timer.builder("orders.processing.time")
                .description("Time to fully process an order")
                .publishPercentileHistogram()
                .register(registry);
 
        // Gauge — reflects current state, not a cumulative count
        Gauge.builder("orders.queue.depth", this, OrderService::currentQueueDepth)
                .register(registry);
    }
 
    public Order placeOrder(OrderRequest request) {
        return orderProcessingTime.record(() -> {
            Order order = doPlaceOrder(request);
            ordersPlaced.increment();
            return order;
        });
    }
 
    private double currentQueueDepth() {
        return pendingOrders.size();
    }
}

Tags: the dimension that makes metrics queryable — and dangerous

Counter.builder("payment.attempts")
    .tag("gateway", "stripe")
    .tag("outcome", "declined")
    .register(registry);

Auto-instrumented metrics worth knowing

4. Distributed Tracing with OpenTelemetry

A single user-facing request in a microservice architecture might touch five, ten, or twenty internal services. Metrics tell you the p99 latency went up; logs from any one service only show that service's slice. Distributed tracing stitches the whole request together into a tree of spans — one span per unit of work, each with a start time, duration, and parent-child relationship — so you can see exactly where the time went.

Wiring OpenTelemetry into Spring Boot

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

management.tracing.sampling.probability=0.1
management.otlp.tracing.endpoint=http://otel-collector:4318/v1/traces
management.tracing.propagation.type=w3c

Spring Boot's tracing bridge auto-instruments RestTemplate, WebClient, @Scheduled methods, Kafka listeners, and JDBC calls — spans are created and propagated automatically for anything going through those integration points. For custom business logic, create spans explicitly:

@Service
public class InventoryReservationService {
 
    private final Tracer tracer;
 
    public ReservationResult reserve(String sku, int quantity) {
        Span span = tracer.nextSpan().name("inventory.reserve").start();
        try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
            span.tag("sku", sku);
            span.tag("quantity", String.valueOf(quantity));
            return doReserve(sku, quantity);
        } catch (Exception e) {
            span.error(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

Trace context propagation across services

The key mechanic making distributed tracing work is propagation: the trace ID and parent span ID travel across the wire as HTTP headers (W3C traceparent) or message headers (for Kafka/RabbitMQ), so the downstream service can attach its own spans to the same trace.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^^ version   ^^ trace-id (32 hex)      ^^ parent-span-id  ^^ flags

Sampling: why you can't trace everything

At high request volume, capturing and exporting a span for every single request is both expensive and mostly useless — you don't need to store 50,000 identical "successful GET" traces per minute. Sampling controls what fraction of traces get fully recorded and exported.

5. Correlation: Tying It All Together

The entire value of the three pillars compounds when they're correlated — when a metric dashboard, a trace, and a log line all reference the same trace ID.

Spring Boot's Micrometer Tracing integration does this automatically for logs: once micrometer-tracing-bridge-otel is on the classpath, the default log pattern includes traceId and spanId via MDC without any manual wiring:

logging.pattern.level=%5p [${spring.application.name},%X{traceId:-},%X{spanId:-}]

2026-08-14 09:32:11.482 ERROR [order-service,4bf92f3577b34da6a3ce929d0e0e4736,00f067aa0ba902b7] 12480 --- [http-nio-8080-exec-4] c.a.orders.OrderService : Payment declined

Key takeaways

• Logs, metrics, and traces answer different questions at different costs — don't try to make one pillar do the job of the other two (e.g., don't grep logs for latency percentiles Micrometer already computes for free).
• Structured (JSON) logging is non-negotiable at any real scale — free-text logs cannot be reliably filtered, aggregated, or alerted on.
• Always clear MDC in a finally block — pooled threads carrying stale context between requests is a classic, hard-to-diagnose bug.
• Tag metrics with bounded, low-cardinality dimensions only. Tagging with an ID or email is the single fastest way to overwhelm a metrics backend.
• Trace context propagation is automatic through instrumented HTTP clients and messaging, but breaks silently across raw threads/executors unless you propagate context explicitly.
• Sample traces — you don't need 100% capture, but you almost always want 100% capture of errors and slow outliers specifically.
• Correlate all three pillars via a shared trace ID; surface that trace ID to users/support so incident triage starts from a lookup, not a search.

Interview Questions

• What distinct question does each of logs, metrics, and traces answer, and why can't one substitute for another?
• Why is structured (JSON) logging preferred over free-text logging in production systems?
• What is MDC, and why is it critical to clear it in a finally block?
• What is metric cardinality, and why is tagging a counter with a user ID dangerous?
• Explain the difference between a Counter, a Gauge, and a Timer in Micrometer.
• How does distributed trace context propagate across service boundaries? What header carries it over HTTP?
• What breaks trace propagation in async code, and how do you fix it?
• What is the difference between head-based and tail-based sampling? When would you choose each?
• Why would you force 100% sampling on error spans even with a low overall sampling rate?
• How would you correlate a slow trace with the exact log lines that explain the slowness?
• What auto-instrumented metrics would you check first if hikaricp.connections.pending started climbing?
• Why shouldn't you log full request/response bodies at INFO level by default?
• Describe the parent-child relationship between spans in a trace, and how it's represented in the traceparent header.
• If a dashboard shows a latency spike but you have no way to find the specific slow requests, what pillar is missing from your setup?