Monitoring and Alerting: Prometheus, Grafana, Golden Signals

Monitoring and Alerting Basics

Collecting metrics is the easy part — Micrometer and Actuator do most of that work for you. The hard part, and the part that actually determines whether your on-call rotation is sustainable, is deciding what to look at, what to alert on, and how to word an alert so the person paged at 3am knows what to do in the first ten seconds of reading it. This guide covers the Prometheus/Grafana stack end to end and, more importantly, the discipline of designing dashboards and alerts that surface real problems without training your team to ignore pages.

1. How Prometheus Scraping Works

Prometheus operates on a pull model: it periodically scrapes an HTTP endpoint on each target and stores whatever numeric samples it finds, each tagged with labels, as a time series. This is the opposite of most logging pipelines, which push data outward — the target doesn't send anything anywhere, it just exposes current values whenever asked.

Making a Spring Boot service scrapable

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

management.endpoints.web.exposure.include=health,prometheus
management.metrics.tags.application=${spring.application.name}

# prometheus.yml
scrape_configs:
  - job_name: 'spring-boot-services'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app

Scrape interval trade-offs

2. PromQL Basics

PromQL is Prometheus's query language for slicing and aggregating time series. A working knowledge of a handful of functions covers the vast majority of production dashboards and alerts.

# Raw counter value (rarely useful directly — counters only go up)
http_server_requests_seconds_count{uri="/orders", method="POST"}
 
# Per-second rate over a 5-minute window — the actual useful signal
rate(http_server_requests_seconds_count{uri="/orders"}[5m])
 
# Error rate as a percentage of total requests
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))
* 100
 
# p99 latency from a histogram
histogram_quantile(0.99,
  sum(rate(http_server_requests_seconds_bucket{uri="/orders"}[5m])) by (le)
)
 
# Connection pool saturation
hikaricp_connections_pending{pool="HikariPool-1"} > 0

3. Grafana Dashboards

Grafana visualizes Prometheus (and other) data sources. The failure mode to avoid is the "wall of graphs" dashboard — 40 panels nobody reads during an incident because nobody can tell which ones matter in the first five seconds.

Dashboard design principles

{
  "title": "Order Service — Overview",
  "templating": {
    "list": [
      { "name": "environment", "type": "query", "query": "label_values(environment)" }
    ]
  },
  "panels": [
    { "title": "Request Rate", "targets": [{ "expr": "sum(rate(http_server_requests_seconds_count{application=\"order-service\", environment=\"$environment\"}[5m]))" }] },
    { "title": "Error Rate %", "targets": [{ "expr": "sum(rate(http_server_requests_seconds_count{application=\"order-service\", status=~\"5..\"}[5m])) / sum(rate(http_server_requests_seconds_count{application=\"order-service\"}[5m])) * 100" }] },
    { "title": "p99 Latency", "targets": [{ "expr": "histogram_quantile(0.99, sum(rate(http_server_requests_seconds_bucket{application=\"order-service\"}[5m])) by (le))" }] },
    { "title": "CPU / Memory Saturation", "targets": [{ "expr": "process_cpu_usage{application=\"order-service\"}" }] }
  ]
}

4. The Four Golden Signals

Google's SRE book distills what to monitor for any service down to four signals. If you only have bandwidth to build four dashboard panels and four alert rules, build these.

5. Designing Alerts That Don't Cause Fatigue

The purpose of an alert is to get a human to take an action they wouldn't have taken otherwise. Every alert that pages someone with nothing actionable to do erodes trust in the entire alerting system — eventually people start silencing channels instead of reading pages, which is how real incidents get missed.

Symptom-based vs cause-based alerting

A well-formed alert rule

# alerting-rules.yml
groups:
  - name: order-service-slo
    rules:
      - alert: OrderServiceHighErrorRate
        expr: |
          sum(rate(http_server_requests_seconds_count{application="order-service", status=~"5.."}[5m]))
          /
          sum(rate(http_server_requests_seconds_count{application="order-service"}[5m]))
          > 0.02
        for: 5m
        labels:
          severity: page
          team: orders
        annotations:
          summary: "order-service error rate above 2% for 5 minutes"
          description: "Error rate is {{ $value | humanizePercentage }}. Check the order-service dashboard: https://grafana.internal/d/order-service"
          runbook_url: "https://runbooks.internal/order-service-high-error-rate"

The alert severity ladder

Routing and deduplication with Alertmanager

Prometheus fires alerts; Alertmanager decides what to do with them — grouping related firings into a single notification, deduplicating repeats, silencing known issues, and routing by team/severity. Without it, a single outage that trips ten alert rules across twenty pods pages your on-call twenty times in the same minute instead of once.

# alertmanager.yml
route:
  receiver: default-slack
  group_by: ["alertname", "application"]
  group_wait: 30s        # wait briefly to batch related alerts into one notification
  group_interval: 5m     # minimum time between updates to an existing group
  repeat_interval: 4h    # don't re-page for an already-acknowledged, still-firing alert
  routes:
    - matchers: ["severity=page"]
      receiver: pagerduty-oncall
      continue: true
    - matchers: ["severity=ticket"]
      receiver: jira-ticket
    - matchers: ["team=orders"]
      receiver: orders-slack
 
receivers:
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_ROUTING_KEY}"
  - name: orders-slack
    slack_configs:
      - channel: "#orders-alerts"

Multi-window, multi-burn-rate alerting (SLO-based)

Simple threshold alerts either fire too late (average over a long window smooths out real spikes) or too often (a short window catches noise). A common refinement pairs a fast, sensitive window with a slow, high-confidence window and requires both to agree:

- alert: OrderServiceErrorBudgetBurnFast
  expr: |
    (
      sum(rate(http_server_requests_seconds_count{application="order-service", status=~"5.."}[5m]))
      / sum(rate(http_server_requests_seconds_count{application="order-service"}[5m]))
    ) > (14.4 * 0.001)
    and
    (
      sum(rate(http_server_requests_seconds_count{application="order-service", status=~"5.."}[1h]))
      / sum(rate(http_server_requests_seconds_count{application="order-service"}[1h]))
    ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: page

Key takeaways

• Prometheus pulls metrics on a schedule from a known endpoint (/actuator/prometheus); Kubernetes deployments almost always use service discovery rather than static scrape targets.
• Never graph or alert on a raw counter — always wrap it in rate() or increase(), or the number is meaningless.
• Build the four golden signals — latency, traffic, errors, saturation — as the top row of every service dashboard before anything else.
• Measure latency separately for success and error responses; blended latency can look artificially good during an outage of fast failures.
• Page on symptoms (user-facing latency/error SLO breaches), not causes (CPU%, memory%) — causes belong on dashboards for root-cause work, not in pages.
• Every alert needs a for: window to avoid flapping, a runbook link, and a live value in the description — a page with no next action is a page that trains people to ignore pages.
• Regularly audit alerts for ones that fire without action or resolve themselves; demote or delete them, because alert fatigue is what causes real incidents to get missed.

Interview Questions

• How does Prometheus's pull-based scraping model differ from a push-based metrics pipeline, and what are the trade-offs?
• Why should you never alert on a raw Prometheus counter value directly?
• What are the four golden signals, and what does each one actually measure?
• Why is it important to measure latency separately for successful and failed requests?
• What's the difference between a symptom-based alert and a cause-based alert? Give an example of each.
• Why might paging on "CPU > 80%" alone be a bad alerting strategy?
• What does the for: clause in a Prometheus alert rule do, and why does it matter?
• Design an alert rule for an error-rate SLO breach. What fields would you include beyond the raw expression?
• What is a multi-window, multi-burn-rate alert, and what problem does it solve compared to a single threshold?
• How would you decide whether an alert should page someone immediately versus just create a ticket?
• What's wrong with a dashboard that has 40 panels and no clear "check this first" section?
• Explain histogram_quantile() — what data does it need, and why can't you compute a percentile from a plain counter?
• How does Kubernetes service discovery in Prometheus avoid the need to hand-maintain a list of scrape targets?
• What would you look for in a monthly alert audit, and what actions might come out of it?