Mini Project 9: Incident Postmortem Drafting Agent
Tool calls pull an incident timeline, an agent drafts a structured postmortem, and a custom LLM-as-judge rubric scores it before a human editor sees it.
The Postmortem Nobody Wants to Write First
Writing a postmortem well — a clear timeline, blameless language, concrete action items — takes real effort, which is exactly why the first draft after a stressful incident is often thin, defensive in tone, or missing a root-cause section entirely. An agent that pulls the actual timeline from logs and drafts a structured first pass doesn't replace the human judgment a postmortem needs; it removes the blank-page problem and gives an editor something concrete to sharpen instead of starting from nothing.
This project is also this roadmap's clearest non-RAG use of the custom-rubric evaluation pattern from the Agent Evaluation guide — most of that guide's examples score retrieval-and-answer quality; this one scores a structured document against a rubric that has nothing to do with retrieval at all.
Setup
// build.gradle
dependencies {
implementation 'org.springframework.ai:spring-ai-ollama-spring-boot-starter'
implementation 'org.springframework.boot:spring-boot-starter-web'
}This project assumes an internal incident-timeline API and log store already exist — the tool implementations below call out to them; swap the method bodies for your actual observability stack (Datadog, an internal incident-management tool, whatever the team already uses).
1. Tool Calls to Pull the Raw Material
The agent doesn't draft from memory or from a human's summary — it pulls the actual timeline and logs itself, the same "ground the agent in real data via tools" pattern from this roadmap's fourth phase:
@Component
public class IncidentDataTool {
@Tool(description = "Get the chronological timeline of events for an incident by ID")
public List<TimelineEvent> getTimeline(String incidentId) {
return incidentApi.fetchTimeline(incidentId);
}
@Tool(description = "Get relevant error logs for an incident within its time window")
public List<LogEntry> getLogs(String incidentId) {
return logStore.fetchLogsForIncident(incidentId);
}
@Tool(description = "Get the list of engineers who responded to the incident and their roles")
public List<Responder> getResponders(String incidentId) {
return incidentApi.fetchResponders(incidentId);
}
}2. Drafting a Structured Postmortem
Shape the draft as a typed contract with required sections, not free-form prose that might skip the parts that matter most:
public record PostmortemDraft(
String title,
String summary,
List<TimelineEntry> timeline,
String rootCause,
String impact,
List<String> actionItems,
List<String> blamelessLanguageFlags
) {}
public record TimelineEntry(String timestamp, String event) {}public PostmortemDraft draft(String incidentId) {
return chatClient.prompt()
.system("""
Draft an incident postmortem using the getTimeline, getLogs, and getResponders
tools. Required sections: summary, timeline, root cause, impact, action items.
Use blameless language — describe what the system and process allowed to happen,
not what a specific person did wrong. If you catch yourself naming an individual
in connection with a mistake, note it in blamelessLanguageFlags instead of the
main draft, so a human editor can decide how to phrase it.
""")
.user("Draft a postmortem for incident " + incidentId)
.call()
.entity(PostmortemDraft.class);
}Asking the model to self-flag potential blameless-language violations, rather than trusting it to always phrase things correctly, is a cheap extra signal — it doesn't replace an editor reading the draft, but it gives that editor a specific list of sentences worth a second look instead of requiring them to re-read the entire document with equal scrutiny.
3. Scoring the Draft With a Custom Rubric
A generic "is this a good response" evaluator doesn't know what a postmortem specifically needs. Build a rubric that does, following the custom-evaluator pattern from this roadmap's Agent Evaluation guide:
public record PostmortemScore(
int rootCauseClarity, // 1-5: is the root cause specific and mechanistic, not vague?
int actionItemQuality, // 1-5: are action items concrete and assignable, not "be more careful"?
int blamelessTone, // 1-5: does the language focus on systems/process, not individuals?
int timelineCompleteness, // 1-5: does the timeline account for detection through resolution?
List<String> feedback
) {
boolean passesBar() {
return rootCauseClarity >= 3 && actionItemQuality >= 3 && blamelessTone >= 4;
}
}@Component
public class PostmortemJudge {
private final ChatClient judgeClient; // a separate call, same reasoning as any LLM-as-judge setup
public PostmortemScore score(PostmortemDraft draft) {
return judgeClient.prompt()
.system("""
Score this incident postmortem draft on four dimensions, 1-5 each:
rootCauseClarity (specific and mechanistic vs vague), actionItemQuality
(concrete and assignable vs generic), blamelessTone (systems/process
focus vs individual blame), timelineCompleteness (detection through
resolution, no gaps). Provide specific feedback for any score below 4.
""")
.user(draft.toString())
.call()
.entity(PostmortemScore.class);
}
}blamelessTone intentionally has the highest bar (>= 4, not >= 3) in passesBar() — a postmortem that's technically thorough but names individuals for blame is a worse outcome for the team than one that's slightly less detailed but genuinely blameless, and the rubric's pass threshold reflects that priority explicitly rather than leaving it implicit.
Putting It Together
@Service
public class PostmortemDraftingService {
private final ChatClient chatClient;
private final PostmortemJudge judge;
public PostmortemReview generateAndScore(String incidentId) {
PostmortemDraft draft = chatClient.prompt()
.system(DRAFTING_SYSTEM_PROMPT)
.user("Draft a postmortem for incident " + incidentId)
.call()
.entity(PostmortemDraft.class);
PostmortemScore score = judge.score(draft);
return new PostmortemReview(draft, score, score.passesBar()
? "Ready for editor review"
: "Needs revision before editor review: " + String.join("; ", score.feedback()));
}
}Tool calls that ground the draft in the incident's actual timeline and logs rather than a vague human recollection, a structured contract that forces every required section to exist, and a rubric-scored quality gate before a human editor's time is spent — the shape that makes "have the agent draft the first pass" a genuine time-saver instead of one more thing an editor has to fact-check line by line.
What's Next
Mini Project 5 (Fraud Investigation Pipeline) is a useful comparison — both projects care deeply about typed, auditable output, but that project's audit trail comes from a deterministic planner's execution path, while this one's quality gate comes from a separate model's judgment. Worth studying both to see when each approach fits.
Frequently asked questions
Why score the draft with a separate LLM call instead of asking the drafting call to also self-critique in the same response?
The same reasoning as the output-validation pattern in this phase's HR Assistant project — a model reviewing its own just-generated draft in the same call is a weaker check than a fresh call whose only job is scoring, since the drafting call's own biases and blind spots (a vague root cause it didn't realize was vague) are exactly what a genuinely independent second pass is positioned to catch.
Should the rubric's numeric thresholds (3, 4) be tuned over time?
Yes — treat the initial thresholds in this guide as a starting point, not a fixed standard. Once real editors have reviewed enough auto-drafted postmortems, compare which ones they accepted with minor edits versus rewrote substantially, and adjust passesBar() to better predict that outcome.
What happens to a draft that fails the rubric — does the agent automatically retry?
This guide routes a failing draft straight to a human editor with the judge's specific feedback attached, rather than an automatic retry loop — for a document this consequential, a human seeing exactly what the automated judge flagged is more useful than another automated attempt. An automatic single retry (as this phase's Weekly Report project uses) is a reasonable addition if data shows most failures are fixable on a second pass.
Could this same pattern draft other structured documents, like an RCA for a customer-facing status page?
Yes — the tool-calls-for-grounding-data, typed-draft-contract, custom-rubric-judge shape isn't postmortem-specific. A customer-facing RCA would need a different rubric (probably weighting clarity-for-a-non-technical-reader higher, and blameless tone lower since it's not about internal responders) and different data-source tools, but the overall architecture transfers directly.