Mini Project 4: GitHub PR Triage Agent over MCP
An MCP client that reads and comments on pull requests, flags risky diffs, and suggests reviewers — with merge, close, and push kept behind human approval.
The Reviewer Queue Nobody Reads Top-to-Bottom
A team with forty open pull requests doesn't have a review problem — it has a triage problem. The urgent, risky diff touching the payments module sits at position 23, sandwiched between two dependency-bump PRs a bot already approved. A human skimming the list in reverse-chronological order finds it eventually, or doesn't, until it's the thing that broke prod.
This project builds an agent whose entire job is that first skim: summarize what's open, flag what actually deserves a careful human look, suggest who should look at it. Unlike every other project in this phase, the end user here is the engineering team itself, not a customer — and that changes the risk calculus in one specific way worth building in from day one.
Setup
// build.gradle
dependencies {
implementation 'org.springframework.ai:spring-ai-ollama-spring-boot-starter'
implementation 'org.springframework.ai:spring-ai-mcp-client-spring-boot-starter'
implementation 'org.springframework.boot:spring-boot-starter-web'
}# application.yml
spring:
ai:
mcp:
client:
stdio:
connections:
github:
command: npx
args: ["-y", "@modelcontextprotocol/server-github"]
env:
GITHUB_PERSONAL_ACCESS_TOKEN: ${GITHUB_TOKEN}This connects as an MCP client to a GitHub-tools MCP server exactly as covered in this roadmap's fourth phase — if MCP client configuration is unfamiliar, that's the prerequisite, not something this project reintroduces.
Scope GITHUB_TOKEN to exactly what this agent needs: read access to pull requests, diffs, and comments, plus comment-write access. Do not grant it merge, push, or branch-deletion scopes at the token level — section 2 below adds an application-level approval gate, but the token-level scope is the real backstop if that gate ever has a bug.
1. Discovering and Filtering the MCP Toolset
Spring AI turns every tool an MCP server exposes into a ToolCallback the model can invoke, discovered automatically at startup:
@Service
public class PrTriageService {
private final ChatClient chatClient;
public PrTriageService(ChatClient.Builder builder, List<McpSyncClient> mcpClients) {
ToolCallbackProvider tools = new SyncMcpToolCallbackProvider(mcpClients);
this.chatClient = builder
.defaultSystem("""
You triage open GitHub pull requests for an engineering team.
Summarize each PR in one sentence. Flag as HIGH_RISK any PR that:
is over 400 lines changed, touches files under /auth or /payments,
or has no test files changed. Suggest a reviewer based on CODEOWNERS
if available, otherwise based on who last touched the changed files.
""")
.defaultToolCallbacks(tools.getToolCallbacks())
.build();
}
}The risk heuristics live in the system prompt deliberately, not hardcoded Java — "over 400 lines" or "touches /payments" is a policy a team should be able to tune without a redeploy, the same reasoning this roadmap's guardrails guide gives for keeping policy text out of compiled logic where it doesn't need to be there.
2. The Authorization Boundary: Read and Comment, Never Merge
The MCP GitHub server exposes tools for merging, closing, and pushing to a PR alongside the read-only ones — and an LLM deciding to call one of those because it seemed helpful is exactly the tool-authorization failure mode this roadmap's AI security guide covers in the abstract. Here it's a concrete, specific list:
private static final Set<String> ALLOWED_TOOLS = Set.of(
"list_pull_requests",
"get_pull_request",
"get_pull_request_diff",
"list_pull_request_files",
"add_pull_request_review_comment"
);
public List<ToolCallback> filterToAllowedTools(List<ToolCallback> allTools) {
return allTools.stream()
.filter(tool -> ALLOWED_TOOLS.contains(tool.getToolDefinition().name()))
.toList();
}this.chatClient = builder
.defaultSystem(SYSTEM_PROMPT)
.defaultToolCallbacks(filterToAllowedTools(tools.getToolCallbacks()))
.build();This is a denylist inverted into an allowlist, and that distinction matters: filtering out merge_pull_request by name is fragile the moment the MCP server adds a new mutating tool with a name you didn't anticipate. Filtering in only the specific read/comment tools this agent needs means a new, unexpected tool from the server is excluded by default, not included by default. The system prompt asking the model to "never merge" is a second layer, not the actual boundary — the actual boundary is that merge_pull_request is never in the tool list the model can see at all.
3. Structured Triage Output
Raw prose summaries don't slot into a dashboard or a Slack digest. Shape the output the same way every structured-output call in this roadmap does:
public record PrTriageResult(
int prNumber,
String title,
String summary,
RiskLevel risk,
String riskReason,
String suggestedReviewer
) {}
public enum RiskLevel { LOW, MEDIUM, HIGH_RISK }public List<PrTriageResult> triageOpenPrs(String repo) {
return chatClient.prompt()
.user("Triage all open pull requests in " + repo)
.call()
.entity(new ParameterizedTypeReference<List<PrTriageResult>>() {});
}A scheduled job runs this every morning and posts the HIGH_RISK subset to a Slack channel — the summary a human actually reads is now three lines instead of forty PR titles in chronological order.
Putting It Together
@Service
public class PrTriageService {
private final ChatClient chatClient;
public PrTriageService(ChatClient.Builder builder, List<McpSyncClient> mcpClients) {
ToolCallbackProvider provider = new SyncMcpToolCallbackProvider(mcpClients);
List<ToolCallback> scopedTools = filterToAllowedTools(provider.getToolCallbacks());
this.chatClient = builder
.defaultSystem("""
You triage open GitHub pull requests. Summarize each PR in one sentence.
Flag as HIGH_RISK any PR over 400 lines changed, touching /auth or
/payments, or with no test files changed. Suggest a reviewer from
CODEOWNERS or recent file history.
""")
.defaultToolCallbacks(scopedTools)
.build();
}
public List<PrTriageResult> triageOpenPrs(String repo) {
return chatClient.prompt()
.user("Triage all open pull requests in " + repo)
.call()
.entity(new ParameterizedTypeReference<List<PrTriageResult>>() {});
}
}Discovery, an allowlisted tool set that makes the risky action structurally unreachable rather than merely discouraged, and a structured result a dashboard can render directly — the full loop from "forty open PRs" to "three that actually need a careful look this morning."
What's Next
This project is a natural predecessor to Mini Project 11 (Agentic Software Delivery Pipeline) — once that pipeline's Review Agent opens a real pull request, this triage agent is one of the things that would pick it up downstream. Worth building this one first if you're planning to build both.
Frequently asked questions
Why an allowlist of tool names instead of just not installing the MCP server's write capabilities?
The official GitHub MCP server exposes read and write tools from the same connection — there's no server-side flag to install a read-only subset. Filtering the ToolCallback list on the client side, as this guide does, is the mechanism actually available; some community MCP servers do offer a read-only mode as a connection option, which is worth preferring if your specific server supports it.
What stops the model from just describing what it would do instead of calling a disallowed tool, and a human acting on that description?
Nothing in this specific project, and it's a real residual risk worth naming rather than hand-waving — an agent that says "I would merge PR #42" in a Slack digest relies on a human reading critically rather than reflexively trusting the bot. Keeping this agent's output to read-only triage information (summaries, risk flags, reviewer suggestions) rather than ever phrasing anything as an actionable command reduces how often that residual risk actually matters.
Should the risk heuristics (400 lines, /payments path) live in the system prompt or in actual Java code that pre-filters before the model sees anything?
Both have a place. The system-prompt version here is fast to iterate on and good enough for a first pass; once a specific heuristic proves valuable and stable (e.g. "always flag /payments changes" turns out to never have false positives), promoting it to a deterministic pre-filter in Java is more reliable and doesn't depend on the model consistently applying a prompt instruction.
Does this pattern work for GitLab or Bitbucket instead of GitHub?
Yes — swap the MCP server connection for a GitLab- or Bitbucket-flavored equivalent (community MCP servers exist for both as of this writing) and everything from section 2 onward is unchanged, since the allowlist-by-tool-name pattern and the structured-output shape aren't GitHub-specific.