Mini Project 1: Internal Engineering Knowledge Base Copilot
A RAG bot that cites its sources and knows when it's citing something dangerously out of date — because the naive version of this project already caused one incident.
The Runbook That Caused an Incident
Acme Fintech's platform team has the problem every engineering org eventually has: hundreds of Markdown docs — runbooks, architecture decision records, onboarding guides — scattered across a wiki export, impossible to search well, and growing stale faster than anyone can review. Someone builds the obvious fix over a weekend: chunk everything, embed it, wire up QuestionAnswerAdvisor, ship a Slack bot.
Two weeks later, a new engineer asks the bot how to restart a stuck payment job. The bot confidently cites a runbook, quotes the exact commands, and the engineer runs them. The runbook describes a payment gateway that was decommissioned eight months earlier — the commands touch nothing, the actual stuck job stays stuck for another hour, and the retro afterward has one finding: the bot never knew the runbook was stale, because nothing about a naive RAG pipeline tracks that.
This project is that bot, rebuilt with the two things the weekend version skipped: every answer names its sources, and every source's age is a first-class part of the retrieval decision, not an afterthought. This is a full application of this roadmap's third phase (RAG, chunking, metadata) and its advanced-RAG guide's DocumentPostProcessor — if either is unfamiliar, that phase is the prerequisite for this project, not a rebuild of it.
Setup
This project builds on the same Ollama-first, Spring Boot 4 project every guide in this roadmap started from (Phase 1). Two dependencies are new here — everything else (the chat starter, spring-boot-starter-web) is already in place if you've been following along:
// build.gradle
dependencies {
// Already present from Phase 1 / Phase 3 if you're continuing the same project:
implementation 'org.springframework.ai:spring-ai-ollama-spring-boot-starter'
implementation 'org.springframework.boot:spring-boot-starter-web'
// New for this project:
implementation 'org.springframework.ai:spring-ai-rag' // RetrievalAugmentationAdvisor, VectorStoreDocumentRetriever, ContextualQueryAugmenter, DocumentPostProcessor
implementation 'org.springframework.ai:spring-ai-pgvector-store-spring-boot-starter' // VectorStore
}# application.yml
spring:
ai:
vectorstore:
pgvector:
initialize-schema: true
datasource:
url: jdbc:postgresql://localhost:5432/acme
username: acme
password: ${DB_PASSWORD}No extra dependency is needed for the YAML frontmatter parsing in the next section — org.yaml.snakeyaml.Yaml ships transitively with every Spring Boot application already (Boot uses it for its own YAML property support), so implementation doesn't need a new line for it. The @Scheduled staleness-audit job in section 4 needs one line of configuration instead of a dependency: add @EnableScheduling to your @SpringBootApplication class.
1. Ingesting Docs With Real Metadata
Real engineering docs aren't bare text — they carry authorship and freshness information, usually as YAML frontmatter:
---
title: Restarting a Stuck Settlement Job
owner: payments-platform
last_updated: 2026-01-14
doc_type: runbook
---
## Restarting a Stuck Settlement Job
...Parse the frontmatter and carry it into Document metadata alongside the chunking pipeline from this roadmap's third phase:
public List<Document> loadMarkdownWithFrontmatter(Path docsDirectory) throws IOException {
List<Document> documents = new ArrayList<>();
for (Path file : Files.walk(docsDirectory).filter(p -> p.toString().endsWith(".md")).toList()) {
String raw = Files.readString(file);
Map<String, String> frontmatter = parseFrontmatter(raw); // simple YAML-block parser
String body = stripFrontmatter(raw);
TextTextSplitter splitter = TokenTextSplitter.builder().withChunkSize(800).build();
for (Document chunk : splitter.apply(List.of(new Document(body)))) {
chunk.getMetadata().putAll(Map.of(
"title", frontmatter.getOrDefault("title", file.getFileName().toString()),
"owner", frontmatter.getOrDefault("owner", "unknown"),
"doc_type", frontmatter.getOrDefault("doc_type", "general"),
"last_updated", frontmatter.getOrDefault("last_updated", "1970-01-01"),
"source_path", file.toString()
));
documents.add(chunk);
}
}
return documents;
}last_updated coming from frontmatter is only as trustworthy as whoever last edited the doc remembering to bump it — which, realistically, is not reliably. Prefer deriving it from the file's actual last-modified timestamp in version control (git log -1 --format=%cI -- path/to/doc.md) at ingestion time over trusting a hand-maintained field, and fall back to frontmatter only when the doc isn't in version control at all.
2. Structured Output With Citations
The single biggest gap in the naive version: the answer was prose with no verifiable link back to a specific document. Make citations part of the response contract, not an afterthought:
public record Citation(String title, String sourcePath, String lastUpdated, String owner) {}
public record CopilotAnswer(
String answer,
List<Citation> citations,
boolean possiblyStale
) {}CopilotAnswer answer = chatClient.prompt()
.advisors(policyRagAdvisor) // RetrievalAugmentationAdvisor, built in section 3
.user("How do I restart a stuck settlement job?")
.call()
.entity(CopilotAnswer.class);.entity(CopilotAnswer.class) on a RAG-backed call works exactly like every other structured-output call in this roadmap — the model is asked to name which retrieved chunks actually informed the answer, as Citation entries, not just to produce free text that happens to mention a doc title. A UI rendering CopilotAnswer can now show "Sourced from: Restarting a Stuck Settlement Job (updated Jan 14, 2026)" directly next to the answer, letting the engineer judge trustworthiness themselves instead of taking the bot's word for it.
3. Staleness: Filter Hard, Boost Soft, Flag Honest
Three complementary techniques, not one silver bullet:
Hard filter: exclude anything past a real deprecation cutoff
DocumentRetriever retriever = VectorStoreDocumentRetriever.builder()
.vectorStore(vectorStore)
.similarityThreshold(0.60)
.filterExpression(new FilterExpressionBuilder()
.gte("last_updated", LocalDate.now().minusMonths(24).toString())
.build())
.build();Docs older than the cutoff are never retrieved at all — the right call for content that's actively wrong once stale (that decommissioned gateway's runbook), not merely dated.
Soft boost: prefer newer docs among otherwise-relevant matches
public class RecencyBoostPostProcessor implements DocumentPostProcessor {
@Override
public List<Document> process(Query query, List<Document> documents) {
return documents.stream()
.sorted(Comparator.comparing(
d -> (String) d.getMetadata().getOrDefault("last_updated", "1970-01-01"),
Comparator.reverseOrder()))
.toList();
}
}This is the same DocumentPostProcessor this roadmap's advanced-RAG guide introduced — reused here for exactly the use case it was built for. Reach for this where being somewhat dated doesn't make content wrong, just less current than something better (two onboarding guides, both accurate, one more recently reviewed).
Honest flag: tell the engineer when the best match still isn't fresh
Advisor knowledgeBaseAdvisor = RetrievalAugmentationAdvisor.builder()
.documentRetriever(retriever)
.documentPostProcessor(new RecencyBoostPostProcessor())
.queryAugmenter(ContextualQueryAugmenter.builder().allowEmptyContext(false).build())
.build();.defaultSystem("""
Answer using the provided context. Each source has a last_updated date
in its metadata. If the most relevant source is more than 6 months old,
set possiblyStale to true and say so explicitly in your answer, even
though it passed the freshness filter.
""")The hard filter (section 3a) catches genuinely dangerous staleness; this instruction catches the softer case — a doc that's still within the cutoff but old enough that a human should know to double-check it. possiblyStale on CopilotAnswer is what makes that distinction visible in the UI instead of silently baked into a ranking score nobody sees.
4. Proactive Hygiene: A Staleness Audit Job
Reactive filtering protects the engineer asking a question right now. It does nothing for the ninety docs nobody's asked about in months that are quietly rotting. Close the loop with a scheduled sweep:
@Component
public class StalenessAuditJob {
@Scheduled(cron = "0 0 6 * * MON") // every Monday at 6am
public void auditKnowledgeBase() {
List<Document> staleDocs = vectorStore.similaritySearch(
SearchRequest.builder()
.query("") // metadata-only filter, not a semantic search
.filterExpression(new FilterExpressionBuilder()
.lt("last_updated", LocalDate.now().minusMonths(6).toString())
.build())
.topK(100)
.build()
);
Map<String, List<Document>> byOwner = staleDocs.stream()
.collect(Collectors.groupingBy(d -> (String) d.getMetadata().get("owner")));
byOwner.forEach((owner, docs) ->
notificationService.notifyOwnerOfStaleDocs(owner, docs));
}
}Every Monday, each team gets a list of their own docs crossing the 6-month mark — a nudge to review and re-date them, or explicitly mark them deprecated, before a hard-filtered cutoff quietly removes them from the knowledge base's answers entirely. This is the difference between a copilot that's stale-aware and a knowledge base that actually stays current.
5. Putting It Together
@Service
public class KnowledgeBaseCopilotService {
private final ChatClient chatClient;
public KnowledgeBaseCopilotService(ChatClient.Builder builder, VectorStore vectorStore) {
Advisor knowledgeBaseAdvisor = RetrievalAugmentationAdvisor.builder()
.documentRetriever(VectorStoreDocumentRetriever.builder()
.vectorStore(vectorStore)
.similarityThreshold(0.60)
.filterExpression(new FilterExpressionBuilder()
.gte("last_updated", LocalDate.now().minusMonths(24).toString())
.build())
.build())
.documentPostProcessor(new RecencyBoostPostProcessor())
.queryAugmenter(ContextualQueryAugmenter.builder().allowEmptyContext(false).build())
.build();
this.chatClient = builder
.defaultSystem("""
You are an internal engineering knowledge base assistant. Answer using
the provided context only. Always cite the specific sources you used.
If the most relevant source is more than 6 months old, set possiblyStale
to true and say so explicitly.
""")
.defaultAdvisors(knowledgeBaseAdvisor)
.build();
}
public CopilotAnswer ask(String question) {
return chatClient.prompt().user(question).call().entity(CopilotAnswer.class);
}
}One ingestion path carrying real frontmatter metadata, one retrieval path that filters out genuinely dangerous staleness and ranks the rest by recency, one structured contract that forces every answer to show its sources and flag its own uncertainty — the three fixes for the incident that opened this guide.
What's Next
This project leaned hardest on Phase 3's metadata filtering and advanced-RAG post-processing. If you want to build the other side of the same coin — filtering retrieval by who's allowed to see it, not by how old it is — Mini Project 8 (Role-Aware HR & Policy Assistant) applies the identical FilterExpressionBuilder mechanism to access control instead of freshness. Any of the other ten mini-projects in this phase are fair game next; they're deliberately self-paced, not sequential.
Frequently asked questions
Why filter on last_updated with FilterExpressionBuilder instead of just asking the model to judge recency itself?
Because the model only ever sees what retrieval already decided to hand it — if a stale, decommissioned-gateway runbook is the closest semantic match and gets retrieved, asking the model to "be careful about old docs" after the fact is far less reliable than never retrieving it in the first place. Filter at retrieval time for anything where staleness makes content actively wrong, not just less ideal.
Should the hard cutoff (24 months in this guide) be the same for every doc_type?
No — this is exactly why doc_type is captured as its own metadata field during ingestion. A security runbook might need a 3-month cutoff; a general architecture-philosophy doc might reasonably stay valid for years. Parameterize the filter's cutoff by doc_type rather than using one global number, once real usage shows the one-size cutoff is wrong for some category.
What happens to a document once the staleness audit job flags it — does it get deleted?
Not automatically, and it shouldn't be — this guide's job only notifies the owning team. Auto-deleting content is a much higher-risk action than auto-filtering it from retrieval, and a human should decide whether a flagged doc needs updating, explicit deprecation, or is a false positive (a legitimately timeless doc that just hasn't needed an edit).
Does this pattern work for non-Markdown sources like Confluence or a wiki?
The metadata-and-freshness pattern is source-agnostic — the specific ingestion step changes (pull last-modified timestamps from Confluence's API instead of parsing YAML frontmatter, for instance), but everything from section 3 onward (filtering, boosting, flagging, the audit job) is identical once documents carry a last_updated field, regardless of where that field came from.