Getting Started with Spring AI: LLM Concepts, Setup & Your First Call

The Friday Afternoon That Started This Guide

Picture a mid-sized fintech's backend team. Product wants an AI assistant inside the support portal by next sprint — "just call OpenAI, how hard can it be?" A backend engineer opens WebClient, hand-builds a JSON payload for the Chat Completions endpoint, wires up an API key from an environment variable, and gets a working demo by lunch.

Then reality sets in over the following weeks:

• Legal wants to evaluate Anthropic instead of OpenAI for data-handling reasons — that means rewriting the request/response mapping from scratch.
• The retry logic, timeout handling, and token-usage logging that were "quick to add" for one provider now need to be duplicated for every provider anyone wants to try.
• A teammate wants to run Ollama locally during development, so CI doesn't burn real API credits on every pull request — different SDK, different response shape, again.
• Nobody can answer "how many tokens did we spend on this feature last week?" without grepping logs.

None of this is exotic — it's the exact same problem Spring solved for JDBC in 2003 and for messaging with JmsTemplate in 2006: too many almost-identical, hand-rolled integrations against different vendor SDKs. Spring AI exists to be that abstraction layer for LLMs. This guide gets you from zero to your first working AI call, and — more importantly — gives you the vocabulary to understand why every line of that code looks the way it does.

1. What Is an LLM, Really?

A Large Language Model (LLM) is a piece of software that has been trained on enormous amounts of text and learned one skill extremely well: predicting the next chunk of text, given everything that came before it. Ask it a question, and under the hood it's repeatedly answering "what token is most likely to come next?" until it decides it's done.

That's the whole trick. Everything else — the illusion of reasoning, the code it writes, the summaries it produces — emerges from that one repeated prediction, applied one small piece of text at a time.

Tokens: the model's unit of currency

Models don't read whole words. They read tokens — sub-word chunks that are roughly ¾ of a word on average in English. "unbelievable" might become un, believ, able. "Spring AI" might become two or three tokens. This matters for three very practical reasons:

1. You pay per token. Every hosted provider (OpenAI, Anthropic, Azure) bills you for input tokens (what you send) and output tokens (what comes back).
2. Models have a token limit — the context window — beyond which they simply can't see earlier parts of the conversation.
3. Latency scales with output tokens. A response that generates 2,000 tokens takes noticeably longer than one that generates 50.

The context window

The context window is the total number of tokens (input + output combined) a model can "see" in a single call. If a model has a 128K-token context window and your conversation history plus your retrieved documents plus your question adds up to more than that, the oldest content simply gets cut off — the model never sees it. This is why, later in this roadmap, an entire skill is dedicated to managing conversation memory instead of just appending every message forever.

Base models vs. instruction-tuned vs. chat vs. reasoning models

Not all LLMs behave the same way out of the box:

Spring AI's ChatModel interface is built around the chat-model shape — every provider integration speaks in the same system/user/assistant vocabulary, even though the raw APIs underneath look different.

Temperature and sampling: the creativity dial

When a model predicts the next token, it doesn't just pick one "correct" answer — it computes a probability distribution over thousands of candidate tokens. Temperature controls how the model samples from that distribution:

• Low temperature (0–0.3): almost always pick the most likely token → consistent, deterministic, "boring" output. Good for classification, extraction, code generation.
• High temperature (0.7–1.2): sample more broadly, including less-likely tokens → more varied, creative output. Good for brainstorming, marketing copy, story generation.

top_p (nucleus sampling) is a related dial: instead of considering all tokens, it only considers the smallest set of tokens whose combined probability crosses a threshold (e.g. the top 90%). top_k is a blunter version of the same idea — instead of a probability threshold, it caps the candidate pool to a fixed number of the k most-likely tokens (e.g. only the top 40). Most teams pick temperature, or one of top_p/top_k, and leave the rest at their default — tuning all three at once mostly just makes output harder to reason about.

System, user, and assistant messages

Every chat-model conversation is a list of role-tagged messages:

• System message — sets the persona, constraints, and ground rules ("You are a support agent for Acme Corp. Never discuss competitor pricing."). Sent once, usually at the start.
• User message — what the human (or your application, on the human's behalf) is asking.
• Assistant message — what the model replied. In multi-turn conversations, you send prior assistant replies back as part of the next request — the model itself has no memory between calls.

That last point surprises a lot of newcomers: an LLM API call is stateless. If you want a "conversation," your application is responsible for resending the relevant history every single time. We'll build exactly that mechanism — ChatMemory — later in this phase.

2. Why Spring AI, Specifically?

You could talk to an LLM provider with plain RestClient calls, and plenty of prototypes do. Spring AI earns its place the same way Spring Data earns its place over hand-written JDBC: not by making the simple case dramatically shorter, but by making the production case tractable.

Concretely, Spring AI gives you:

• A portable ChatModel interface. Your business logic calls the same Java interface regardless of which provider is configured underneath — exactly the same idea as coding against JdbcTemplate instead of a vendor-specific driver API.
• Spring Boot auto-configuration. Add a starter dependency and a couple of properties, and ChatModel, EmbeddingModel, and ImageModel beans are ready to @Autowired — no manual client construction, no manual retry/timeout wiring.
• A consistent request/response model. Prompt, Message, ChatResponse, ChatOptions mean the same thing whether you're talking to GPT-5 or a local Ollama model.
• Cross-cutting concerns solved once. Observability (Micrometer), structured output parsing, chat memory, and RAG all plug into the same ChatClient via advisors — you'll meet these in the next two phases.

That fintech team from the opening story, six months later with Spring AI in place, handles "try Anthropic instead" as a one-line property change and a redeploy — not a rewrite. That's the entire value proposition of this phase.

3. Setting Up Spring AI

This roadmap defaults to Gradle as the build tool (Maven works identically — every artifact coordinate below is the same, only the syntax differs) and defaults to Ollama as the first provider, so you can follow every guide in this roadmap without an API key or a cent of spend. Hosted providers (OpenAI, Anthropic, Azure) are covered right after as a drop-in swap.

Scaffold the project with Spring Initializr

Don't hand-write a build.gradle from scratch — start.spring.io (the same Spring Initializr that powers File → New → Spring Project in your IDE) already knows the correct Spring AI starter coordinates and wires the BOM for you. Pick Gradle - Groovy, Java, the latest Spring Boot 4.1.x release, and search for the Ollama dependency under "AI."

Prefer the terminal? Spring Initializr is just an HTTP API — this generates and downloads an equivalent project in one line:

curl https://start.spring.io/starter.zip \
  -d type=gradle-project \
  -d language=java \
  -d bootVersion=4.1.0 \
  -d javaVersion=21 \
  -d dependencies=web,spring-ai-ollama \
  -o spring-ai-demo.zip && unzip spring-ai-demo.zip -d spring-ai-demo

Swap spring-ai-ollama for spring-ai-openai or spring-ai-anthropic in that dependencies list (or add more than one — Initializr accepts a comma-separated list) if you already know you want a hosted provider from day one.

What Initializr generated: the BOM

Open the generated build.gradle and you'll find Spring AI's Bill of Materials (BOM) already wired up — the same pattern as spring-boot-dependencies, ensuring every Spring AI module you add resolves to compatible versions:

// build.gradle
plugins {
    id 'java'
    id 'org.springframework.boot' version '4.1.0'
    id 'io.spring.dependency-management' version '1.1.7'
}
 
dependencyManagement {
    imports {
        mavenBom "org.springframework.ai:spring-ai-bom:2.0.0"
    }
}
 
dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.ai:spring-ai-ollama-spring-boot-starter'
}

You'll rarely hand-edit this section — it matters mainly when you add a second provider starter later and want to confirm it resolves against the same BOM version.

Configure it — Ollama first

Ollama needs one piece of software installed outside your Spring app: the Ollama runtime itself, which serves models locally on http://localhost:11434.

# Install Ollama (see ollama.com for your OS), then pull a model once:
ollama pull llama3.1
 
# Ollama runs as a background service after install — verify it's up:
curl http://localhost:11434

# application.yml — Ollama (local, runs on http://localhost:11434 by default)
spring:
  ai:
    ollama:
      chat:
        options:
          model: llama3.1

That's it — no API key, no billing account, no rate limit to watch. This is exactly why the fintech team in the opening story ran their whole CI suite against Ollama and only pointed at a hosted provider for staging and production.

Configure it — hosted providers (OpenAI, Anthropic, Azure)

Swap the starter dependency (spring-ai-openai-spring-boot-starter, spring-ai-anthropic-spring-boot-starter, …) and the matching properties. Your Java code — the part you're about to write in the next section — does not change at all.

// build.gradle — add alongside, or instead of, the Ollama starter
implementation 'org.springframework.ai:spring-ai-openai-spring-boot-starter'

# application.yml — OpenAI
spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-5-mini
          temperature: 0.7

Now the payoff for that opening story: Legal asked for Anthropic instead of OpenAI. Here's the entire change — one dependency, one properties block, zero Java:

// build.gradle — swap the starter, nothing else in the project changes
implementation 'org.springframework.ai:spring-ai-anthropic-spring-boot-starter'

# application.yml — Anthropic
spring:
  ai:
    anthropic:
      api-key: ${ANTHROPIC_API_KEY}
      chat:
        options:
          model: claude-sonnet-4-5
          temperature: 0.7

Azure OpenAI and Google Vertex AI follow the identical shape — a dedicated starter (spring-ai-azure-openai-spring-boot-starter, spring-ai-vertex-ai-gemini-spring-boot-starter) plus a spring.ai.<provider>.* properties block. The WelcomeMessageService from the next section is about to be written once and never touched again, no matter which of these four you point it at.

With the starter and these properties in place, Spring Boot auto-configures a ChatModel bean. There is no new OpenAiChatModel(...), new AnthropicChatModel(...), or new OllamaChatModel(...) anywhere in your code — that's the auto-configuration doing its job, identically to how a DataSource bean appears once you add a JDBC driver and a connection URL.

4. Your First Call: ChatModel

ChatModel is the lowest-level interface in Spring AI — one method, call(Prompt), returning a ChatResponse.

@Service
public class WelcomeMessageService {
 
    private final ChatModel chatModel;
 
    public WelcomeMessageService(ChatModel chatModel) {
        this.chatModel = chatModel;
    }
 
    public String greet(String customerName) {
        Prompt prompt = new Prompt(List.of(
            new SystemMessage("You are a friendly onboarding assistant for a fintech app. Keep replies under 2 sentences."),
            new UserMessage("Write a welcome message for a new customer named " + customerName + ".")
        ));
 
        ChatResponse response = chatModel.call(prompt);
        return response.getResult().getOutput().getText();
    }
}

Walk through what just happened:

1. SystemMessage sets the persona and a constraint ("under 2 sentences") — this is prompt engineering in its simplest form: telling the model the rules of the conversation before asking it anything.
2. UserMessage is the actual request.
3. chatModel.call(prompt) sends both to the configured provider and blocks until a response arrives.
4. ChatResponse wraps one or more Generation results (usually one) — .getResult().getOutput().getText() drills down to the plain string reply.

Because ChatModel is injected as a Spring bean, this exact code works unchanged whether application.yml points at OpenAI, Anthropic, or a local Ollama model. Swap the starter dependency and the properties — the WelcomeMessageService class doesn't move.

5. The Nicer Way: ChatClient

ChatModel is deliberately minimal. For anything beyond a single call, Spring AI gives you ChatClient — a fluent builder, conceptually similar to RestClient or WebClient, which will also be the extension point for chat memory, structured output, and RAG later in this roadmap.

@Service
public class WelcomeMessageService {
 
    private final ChatClient chatClient;
 
    public WelcomeMessageService(ChatClient.Builder chatClientBuilder) {
        this.chatClient = chatClientBuilder
            .defaultSystem("You are a friendly onboarding assistant for a fintech app. Keep replies under 2 sentences.")
            .build();
    }
 
    public String greet(String customerName) {
        return chatClient.prompt()
            .user("Write a welcome message for a new customer named " + customerName + ".")
            .call()
            .content();
    }
}

A few things worth noticing:

• ChatClient.Builder is itself an auto-configured, injectable bean — Spring Boot wires it up once the starter and spring.ai.chat.client.enabled (default true) are in place.
• .defaultSystem(...) sets the persona once, at construction time, instead of repeating it on every call — build one immutable ChatClient per use case and reuse it, the same way you'd reuse a single RestClient instance rather than rebuilding it per request.
• .call().content() is the shortest path to a plain-text reply. Later phases replace .content() with .entity(MyRecord.class) for typed output, or add .advisors(...) for memory and RAG — the fluent chain is designed to grow without changing its shape.

Per-request options

Sometimes one call in your service needs a different temperature or model than the default — a classification endpoint that wants temperature 0, sitting next to a creative-writing endpoint that wants 0.9. Override ChatOptions per call instead of per bean:

chatClient.prompt()
    .user("Classify this support ticket as BILLING, TECHNICAL, or GENERAL: " + ticketText)
    .options(ChatOptions.builder().temperature(0.0).build())
    .call()
    .content();

6. Putting It Together: A Support-Ticket Triage Endpoint

Back to the fintech team. Here's a small, realistic slice of what they actually shipped in week one — an endpoint that classifies an incoming support ticket before it's routed to a human queue.

@RestController
@RequestMapping("/api/tickets")
public class TicketTriageController {
 
    private final ChatClient chatClient;
 
    public TicketTriageController(ChatClient.Builder builder) {
        this.chatClient = builder
            .defaultSystem("""
                You triage customer support tickets for a fintech app.
                Reply with exactly one word: BILLING, TECHNICAL, FRAUD, or GENERAL.
                """)
            .build();
    }
 
    @PostMapping("/triage")
    public String triage(@RequestBody String ticketText) {
        return chatClient.prompt()
            .user(ticketText)
            .options(ChatOptions.builder().temperature(0.0).build())
            .call()
            .content()
            .trim();
    }
}

Notice temperature 0.0 here — this is a classification task, not a creative one. Deterministic, consistent categorization is exactly what a support-routing queue needs, and it's a two-character config change away from the same code powering a much more freewheeling "draft a reply" feature elsewhere in the app.

What's Next

You now know what a token, a context window, and temperature actually mean — and you've made real calls through both ChatModel and ChatClient, backed by an auto-configured provider. The next guide in this roadmap builds multi-turn conversations with ChatMemory, structures model output into typed Java records instead of raw strings, and templates prompts properly with PromptTemplate.