AI Awareness for Backend Engineers: LLMs and Tool Calling

AI Awareness for Backend Engineers

You don't need to train a model or understand transformer attention math to ship an AI feature responsibly. What you need — and what most backend teams are missing — is the same rigor you already apply to any external dependency: know its failure modes, bound its cost, validate its output, and never trust it with more authority than it's earned. This guide gives you exactly enough LLM literacy to design, review, and operate an AI-adjacent feature in a backend service. It is deliberately not a full LLM engineering course — for that depth (RAG architectures, fine-tuning, evaluation harnesses, agent frameworks), this site's dedicated Gen AI track goes much further.

1. What an LLM Actually Is, at the Level You Need

A large language model is a function: given a sequence of text, it predicts a probability distribution over the next token, samples one, and repeats. That's it. Everything else — chat behavior, "reasoning," tool use — is built on top of that one primitive, shaped by how the model was trained (pretraining on massive text corpora, then fine-tuned on instruction-following and human-preference data).

The API shape you'll actually integrate against

Every major LLM provider (Anthropic, OpenAI, and others) exposes essentially the same shape: you send a list of role-tagged messages, you get back generated text (optionally streamed token-by-token) plus usage metadata.

{
  "model": "claude-sonnet-4-5",
  "max_tokens": 1024,
  "system": "You are a support-ticket summarizer. Output valid JSON only.",
  "messages": [
    { "role": "user", "content": "Summarize: Customer reports checkout button unresponsive on mobile Safari after promo code applied." }
  ]
}

2. Tokens and Context Windows

Models don't operate on characters or words — they operate on tokens, sub-word chunks produced by a tokenizer. This matters for three very concrete backend reasons: cost, latency, and hard input limits.

// Rough token estimation for cost/limit checks before calling the API —
// use the provider's official tokenizer for anything precision-critical
int estimatedTokens = text.length() / 4;  // ballpark only, not exact

3. Prompt Engineering Fundamentals

A prompt is the only interface you have to steer model behavior — there's no config file, no method override. Treat it like an API contract you're writing against an unpredictable collaborator.

The three roles

Practical techniques that consistently work

BAD (vague, underspecified):
"Summarize this support ticket."
 
BETTER (explicit format, constraints, and failure behavior):
"Summarize the support ticket below in exactly 2 sentences.
Extract the customer's core issue and any error messages verbatim.
If no clear issue is stated, respond with exactly: {"issue": "unclear"}.
Respond only with valid JSON matching this schema: {"issue": string, "severity": "low"|"medium"|"high"}.
 
Ticket: <ticket text>"

4. Why Hallucinations Happen, and How to Mitigate Them

A hallucination is the model generating fluent, confident, and factually wrong output — a fabricated API method, a citation to a paper that doesn't exist, a summary that includes a detail never present in the source. This isn't a bug in a particular model; it's an inherent property of next-token generation without a built-in fact-checking mechanism.

Mitigation strategies, from cheapest to most involved

5. Function / Tool Calling

Tool calling (also called function calling) is how you let a model take actions or fetch live data instead of only generating text. You describe available functions with a schema; the model decides when to invoke one and with what arguments; your code executes the actual function — the model never runs anything itself.

Defining a tool

{
  "name": "get_order_status",
  "description": "Look up the current status and ETA for a customer order by its order ID.",
  "input_schema": {
    "type": "object",
    "properties": {
      "orderId": {
        "type": "string",
        "description": "The order ID, e.g. '4521'."
      }
    },
    "required": ["orderId"]
  }
}

6. Wiring an LLM API into a Spring Boot Service

Once you get past the prompt, an LLM API call is an external HTTP dependency — treat it with the same production discipline as any third-party API call: timeouts, retries, circuit breaking, and observability.

@Configuration
public class LlmClientConfig {
 
    @Bean
    RestClient llmRestClient(@Value("${llm.api.base-url}") String baseUrl,
                              @Value("${llm.api.key}") String apiKey) {
        return RestClient.builder()
            .baseUrl(baseUrl)
            .defaultHeader("x-api-key", apiKey)
            .defaultHeader("anthropic-version", "2023-06-01")
            .requestFactory(clientHttpRequestFactory())
            .build();
    }
 
    private ClientHttpRequestFactory clientHttpRequestFactory() {
        var factory = new SimpleClientHttpRequestFactory();
        factory.setConnectTimeout(3_000);   // fail fast on connection issues
        factory.setReadTimeout(20_000);     // generation can legitimately take several seconds
        return factory;
    }
}

@Service
public class TicketSummarizerService {
 
    private final RestClient llmRestClient;
    private final MeterRegistry meterRegistry;
 
    public TicketSummarizerService(RestClient llmRestClient, MeterRegistry meterRegistry) {
        this.llmRestClient = llmRestClient;
        this.meterRegistry = meterRegistry;
    }
 
    @CircuitBreaker(name = "llmApi", fallbackMethod = "fallbackSummary")
    @Retry(name = "llmApi")
    @RateLimiter(name = "llmApi")
    public TicketSummary summarize(String ticketText) {
        // 1. Bound the input — never send unbounded user content
        String truncated = truncateToTokenBudget(ticketText, 4_000);
 
        long start = System.nanoTime();
        LlmResponse response = llmRestClient.post()
            .uri("/v1/messages")
            .body(buildRequest(truncated))
            .retrieve()
            .body(LlmResponse.class);
        meterRegistry.timer("llm.request.latency").record(Duration.ofNanos(System.nanoTime() - start));
        meterRegistry.counter("llm.tokens.used",
            "type", "total").increment(response.usage().totalTokens());
 
        // 2. Validate before trusting the output
        TicketSummary summary = parseAndValidate(response.text());
        return summary;
    }
 
    private TicketSummary fallbackSummary(String ticketText, Throwable t) {
        // Degrade gracefully — never let an AI feature take down the whole request
        return TicketSummary.unavailable();
    }
}

# application.yml — Resilience4j config for the LLM dependency
resilience4j:
  circuitbreaker:
    instances:
      llmApi:
        sliding-window-size: 20
        failure-rate-threshold: 50
        wait-duration-in-open-state: 30s
  retry:
    instances:
      llmApi:
        max-attempts: 2
        wait-duration: 500ms
        retry-exceptions:
          - java.io.IOException
          - org.springframework.web.client.HttpServerErrorException
  ratelimiter:
    instances:
      llmApi:
        limit-for-period: 50
        limit-refresh-period: 1m
        timeout-duration: 0s

Cost control checklist

PII and data handling

7. Observability for AI Features

An LLM call is a dependency like any other, but its failure modes are different enough to need their own signals.

Key takeaways

• An LLM predicts the next token by statistical plausibility, not by fact-checking — this single property explains hallucination and why validation is mandatory, not optional.
• Tokens, not characters, are the unit of cost and context-window limits — bound your input size explicitly rather than trusting the provider's limit as your only safeguard.
• Prefer native structured-output/tool-calling modes over "please respond in JSON" prompt instructions — they're validated and far more reliable.
• Grounding (supplying real source data in the prompt) is the highest-leverage, lowest-effort mitigation for hallucination.
• In tool calling, the model only decides what to call and with what arguments — your backend code executes it and must independently validate and authorize every argument.
• Never let LLM output directly trigger an irreversible action (payment, email, deletion) without validation or human confirmation.
• Wire an LLM API exactly like any other external dependency: explicit timeouts, capped retries, a circuit breaker, and cost/latency metrics — plus token-usage tracking, which has no equivalent in typical REST integrations.
• Treat every LLM API call as third-party data transmission — mask PII, understand the provider's retention/training policy, and don't over-log full transcripts.

Interview Questions

• In your own words, what does an LLM actually compute when generating a response?
• Why do LLMs hallucinate, and why is this not simply a "bug that will be fixed" in the next model version?
• What is grounding, and why does it reduce hallucination more reliably than prompt instructions alone?
• What's the difference between a token and a word, and why does that distinction matter for cost and context-window limits?
• How does function/tool calling actually work — what does the model do, and what does your backend code do?
• Why should a tool exposed to an LLM never be a raw SQL-execution or shell-execution function?
• If a model calls a tool with an order ID that doesn't belong to the requesting user, whose responsibility is it to catch that, and why?
• How would you design retry behavior for a call to an LLM API? What should and shouldn't be retried?
• What cost controls would you put in place before shipping an LLM-backed feature to production?
• Why is max_tokens an important parameter to set explicitly on every request?
• What PII-handling considerations apply specifically to sending data to an LLM API, beyond a normal third-party API call?
• Why might you prefer a provider's native structured-output feature over asking the model to "output JSON" in a plain-text prompt?
• Describe how you would build a circuit breaker and fallback path around an LLM dependency in a Spring Boot service.
• What metrics would you add to a dashboard specifically for an AI-backed feature, beyond standard latency/error-rate?