Structured Outputs & Function Calling
Why 'respond with only the category name' fails silently, and how schema-constrained decoding and tool calling turn a model's output into a real contract.
The Bug That Prompting Alone Can't Fix
The previous guide got the categorizer reasoning about the right boundary between BILLING and FRAUD using few-shot examples. It's still breaking a downstream switch statement on about 1 in 20 requests, because the model phrases a correct classification differently every time — "Billing", "This looks like a billing issue.", sometimes a full sentence with the category buried in the middle. The classification is fine. The contract was never actually a contract — it was a polite request, and the model treated it as one.
"Please respond with only the category name, nothing else" is exactly the kind of instruction a model will follow 95% of the time and quietly ignore the other 5%, because it's still just a polite request sitting in plain text, competing with everything else in the prompt for the model's attention. Fixing this needs a different kind of guarantee than a better-worded instruction.
From "Please Return JSON" to Schema-Constrained Output
The reliable fix isn't a more emphatic instruction — it's telling the API to constrain decoding itself so the model is mechanically restricted to producing only tokens that satisfy a schema. Most providers expose some form of this today, either as a dedicated structured-output mode or by repurposing function/tool calling:
{
"type": "object",
"properties": {
"category": {
"type": "string",
"enum": ["BILLING", "TECHNICAL", "FRAUD", "GENERAL"]
},
"confidence": { "type": "number", "minimum": 0, "maximum": 1 }
},
"required": ["category", "confidence"]
}Pass a schema like this (JSON Schema is the format nearly every provider has converged on) and the difference from prompting alone is structural, not statistical: the model literally cannot emit "category": "Billing-ish" and have it validate, because the decoding process itself is constrained to the schema's allowed values. This closes the categorizer's format bug completely — not "usually," completely, because it's enforced by the API contract rather than by the model choosing to comply.
See OpenAI's Structured Outputs guide and Anthropic's Tool Use docs for the current provider-specific mechanics — the concept (schema-constrained decoding) is consistent across providers even though the API surface for requesting it differs.
Function Calling: Schemas for Actions, Not Just Answers
Structured output constrains a single response. Function calling (also called tool use) extends the same idea to letting the model decide which of several defined actions to take, and with what structured arguments — the model doesn't execute anything itself, it emits a structured request that your application code executes and feeds the result back.
For the categorizer, this is what actually closes the FRAUD-vs-BILLING ambiguity properly: instead of asking the model to guess from ticket text alone, give it a lookup_account_history tool. A brand-new account with two prior disputes and a password change is a very different signal than a five-year account with a single duplicate charge — and that's account data the model has no way to know without a tool call, no matter how good the prompt is. This pattern — letting the model decide when and how to reach for an external capability, rather than hard-coding the sequence — was formalized in Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., 2023), and it's the same primitive that scales up into full agent loops in Phase 7 of this roadmap.
Check yourself
Why does passing a JSON Schema with an enum for 'category' fix the inconsistent-output bug more reliably than adding the instruction 'respond with only the category name, nothing else'?
What's Next
The categorizer now returns a schema-valid category every time, and can pull account history before deciding. Its second bug is unrelated to format or reasoning: a ticket that tries to talk the model into overriding its own instructions. That's the subject of the next guide.
Frequently asked questions
Does structured output / function calling eliminate the need for good prompting?
No — it constrains the shape of the answer, not the quality of the reasoning that produces it. A schema-constrained response can still be wrong; it just can't be malformed. You still need the patterns from the previous guide to get the reasoning right, and the schema to make sure the answer is usable once it is.
What's the difference between structured output and function calling if they use the same schema mechanism?
Structured output constrains a single, final response — useful when you just need a well-shaped answer. Function calling additionally lets the model choose among multiple defined actions and supply arguments for one it selects, which is the primitive that scales up into full tool-using agents in Phase 7.
Can a model call multiple tools in one turn?
Most current provider APIs support this — the model can request several tool calls in a single response, which your application executes (in parallel or in sequence) before sending all the results back together. This matters for latency: fetching account history and prior-ticket count in one round trip is faster than two sequential tool-call turns.