Multimodal: Image Generation & Audio
A voice support line that reuses the entire existing chat pipeline unchanged — transcription in, the same SupportChatService in the middle, speech synthesis out — plus image generation for auto-illustrated reports.
Two Requests From Very Different Teams
Product wants a voice support line — customers call in, leave a message describing their issue, and get a spoken response back, both for phone-based support and as an accessibility option for customers who prefer not to type. Separately, whoever owns the internal weekly digest (this roadmap's automated-report mini-project) wants a simple auto-generated chart image alongside the text summary, not just prose. Both requests are the same lesson: Spring AI wraps image generation, transcription, and speech synthesis behind the identical Model<Request, Response> shape as ChatModel — auto-configured beans, not a new architecture to learn.
1. Transcription: Turning a Voicemail Into Text
@Autowired
TranscriptionModel transcriptionModel;
String transcript = transcriptionModel.transcribe(voicemailAudioResource);That's the convenience method for the common case. For more control:
AudioTranscriptionOptions options = AudioTranscriptionOptions.builder()
.language("en")
.responseFormat(AudioTranscriptionOptions.TranscriptResponseFormat.TEXT)
.build();
AudioTranscriptionPrompt prompt = new AudioTranscriptionPrompt(voicemailAudioResource, options);
AudioTranscriptionResponse response = transcriptionModel.call(prompt);
String transcript = response.getResult().getOutput();The payoff is that nothing downstream needs to know the input was ever audio. Once transcript exists, it's a String — it goes straight into SupportChatService.reply(...) from this roadmap's second guide, completely unchanged, memory advisor and all:
@PostMapping("/voicemail")
public String handleVoicemail(@RequestParam String conversationId, @RequestParam MultipartFile audio) throws IOException {
String transcript = transcriptionModel.transcribe(new ByteArrayResource(audio.getBytes()));
return supportChatService.reply(conversationId, transcript);
}A transcribed voicemail is exactly as untrusted as typed chat input — this phase's security guide's "content is data, not instructions" discipline doesn't get to relax just because the text arrived via speech. A caller reciting "ignore previous instructions and..." into a voicemail is the identical direct-injection attempt from that guide's opening story, just spoken instead of typed.
2. Text-to-Speech: Turning the Reply Back Into Audio
@Autowired
TextToSpeechModel textToSpeechModel;
byte[] audioBytes = textToSpeechModel.call(replyText);With explicit options — voice, format, speed, depending on what the configured provider supports:
TextToSpeechOptions options = TextToSpeechOptions.builder()
.voice("alloy")
.responseFormat("mp3")
.build();
TextToSpeechResponse response = textToSpeechModel.call(new TextToSpeechPrompt(replyText, options));
byte[] audioBytes = response.getResult().getOutput();The full round trip
@PostMapping(value = "/voicemail", produces = "audio/mpeg")
public byte[] handleVoicemail(@RequestParam String conversationId, @RequestParam MultipartFile audio) throws IOException {
String transcript = transcriptionModel.transcribe(new ByteArrayResource(audio.getBytes()));
String replyText = supportChatService.reply(conversationId, transcript);
return textToSpeechModel.call(replyText);
}Three model calls chained — transcription, chat, speech synthesis — and the middle one is the exact SupportChatService this roadmap has been building since its second guide, with RAG, memory, and tools all still attached, entirely unaware its input originated as a phone call.
Each of these three calls is a separate network round-trip to a model provider, with its own failure mode — wrap all three with this phase's resilience guide's timeout and circuit-breaker patterns, not just the chat call in the middle. A voicemail pipeline that hangs on a slow transcription call is exactly the failure story that guide opened with, just with an extra step.
3. Image Generation
@Autowired
ImageModel imageModel;
ImageResponse response = imageModel.call(
new ImagePrompt("A simple bar chart icon representing weekly support ticket volume, flat design, blue and white",
OpenAiImageOptions.builder()
.quality("standard")
.n(1)
.height(512)
.width(512)
.build())
);
String imageUrl = response.getResult().getOutput().getUrl();ImagePrompt in, ImageResponse out — the same request/response wrapper shape as Prompt/ChatResponse, just for a different model type. This is the piece behind Phase 9's automated weekly report generator mini-project: pair a generated summary from ChatClient with a generated illustrative image from ImageModel in the same report, both auto-configured beans in the same Spring context.
Image generation is not a fit for anything that needs to be factually precise — asking a model to generate a chart from data (as opposed to describing or summarizing data you already charted with a real charting library) risks a plausible-looking image with numbers that don't actually match your data. Use it for illustration and decoration, not as a substitute for an actual data-visualization pipeline.
Phase Wrap-Up
Across this phase's four guides: prompt injection defense (direct, and indirect through RAG and tool results), output validation before acting on model decisions, resilience patterns that keep one bad provider day from taking down the whole app, and now vision, image generation, and audio — all still the same ChatClient, Media, Model<Request,Response> shapes this entire roadmap has built on since its first guide. Nothing here was a new architecture; it was the same patterns applied to new failure modes and new input types.
What's Next
That closes the structured, phase-by-phase part of this roadmap. Phase 9 is where all of it gets applied — eleven self-paced mini-projects, each drawing on a different combination of everything built across all eight phases, including the security, resilience, and multimodal skills from this one specifically (the invoice extractor's scanned-document handling and the failover router's circuit breaker both lean directly on this phase).
Frequently asked questions
Do transcription and text-to-speech calls support streaming, like chat does?
Text-to-speech commonly does, for reducing time-to-first-audio the same way chat streaming reduces time-to-first-token — check the specific provider's support and Spring AI's StreamingTextToSpeechModel interface. Transcription is typically request-response only, since it needs the complete audio input before it can produce a transcript.
Can I use a different provider for chat versus transcription versus text-to-speech in the same pipeline?
Yes — each is a separately auto-configured bean, so nothing stops picking OpenAI for chat, a different provider for transcription, and ElevenLabs for speech synthesis in the same application, based on whichever is strongest or cheapest for each specific task. This is the same model-mixing idea this roadmap's cost-optimization guide applied to chat models, extended across modalities.
Should the voicemail transcript be saved verbatim, or only the chat reply?
Save both if you can, for the same reason this roadmap's evaluation guide argued for keeping real production examples in a golden set — a saved transcript lets you audit what the customer actually said versus what the model responded to, which matters if a reply is ever disputed or reviewed.
Is there a simpler way to add voice support than building this whole pipeline by hand?
This roadmap deliberately shows the underlying primitives so you understand what's actually happening at each step, but managed voice-agent platforms exist that bundle telephony, transcription, and synthesis behind a higher-level API. Whether that tradeoff — less code, less control, another vendor dependency — makes sense depends on how deeply voice needs to integrate with logic you already own, like this roadmap's existing memory and RAG pipeline.