AI Therapy Microservice
An AI service for a healthcare client that turns therapy session audio into transcripts and summaries, 300+ sessions a day. It first shipped CPU-only and now runs on Azure AI Foundry.
My roleSole backend engineer
The idea
The problem
Therapy sessions are recorded as audio. Turning an hour of speech into a transcript and a summary is slow, and a request that waits for the answer would time out. The data is also private, so it has to be handled with care at every step.
The answer
An event-driven service. The API accepts the audio and returns at once. A queue carries the work to background workers that transcribe and then summarise, so the slow part never blocks anyone.
How a session travels
Audio goes in one end and a transcript plus a summary comes out the other. Every step can fail and be retried on its own.
- 1UPLOAD
- 2QUEUE
- 3TRANSCRIBE
- 4SUMMARISE
- 5DELIVER
The API verifies the caller's token against the identity provider's JWKS, stores the audio encrypted at rest, and answers straight away.
A message on Azure Service Bus describes the job. This decouples receiving audio from processing it.
Audio is transcribed in batches. The first build ran Faster-Whisper on CPU.
GPT-5.4 mini on Azure AI Foundry writes the summary from the transcript. The first build used Phi-3.5-Mini, quantised to INT8.
The result is stored and the caller is told. MLflow records how each run behaved.
The move to Azure AI Foundry
The service started on self-hosted models and has since shifted to Azure AI Foundry.
Azure AI Foundry
Azure resources only in the region the client specified
GPT-5.4 mini
Audio transcribed in batches
The hard parts
Running language and speech models on CPU, at volume, with private data.
The first build ran Faster-Whisper and Phi-3.5-Mini (INT8) CPU-only on 15 GB pods in Azure Kubernetes Service. Quantising to INT8 is what made that fast and small enough.
Threads fought over locks and ONNX ran out of memory under load. A multi-processing worker pool gave each worker its own memory and removed the contention.
Jobs that fail are retried, and the ones that keep failing land in a dead-letter queue, where they wait to be looked at instead of being lost.
JWKS token verification on every call, audio encrypted at rest, Azure resources only in the region the client specified, and MLflow observability so that problems show up as numbers before they show up as complaints.
Built with
| Platform | Azure AI Foundry, with Azure resources only in the region the client specified |
|---|---|
| Models | GPT-5.4 mini for summaries, batched audio transcription |
| First build | Faster-Whisper and Phi-3.5-Mini (INT8) on CPU-only 15 GB pods in Azure Kubernetes Service |
| Messaging | Azure Service Bus, with dead-letter queues and retry logic |
| API | FastAPI, JWKS token verification |
| Observability | MLflow |