Abdul HaqueHire me
Back to my work
WRK-02 · Dec 2025 – Present · client work

AI Therapy Microservice

An AI service for a healthcare client that turns therapy session audio into transcripts and summaries, 300+ sessions a day. It first shipped CPU-only and now runs on Azure AI Foundry.

Azure AI FoundryGPT-5.4 miniAzure AKSService Bus

My roleSole backend engineer

Daily sessions300+through the pipeline
Memory per pod15 GBCPU-only first build, no GPU
Model precisionINT8first build, quantised for CPU speed

The idea

The problem

Therapy sessions are recorded as audio. Turning an hour of speech into a transcript and a summary is slow, and a request that waits for the answer would time out. The data is also private, so it has to be handled with care at every step.

The answer

An event-driven service. The API accepts the audio and returns at once. A queue carries the work to background workers that transcribe and then summarise, so the slow part never blocks anyone.

How a session travels

Audio goes in one end and a transcript plus a summary comes out the other. Every step can fail and be retried on its own.

  1. 1UPLOAD
  2. 2QUEUE
  3. 3TRANSCRIBE
  4. 4SUMMARISE
  5. 5DELIVER
1. UPLOAD

The API verifies the caller's token against the identity provider's JWKS, stores the audio encrypted at rest, and answers straight away.

2. QUEUE

A message on Azure Service Bus describes the job. This decouples receiving audio from processing it.

3. TRANSCRIBE

Audio is transcribed in batches. The first build ran Faster-Whisper on CPU.

4. SUMMARISE

GPT-5.4 mini on Azure AI Foundry writes the summary from the transcript. The first build used Phi-3.5-Mini, quantised to INT8.

5. DELIVER

The result is stored and the caller is told. MLflow records how each run behaved.

The move to Azure AI Foundry

The service started on self-hosted models and has since shifted to Azure AI Foundry.

Platform

Azure AI Foundry

Region

Azure resources only in the region the client specified

Summaries

GPT-5.4 mini

Transcription

Audio transcribed in batches

The hard parts

Running language and speech models on CPU, at volume, with private data.

No GPU

The first build ran Faster-Whisper and Phi-3.5-Mini (INT8) CPU-only on 15 GB pods in Azure Kubernetes Service. Quantising to INT8 is what made that fast and small enough.

Lock contention and out-of-memory

Threads fought over locks and ONNX ran out of memory under load. A multi-processing worker pool gave each worker its own memory and removed the contention.

Failures are normal

Jobs that fail are retried, and the ones that keep failing land in a dead-letter queue, where they wait to be looked at instead of being lost.

Private by default

JWKS token verification on every call, audio encrypted at rest, Azure resources only in the region the client specified, and MLflow observability so that problems show up as numbers before they show up as complaints.

Built with

PlatformAzure AI Foundry, with Azure resources only in the region the client specified
ModelsGPT-5.4 mini for summaries, batched audio transcription
First buildFaster-Whisper and Phi-3.5-Mini (INT8) on CPU-only 15 GB pods in Azure Kubernetes Service
MessagingAzure Service Bus, with dead-letter queues and retry logic
APIFastAPI, JWKS token verification
ObservabilityMLflow

Next voyage