Building a Real-Time Customer Support Voice Agent with Twilio, FastAPI, Gemini Live, ElevenLabs, and Celery

By Rustamjon Akhmedov, AI/ML engineer and software architect

API details below were verified against official vendor documentation on 2026-08-08. Live demo: https://callie.flance.info

A voice agent that answers a phone call in under a second feels like magic. One that stays coherent when the caller interrupts mid-sentence, when the WebSocket drops at second 43, and when the same webhook arrives twice — that is engineering.

I built a phone receptionist called Callie to find out where the second problem starts. This is the architecture that came out of it, with the hard parts called out honestly.

The customer support use case

The target is narrow on purpose: a business phone number that always answers. Callie greets the caller, answers service questions, takes callback requests, and captures support issues. When the call ends, the system turns the conversation into a structured record — intent, customer details, priority, sentiment — and opens a ticket if warranted.

That split matters. The live call needs sub-second reactions; the record-keeping needs correctness and retries. Two different systems, and most of the design follows from refusing to merge them.

Why a demo is easier than a reliable system

A demo needs one clean path: audio in, audio out, nobody interrupts. Production needs everything else. Callers talk over the agent. Networks stall. Providers cap sessions — Gemini Live audio-only sessions are capped at 15 minutes without context compression, and the WebSocket lives roughly 10 minutes, with the server sending a GoAway message carrying the time remaining before it closes. Twilio retries webhooks.

To be clear about my position: this is a production-oriented architecture I have built and validated end to end. I am not claiming thousands of live production calls. An earlier public voice demo of mine sits at https://google-azure-vagent.flance.info/.

High-level architecture

The real-time path is:

Caller → Twilio number → HTTP webhook → FastAPI returns TwiML (Twilio Markup Language) → Twilio opens a bidirectional Media Stream WebSocket → FastAPI bridges audio to the Gemini Live API → response audio streams back to Twilio → the caller hears it.

There are two voice configurations. Configuration A is native Gemini Live speech-to-speech: Twilio → FastAPI → Gemini → FastAPI → Twilio. Configuration B keeps Gemini for understanding and routes the voice through ElevenLabs streaming text-to-speech (TTS).

Everything after the call — summarising, extracting, ticketing — runs on Celery, a distributed task queue, with Redis as broker and PostgreSQL as the durable store. Docker Compose runs the API, the Celery worker, Redis, and PostgreSQL.

How Twilio and FastAPI establish the call

The most misunderstood point: the HTTP webhook does not become the WebSocket. Twilio makes a form POST, reads your TwiML response, closes that request, then opens a brand-new WebSocket to the URL you named. Two connections, two handlers.

@router.post("/incoming")
async def incoming_call(request: Request) -> Response:
    form = await _validated_form(request)  # X-Twilio-Signature check
    ws_url = settings.public_base_url.replace("https://", "wss://") + "/voice/media-stream"

    response = VoiceResponse()
    connect = Connect()
    connect.stream(url=ws_url)      # <Connect><Stream> = bidirectional
    response.append(connect)
    return Response(content=str(response), media_type="application/xml")

<Connect><Stream> is the bidirectional form; <Start><Stream> is receive-only — it forks a copy of the call audio to you but gives you no way to send audio back. wss:// is the only supported protocol. Signature validation uses Twilio's RequestValidator against the exact public URL Twilio signed — never the internal one behind your proxy.

The media socket is a thin dispatcher; state lives in a per-call session object.

@router.websocket("/voice/media-stream")
async def media_stream(ws: WebSocket) -> None:
    await ws.accept()
    session = None
    try:
        while True:
            msg = json.loads(await ws.receive_text())
            if msg["event"] == "start":
                session = CallSession(ws, msg["start"])
                await session.start()
            elif msg["event"] == "media" and session:
                await session.on_caller_audio(msg["media"]["payload"])
            elif msg["event"] == "stop":
                break
    finally:
        if session:
            await session.close()   # cleanup always runs

Twilio sends audio/x-mulaw at 8000 Hz, mono, base64-encoded; the server can send media, mark, and clear back.

Connecting FastAPI to Gemini Live

Gemini Live wants raw 16-bit PCM (pulse-code modulation) audio at 16 kHz in (audio/pcm;rate=16000) and always returns 16-bit PCM at 24 kHz. Twilio speaks 8 kHz μ-law. So every frame needs a codec conversion and a resample, both directions — and resampler state must be kept per direction, per call, or you get audible clicks at every frame boundary.

def mulaw8k_to_pcm16k(mulaw: bytes, state=None):
    pcm8k = audioop.ulaw2lin(mulaw, 2)
    return audioop.ratecv(pcm8k, 2, 1, 8000, 16000, state)

def pcm24k_to_mulaw8k(pcm24k: bytes, state=None):
    pcm8k, state = audioop.ratecv(pcm24k, 2, 1, 24000, 8000, state)
    return audioop.lin2ulaw(pcm8k, 2), state

The bridge splits the work three ways: the WebSocket handler feeds caller frames to Gemini as they arrive, one background task drains Gemini's responses onto a bounded outbound queue, and a second pumps that queue back to Twilio as small fixed-size frames. The queue cap is the backpressure mechanism — a few seconds of audio at most. Without it, a slow consumer becomes a memory leak.

async def on_caller_audio(self, payload_b64: str) -> None:
    pcm16k, self._in_state = mulaw8k_to_pcm16k(base64.b64decode(payload_b64), self._in_state)
    await self._gemini.send_realtime_input(
        audio=types.Blob(data=pcm16k, mime_type="audio/pcm;rate=16000"))

Turn-taking is not mine to implement. Gemini Live runs automatic voice activity detection (VAD) server-side, and its default activityHandling is START_OF_ACTIVITY_INTERRUPTS — the caller speaking cancels the model's generation.

When ElevenLabs should be used

Configuration B is optional. Gemini Live already returns speech, so Configuration A is complete on its own; ElevenLabs is a voice-quality layer or a fallback.

Here is where I have to correct a common design assumption, including my own. The clean version of Configuration B would run Gemini in text-only mode and send text to the TTS provider. As of 2026-08-08, current Gemini API Live models are native-audio and support only the AUDIO response modality; the docs direct you to output audio transcription when you need text. I could not find a documented text-only Live session for any current model.

So the implementation runs AUDIO with outputAudioTranscription, discards Gemini's audio in this mode, and streams the transcription text into the ElevenLabs stream-input WebSocket with output_format=ulaw_8000 — already Twilio-ready, so no transcoding on that branch.

url = (f"wss://api.elevenlabs.io/v1/text-to-speech/{voice_id}"
       f"/stream-input?model_id={model_id}&output_format=ulaw_8000")
ws = await websockets.connect(url, additional_headers={"xi-api-key": os.environ["ELEVENLABS_API_KEY"]})
await ws.send(json.dumps({"text": " ",
    "generation_config": {"chunk_length_schedule": [90, 120, 160, 250]}}))

The trade-off is real and I would rather state it than hide it: you pay Gemini's generation, wait on transcription, then pay TTS. eleven_flash_v2_5 is documented at around 75 ms model latency excluding application and network time, and a tighter chunk_length_schedule starts audio sooner — but transcription lag remains. Use B when voice matters more than the last hundred milliseconds.

Barge-in and conversation state

Each call is a small state machine: connecting, listening, processing, speaking, interrupted, completed. Interruption is the interesting transition, and it takes three actions.

async def _handle_barge_in(self) -> None:
    self.state = SessionState.INTERRUPTED
    if self._el:
        await self._el.abort()          # 1. stop the TTS generation
    while not self._out_queue.empty():
        self._out_queue.get_nowait()    # 2. drop audio we already produced
    await self._send_json({"event": "clear", "streamSid": self.stream_sid})  # 3. flush Twilio
    self.state = SessionState.LISTENING

The trigger is server_content.interrupted from Gemini. The third step is the one people miss: audio you already handed to Twilio is buffered on Twilio's side, and only clear flushes it. Skip it and the agent keeps talking over the caller for a second after it has logically stopped.

The mirror of clear is mark. You send a labelled mark after a turn; Twilio echoes it back once that audio has finished playing. That echo is how you know the caller heard everything, rather than guessing.

Each call gets its own session object, Gemini connection, queues, and resampler state, so concurrency isolation is structural rather than remembered.

Why Celery stays outside the live audio path

Celery is excellent at durable, retryable work and completely wrong inside an audio loop that moves a new frame every few tens of milliseconds. A task round-trip through Redis costs more than the whole latency budget for one frame, and a worker restart would drop a live conversation.

So the rule is absolute: during the call, nothing touches Celery, and nothing touches the database except one transcript write at close. Celery starts only after Twilio reports the call completed.

The post-call asynchronous workflow

Twilio may deliver the same status callback more than once, so idempotency comes first — enforced by the database, not application logic:

@router.post("/status")
async def call_status(request: Request) -> Response:
    form = await _validated_form(request)   # X-Twilio-Signature check
    ...
result = await db.execute(
    pg_insert(CallEvent)
    .values(call_sid=call_sid, event_key=f"status:{status}", payload=form)
    .on_conflict_do_nothing(constraint="uq_call_event")   # UNIQUE (call_sid, event_key)
)
if not result.rowcount:
    return Response(status_code=204)     # duplicate: already handled
if status == "completed":
    process_completed_call.delay(call_sid)

ON CONFLICT DO NOTHING against a unique constraint makes duplicate delivery a no-op under concurrency, which an application-level check-then-insert does not.

The worker then does the slow, failure-prone work.

@celery_app.task(bind=True, max_retries=3, autoretry_for=(Exception,),
                 retry_backoff=True, retry_backoff_max=120,
                 retry_jitter=True, acks_late=True)
def process_completed_call(self, call_sid: str) -> None:
    ...

retry_backoff with jitter avoids retry storms, acks_late means a killed worker's task is redelivered rather than lost, and bounded max_retries stops a broken call cycling forever. Tasks that exhaust their retries belong in a dead-letter queue for human review, not in silence.

Structured extraction and validation

Free-form LLM (large language model) text is not a data integration. The post-call extraction uses Gemini's standard generate_content API with a Pydantic model as the response_schema, so output is JSON matching a declared contract:

class CallExtraction(BaseModel):
    intent: CallIntent = CallIntent.other
    summary: str = ""
    customer: CustomerInfo = Field(default_factory=CustomerInfo)
    callback: CallbackRequest = Field(default_factory=CallbackRequest)
    support_issue: SupportIssue = Field(default_factory=SupportIssue)
    follow_up_required: bool = False
    sentiment: str = Field(default="neutral", pattern="^(positive|neutral|negative)$")

Enums and regex-constrained fields mean a ticket priority is one of four known values or validation fails loudly. That is the difference between a system your customer relationship management (CRM) platform can trust and one that quietly writes nonsense.

Production failures and recovery

The failures worth designing for are ordinary. Gemini's ten-minute WebSocket lifetime means long calls need session resumption and context window compression rather than optimism. Twilio can disconnect mid-stream, so cleanup runs in a finally block — cancel tasks, close the upstream session, persist what you have. Every external call needs a timeout and bounded retries. And there should always be a path to a human, because "I could not help with that" is worse than a warm transfer.

Security, privacy, and HIPAA awareness

Transport Layer Security (TLS) everywhere — wss:// is mandatory for Media Streams anyway. Validate X-Twilio-Signature on every webhook, including the media connection. Keep secrets in environment variables and a managed secret store, never in code. Apply least privilege to database and provider credentials, encrypt at rest and in transit, and keep transcripts and caller identifiers out of ordinary logs — log the call identifier (the SID), not the caller's words.

Recording requires explicit consent, plus a retention and deletion policy that is actually executed. Add role-based access control, audit logging of transcript access, and tenant isolation if you serve multiple businesses.

If you serve a covered entity — or act as its business associate — and medical information can reach the system, the United States Health Insurance Portability and Accountability Act (HIPAA) imposes obligations, and Business Associate Agreements with your providers are a legal requirement. Encryption alone does not make a system HIPAA-compliant — compliance is an organisational programme, and the architecture is only one input to it.

Observability and metrics

The metric that predicts caller satisfaction is time to first audio. Track it as a distribution, per configuration, not as an average. Alongside it: barge-in count and how quickly clear followed, outbound queue depth (rising depth means you are outproducing Twilio's drain), session duration against provider caps, task retry and dead-letter counts, and extraction validation failure rate.

Then cost per call and cost per successful outcome — a resolved question, a captured callback — because the second number tells you whether the agent is worth running.

Practical lessons

Draw the boundary between the real-time and durable paths before writing code; almost every later decision falls out of it. Let the database enforce idempotency. Keep interruption to three explicit steps and test it deliberately. Verify audio formats against current documentation rather than memory — the Gemini text-only assumption cost me a redesign. Treat the optional TTS layer as a trade-off, not a free upgrade.

Conclusion

Real-time voice AI is a latency and state-management problem wearing an AI costume. The model is the easy part now. What separates a demo from a system is the boundary between the live loop and the durable pipeline, disciplined interruption handling, and schema-validated output downstream systems can rely on.

Callie is at https://callie.flance.info. Happy to compare notes with anyone building in this space.

References

Twilio - https://www.twilio.com/docs/voice/twiml/stream - https://www.twilio.com/docs/voice/media-streams/websocket-messages - https://www.twilio.com/docs/usage/webhooks/webhooks-security - https://www.twilio.com/docs/voice/api/call-resource

Google Gemini API - https://ai.google.dev/gemini-api/docs/live-api/capabilities - https://ai.google.dev/gemini-api/docs/live-guide - https://ai.google.dev/gemini-api/docs/live-session - https://ai.google.dev/api/live

ElevenLabs - https://elevenlabs.io/docs/websockets - https://elevenlabs.io/docs/models - https://elevenlabs.io/docs/cookbooks/text-to-speech/twilio

FastAPI, Celery, PostgreSQL - https://fastapi.tiangolo.com/advanced/websockets/ - https://docs.celeryq.dev/en/stable/userguide/tasks.html - https://www.postgresql.org/docs/current/sql-insert.html

The full grouped reference list with verification dates is in references.md.