Real-time AI dubbing explained: the pipeline, the latency, the trade-offs

The four hops

Real-time dubbing is four operations in sequence, and the latency is the sum:

1. 01Transcribe — streaming speech recognition on the floor feed
2. 02Translate — source text to each target language
3. 03Synthesise — text to speech, voice-matched to the speaker
4. 04Deliver — to a delegate's phone or a venue receiver

On our standard two-node rig we measure a 2.4-second median from speaker's lips to delegate's ear. It divides roughly: 0.3s transcribe, 0.6s translate, 0.9s synthesise, 0.6s deliver.

Why the synthesis step is the expensive one

Transcription and translation are close to solved for conference speech. Synthesis is where the seconds and the money go, because voice-matched output has to wait for enough context to sound like a sentence rather than a stream of words.

You can cut synthesis latency by shortening that context window. The output then sounds choppy and delegates stop using it — which is the actual failure mode, not the number on a spec sheet.

Latency is not the metric that matters most

Below about three seconds, delegates stop noticing lag and start following the content. Above about five, they give up and read the slides. The useful question is not "how low can you go" but "does it hold under load".

Ours holds at 2.4 seconds for six languages on the standard rig. Beyond eight target languages we add a second GPU node, because the synthesis step is what saturates first.

Run it at the venue, not in the cloud

This is the decision that matters most and gets the least attention.

Cloud dubbing means the pipeline depends on the venue uplink. We have watched a nine-minute uplink failure take out a cloud translation trial mid-session while the room sat in silence. With the pipeline on hardware at the venue, that same failure took out the public stream and nothing in the hall.

Venue uplinks fail. Design as though yours will.

Where machine dubbing is the wrong answer

We say this before contracts, not after:

• //Regulatory and legal sessions. If a translation error has consequences, use a certified human interpreter. We keep booths for these even on events where everything else is machine-dubbed.
• //Heavy accents plus dense jargon plus poor mic discipline. Any two are manageable. All three degrade transcription enough to matter, and the fix is a better mic plan, not a better model.
• //Panel sessions with crosstalk. Speaker separation is the weak point. A moderator who enforces one voice at a time is worth more than any model upgrade.

What to ask a supplier

Ask for median and 95th-percentile latency, measured on site rather than in a lab. Ask what happens when the uplink drops. Ask how many languages before the rig needs another node. Ask which session types they would recommend a human interpreter for — a supplier who says "none" has not run enough events.