Where the 2.4 seconds go in real-time AI dubbing

Latency is the whole product

For a delegate wearing a receiver, dubbing quality is dominated by one thing: how far behind the speaker the translated audio arrives. Accuracy matters, but a perfect translation that lands four seconds late is unusable in a room where people are laughing and applauding on the original.

The number worth targeting is under three seconds end to end. Here is what that budget is spent on.

The five stages

1. Capture and transport to the processor — 80 to 200 ms. Analogue from the desk, converted and carried to wherever the inference runs. On-venue this is tens of milliseconds. Sent to a cloud region it is whatever the round trip costs, which from an Indian venue to a Singapore region is typically 60 to 90 ms each way before anything is computed.

2. Speech recognition — 300 to 700 ms. Streaming recognisers emit partial results quickly but revise them. Committing too early produces text that gets corrected mid-sentence; waiting produces latency. Most of the tuning effort in a dubbing pipeline is spent on exactly this trade.

3. Translation — 200 to 500 ms. Faster than recognition, and the stage where accuracy is most negotiable against speed. Translating phrase by phrase is quick and produces stilted output; waiting for a clause boundary is better and costs several hundred milliseconds.

4. Speech synthesis — 300 to 600 ms to first audio. Modern neural voices can begin emitting before the full sentence is synthesised, which is what makes real-time viable at all. A system that synthesises whole sentences before playing adds the length of the sentence to your latency.

5. Distribution to receivers — 100 ms to several seconds. This is the stage people forget. Dedicated RF receivers are fast. Delivering over the venue Wi-Fi to delegates' phones through a standard streaming protocol can add two to five seconds on its own, which will dominate everything above it.

The stage most worth fixing is usually the last one

Teams spend their optimisation effort on the model stages, because that is where the interesting engineering is. But a pipeline with a 1.4-second inference chain delivered over HLS with six-second segments has a six-second problem, not a 1.4-second one.

If the delivery path is a standard segmented streaming protocol, changing it to a low-latency transport is worth more than any amount of model tuning.

Where clause boundaries hurt

Languages that place the verb late — and several Indian languages do, relative to English — force the translation stage to wait for more of the sentence before it can commit. This is a property of the language pair, not a deficiency in the system, and it means your latency budget is not the same for every target language on the same event.

Plan for the worst pair on your list, not the average.

What to measure on the day

Put a clap track through the system at rehearsal and measure the gap on a receiver with a stopwatch app. It is crude and it is the number your delegates experience.

Measure per target language, not once. Measure with the hall full, because the Wi-Fi behaves differently with 800 phones on it than with four.

And measure again after the first session, because thermal throttling on the GPU rig is real and the second hour is not always the first hour.