How Browser-Based Video Captioning Actually Works (And What It Costs You)

Almost every online captioning tool works the same way: you upload your video to a server, the server transcribes it, and you download the result. That model is simple to build and it scales predictably. It also means your unpublished footage sits on somebody else's disk.

We built MakeCaption the other way round — the entire pipeline runs inside your browser tab, and no video data is ever transmitted anywhere. This article is a plain description of how that works and, more usefully, what it costs. If you are weighing up captioning tools, the trade-offs below matter more than any feature list.

The four pieces

A browser-based captioning pipeline needs four things that historically only existed as native desktop software. All four now compile to WebAssembly, which is what makes this approach possible at all.

• A media decoder. FFmpeg compiled to WebAssembly, used to pull a 16 kHz mono WAV track out of whatever container you dropped in — MP4, MOV, MKV, WebM.
• A speech recognition model. OpenAI's Whisper, running through Transformers.js. We use the `tiny.en` variant, which is the smallest English model that still produces usable word-level timestamps.
• A timing refinement pass. Raw ASR timestamps drift. We compute an audio energy envelope and nudge word boundaries toward nearby onset peaks.
• A renderer. Caption frames are drawn to an HTML canvas and composited back over the video, so what you preview is produced by the same code that produces the export.

Why word-level timing is the hard part

Getting a transcript is the easy half. Whisper will hand you accurate text for clean audio without much coaxing. What it will not reliably hand you is precise word boundaries — and word boundaries are the entire point of the karaoke-style captions that perform well on short-form video.

Whisper predicts timestamps as part of its output sequence rather than measuring them from the waveform, so the numbers are estimates. On conversational speech they are typically close enough for sentence-level subtitles but visibly loose for word-level highlighting: a word lights up slightly before or after it is actually spoken, and the effect reads as broken even when the text is perfect.

Our fix is unglamorous. We compute a short-time energy envelope over the extracted audio, find its local onsets, and snap each predicted word boundary to the nearest plausible onset within a small window. Where no confident onset exists, we leave the model's estimate alone rather than inventing precision. If you want the deeper background on how the model produces those numbers in the first place, we wrote about that in how AI transcription works.

What it actually costs

Here is the part most privacy-first tools skip over. Running everything locally is not free — you pay for it in bandwidth and CPU on the first run.

• A one-time download of roughly 100 MB. The FFmpeg core is about 32 MB and the Whisper model weights are about 74 MB. Your browser caches both, so this is a first-visit cost, not a per-video cost.
• Real CPU time. Transcription runs on your machine. On a modern laptop a one-minute clip takes well under a minute; on an older phone it is meaningfully slower.
• Memory pressure on long videos. Decoding and re-encoding in a browser tab has a ceiling. Short-form vertical video is comfortably inside it; a 40-minute recording is not.
• English only, for now. Whisper supports many languages, but the `tiny.en` model we ship does not. Larger multilingual weights mean a much larger download.

When server-side is the better choice

We are not going to pretend local processing wins every time. If you are captioning hour-long webinars, batching hundreds of files, or need the highest-accuracy multilingual models, a server-side service will do a better job. Those workloads want hardware you do not have in a browser tab.

Local processing wins when the content is short, when it is unpublished, and when you do not want a copy of it living in someone's storage bucket — which describes most social video before it goes out. It also wins on cost, because there is no per-minute bill to pass on to you.

The accessibility floor

Whichever route you take, automatic output is a starting point rather than a finished caption track. Any ASR system makes mistakes on proper nouns, accents, overlapping speech and technical vocabulary, and the accessibility standards do not grade on a curve. WCAG 2.2 Success Criterion 1.2.2 expects captions to convey the audio accurately, which in practice means a human read-through before publishing.

This matters commercially as well as ethically. The World Health Organization estimates that around 430 million people worldwide require rehabilitation for disabling hearing loss, and captions are also used heavily by people with no hearing impairment at all — 80% of consumers in one large survey said they are more likely to finish a video when captions are available. We cover the legal dimension in our guide to ADA compliance for video captions.

Try it and check the network tab

The claim that nothing is uploaded is the kind of thing every tool asserts and few let you verify. This one you can check yourself: open your browser's developer tools, switch to the Network tab, and run a video through the captioning tool. You will see the model weights come down. You will not see your video go up.