I am honestly surprised that today's technology still cannot properly handle face-to-face voice translation.

Take Google Translate. Leaving translation quality aside, its speech recognition accuracy alone is a problem. When two people are standing in front of each other trying to talk, even one misheard or mistranslated sentence breaks the flow of the conversation.

When ChatGPT's voice mode came out, I thought the problem was finally solved. My idea was simple: just tell it, "I speak Language A, the other person speaks Language B. Whenever one of us speaks a sentence, translate it. Do not say anything else." That should have turned it into a faithful interpreter.

In practice, after four or five turns, it fell apart. Sometimes it stopped translating, sometimes it translated wrongly, and sometimes it forgot its job entirely and joined the discussion.

Later, when gpt-realtime-2.0 and 2.1 came out one after another, I assumed this had been fixed. In actual use, the drift was still there: the longer you keep a single voice-model session open, the more it starts acting like a conversation partner instead of a translator—answering questions instead of translating them. On top of that, across 120 runs of read Chinese and Thai recordings, gpt-realtime-2.1 made serious errors 17.5% of the time, mostly by mishearing.

I was genuinely shocked.

So I decided to build my own product specifically for face-to-face voice translation. My priorities were simple: accuracy and response time come first; cost is secondary. Cost is a problem that will be solved sooner or later.

The product is built for one situation: two people who share no language, with one phone between them. One person taps or holds a button and speaks; about one second after they stop, the other person hears the translation in their own language and both sentences appear on the screen. Then the other person replies the same way.

To stop the model from drifting into conversation, I stopped reusing sessions. Instead, the translator opens a fresh session for every single sentence, pre-connected in the background so nobody has to wait for it. In my own tests of the model the translator uses, a single session reused across a six-turn conversation answered in the wrong language twice; opening a fresh session per sentence never did. Every response is also checked automatically—for the wrong language, for repeating the original sentence untranslated, for cut-off audio, or for silence. If a check fails, it retries once; if it still fails, the screen says plainly that the sentence was not translated.

Why open-source it?

Because there are far too few products that handle face-to-face translation well. Of everything I have tried, my current product is the best I know of. Right now it uses Gemini 3.8 Live, which cut the serious error rate on those same 120 read recordings down to 6.7%. But Gemini 3.8 Live is simply the best choice at the moment; when a better model comes along, I will switch to it. In fact, because the technology is moving so fast, every technical setup today is temporary. What I really want is to keep getting as close as possible to the best experience that current technology can deliver.

Even so, what I have built still does not meet my own standard for what this product should be. On real, unscripted speech—casual chat, interviews, vlogs—it still makes a serious error on about one sentence in six. I tested a slower model that made far fewer mistakes, but it took about 3.5 seconds and produced only text, no voice, which makes it unusable for live conversation.

Because it is still not good enough, I decided to publish the entire technical setup, including my test sets, in the hope that it helps this category of product move forward and removes the barrier to talking face to face.

I do run a hosted version at livetalktranslate.com. Frankly, it does not get much traffic. Rather than keeping the technology to myself in exchange for a small commercial return, I would much rather push the iteration of face-to-face translation products forward.

So I released the entire framework under the MIT license—the most permissive common open-source license—along with everything I could provide:

I hope that one day someone takes this code, builds on it, wins users, and makes money from it. Business is the strongest force we have for pushing products forward.

You are welcome to use it, build on it, or improve it: