In today’s interconnected world, effective communication across language barriers is crucial. From global businesses seeking to expand their reach to individuals connecting with friends and family across borders, the need for seamless multilingual communication is ever-growing. Speech-to-speech translation solutions are revolutionizing how we overcome language barriers, and the integration of AI voice cloning technology is taking this innovation to new heights.
This technology offers a range of benefits across various sectors:
- Video bloggers:
- Reach a larger, global audience
- Share content across languages while maintaining their unique personality through voice translation.
- Media companies:
- Expand coverage internationally
- Deliver localized content that feels authentic and engaging to diverse audience.s
- Businesses:
- Translate marketing materials, customer service interactions, and internal communications.
- Streamline operations and enhance global reach
- Education:
- Personalize language learning with AI voice cloning
- Provide immersive, engaging experiences to boost language acquisition effectiveness
This article will explore the exciting possibilities of speech-to-speech translation solutions with AI voice cloning, examine its possibilities with existing models, and discuss the future of this transformative technology.
Upgrading Speech-to-speech Translation with AI Voice Cloning
Speech-to-speech translation has long struggled with a crucial missing piece: preserving the human voice itself. While traditional methods could accurately translate words, they couldn’t capture the speaker’s unique voice – that personal touch that makes communication truly meaningful. Think about watching your favorite TED talk in another language, only to hear a completely different voice delivering those inspiring ideas. Something gets lost in the overall impression, right?
Here is where AI voice cloning steps in to change the game. It’s not just about speech synthesis of translated words with one pre-saved voice anymore – it’s about recreating the entire speaking experience. Modern AI technology can now do something remarkable: it learns the unique qualities of a person’s voice – how they express emotion, their natural rhythm, even their distinctive accent. Then, it recreates that same voice, speaking fluently in other languages. This breakthrough in voice preservation is just the beginning of what AI voice cloning brings to the table.
AI voice cloning revolutionizes video translation by offering several key advantages:
- Maintaining Brand Consistency:
- AI voice cloning ensures a consistent brand/person voice by replicating the original speaker’s voice across languages.
- It builds strong brand and personal recognition and trust with audiences worldwide.
- Preserving Emotional Authenticity:
- Unlike traditional dubbing, AI voice cloning accurately replicates the speaker’s unique intonations, emotions, and delivery style.
- This results in a more natural and engaging translated experience for viewers.
- Cost-Effectiveness:
- Eliminating the need to hire multiple voice actors for each language significantly reduces production costs.
- It makes video translation more accessible and cost-effective for businesses of all sizes.
- Accelerated Production:
- AI voice cloning streamlines the translation process.
- Generating audio in multiple languages becomes significantly faster with a cloned voice, enabling quicker production of new and new videos.
Here are several quick examples of how even simple translation significantly improved video stats.
Example 1: How Google Voice Translation brings new audience
“Історія без міфів”(History without myths) is a popular Ukrainian historical channel.
The video on the top of the image is the most popular video in the Ukrainian language, and the second one is the most popular video dubbed with Google Text to Speech.
Even though Google TTS didn’t copy a speaker’s voice, there is a big demand for the same translated content in English.
Example 2: Video dubbing shows great potential to get a bigger audience.
Mr Beast’s initiative to dub his content into multiple languages has significantly expanded his global reach. For instance, his Spanish-language channel, MrBeast en Español, boasts over 26.5 million subscribers and has accumulated over 3.18 billion views across 67 videos.
This strategic move has increased his subscriber base and enhanced viewer engagement worldwide.
In fact, creators testing multi-language dubbed videos observed that over 15% of their watch time originated from viewers accessing dubbed audio tracks in non-primary languages.
In fact, creators testing multi-language dubbed videos observed that over 15% of their watch time originated from viewers accessing dubbed audio tracks in non-primary languages.
The numbers speak for themselves – AI voice cloning delivers measurable results across different content niches. The language barriers that once limited content reach are steadily falling. Today, every creator has the potential to build a truly global audience – efficiently, authentically, and without breaking the bank. To better understand how to implement these powerful capabilities, let’s examine the core methodologies behind speech-to-speech translation.
Speech-to-Speech Translation Implementation Methodology
When implementing speech-to-speech translation solutions, developers can choose between fundamental approaches, each with its own trade-offs and technical challenges.
Speech-to-Speech Translation (S2ST) primarily employs two methodologies:
- Cascade Approach: This traditional method executes translation through three consecutive components. The process begins with Speech-to-Text (S2T), which converts audio input into written form. Next, Machine Translation (MT) renders the content in the target language. Finally, Text-to-Speech (T2S) synthesis generates natural-sounding audio using voice cloning to preserve the original speaker’s characteristics or use one of the pre-saved voices only, like Google T2S.
- Direct Approach: This more advanced methodology translates audio directly into the target language without text conversion steps. The approach offers several unique advantages: it operates as a single unified model, potentially reduces processing time, and better preserves prosodic elements of speech. However, current implementations face notable challenges for language pair coverage and training data requirements, making it a promising but still-maturing technology in the S2ST landscape.
Our Approach:
We chose the Cascade approach due to its simplicity, ongoing development, and the maturity of its individual components. This strategy offers flexibility and positions us well to adopt future advancements without being locked into a single rigid pipeline.
Similar to how prompt engineering has transformed the use of large language models—often achieving results comparable to fine-tuning with significantly less effort—we see a parallel in voice cloning technology. We believe that as zero-shot voice cloning continues to evolve, the Cascade approach will benefit from these improvements and may ultimately outperform more rigid, end-to-end solutions like Direct S2ST. This forward-looking perspective guided our decision to follow a modular, adaptable path.
The choice between these methodologies often depends on the specific models available for implementation. Here are the leading models that represent each approach:
- YourTTS: A zero-shot multilingual text-to-speech model that excels in the Cascade approach. It can clone voices with just a few seconds of audio input and supports over 40 languages.
- Translatotron and Seamless Expressive: These end-to-end models pioneer the Direct approach, with a focus on preserving speech characteristics across languages while eliminating the need for intermediate steps.
While these models do a great job of speech-to-speech translation, there are also specialized solutions such as TorToiSe and StyleTTS2 that are specifically focused on voice cloning for video content. Let’s take a look at these two powerful models in this area.
TorToiSe Voice Cloning Technology Analysis
Among video voice cloning solutions, TorToiSe takes a unique neural approach. This powerful system prioritizes quality over processing speed. Let’s examine how it works and what makes it distinct.
What is TorToiSe?
TorToiSe was developed by James Betker, a former Google Software Developer and Research Engineer at OpenAI who contributed to DALL-E image generation models. The system comprises five independently trained neural networks that innovatively combine transformer-based autoregressive decoders with diffusion models to generate MEL spectrograms. The final component decodes these highly compressed speech representations back into actual waveforms.
Both auto-regressive generation and diffusion steps contribute to making the model as slow as a turtle… or tortoise!
See more on architecture here, check the TorToiSe Github. Article on model
Setting Up TorToiSe: Core Requirements
TorToiSe model is said to perform voice cloning from the voice reference samples and then generate a speech with a given text using the cloned voice features. Documentation mentioned to have 1-10 audios of 5-10 seconds each.
This setting frames the requirements for the system to be built around. We need to:
- Transcribe audio and translate text
- Create a dataset of a person speaking
- Generate audio for each text segment
AI Dubbing pipeline with TorToiSe-TTS
Figure <number> – General AI dubbing pipeline
Our TorToiSe-based AI dubbing pipeline combines five open-source models with Google Translate API to create a comprehensive speech-to-speech translation system.
The process begins with source separation using SpeechBrain models, which employ Sepformer architecture—a sophisticated dual-path transformer network that isolates voice from background noise. This step improves WER for Speech to Text step and clears the audio for smooth voice cloning. While effective, this step can become a quality bottleneck as some models are trained on 8-16kHz sample rate datasets, and the audio would need to be downsampled, which inevitably damages the quality.
Figure <number> – AI dubbing pipeline based on TorToiSe and open-source models
After initial separation, we enhance the audio quality using SpeechBrain’s Mtl Mimic Voicebank model. This advanced system utilizes a unique approach where a separate Speech Recognition model provides perceptual loss functions for the Mimic Loss system, resulting in clearer, crisper audio, correcting any downsampling artifacts from the previous stage.
For videos containing multiple speakers, we implement Speaker Diarization through a cascade process: first applying speaker segmentation with sliding windows, then extracting and clustering speaker embeddings to identify distinct voices.
This crucial step allows us to create separate voice samples for each speaker, enabling individualized voice cloning.
We then use OpenAI’s Whisper model to receive text transcription. This transformer-based network with an encoder-decoder architecture will ineratively transcribe the separated speech segments.
The transcribed content passes through Google API for text translation to the target language. Though we identified limitations in contextual accuracy that we addressed in subsequent pipeline iterations.
Finally, TorToiSe extracts voice embeddings from speech segments prepared earlier and synthesizes the translated text using the cloned voice characteristics. As a diffusion-based system combining five independently trained neural networks, TorToiSe offers exceptional voice quality but requires significant processing time—generating samples can take 80-120 minutes depending on configuration settings like autoregressive samples (i.e. 10) and diffusion iterations (i.e. 200-400).
Our Experience with TorToiSe: Strong and Weak Points
We tested these capabilities with real-world content, most notably in dubbing the travel documentary “10 Days in VIETNAM: Hanoi, Ha Long Bay, Hoi An, Ho Chi Minh, Hue | Full Travel Vlog & Guide.” The results showcased TorToiSe’s ability to maintain the speaker’s vocal characteristics while transitioning to English, creating a seamless viewing experience for international audiences. Also, we achieved significantly improved audio quality compared to standard TTS systems.
10 Days in VIETNAM: Hanoi, Ha Long Bay, Hoi An, Ho Chi Minh, Hue | Full Travel Vlog & Guide
Excessive training data and sophisticated architecture allow TorToiSe to generate a high-quality voice-over TTS. An author used standard datasets LibriTTS and HiFiTTS, which gave him 896 hours of high-quality transcripted audio. But then, he scraped 49,000 hours more!
For content with multiple speakers, such as “First Impressions of SHANGHAI, CHINA! Travel Vlog,” we leveraged TorToiSe alongside our custom pipeline to not only translate the dialogue but also enhance translation accuracy beyond what Google Translate initially provided. This combined approach allowed us to preserve each speaker’s unique vocal identity while delivering contextually accurate translations.
First Impressions of SHANGHAI, CHINA! Travel Vlog
However, despite these strengths, TorToiSe has notable limitations. The model struggles with cross-lingual voice cloning unless fine-tuned with English samples from the specific speaker, which is impractical for most automated dubbing scenarios. This limitation forced us to develop workarounds when processing non-English source content. This couldn’t be the case for Speech-to-speech translation for automatic AI Dubbing.
Perhaps the most significant drawback is TorToiSe’s processing time requirements. Real-Time Factor varied from 500 to 1000 for the generation of voice samples on the previous slide, even though we used a faster inference TorTuiSe (KV cache, quantization), which was not its original, even slower version. This means that for just 10 seconds of audio, processing could take 83-166 minutes—making it impractical for large-scale production without substantial computational resources.
StyleTTS2 System Analysis
For our second stage of development, we simplified the system’s architecture by replacing the computationally intensive TorToiSe with the Real-Time speech generator, StyleTTS2. While StyleTTS2 is less effective at cross-lingual voice cloning, it excels at capturing the speaker’s intonations.
This model doesn’t require a speaker dataset. It only needs reference audio of the segment you want to synthesise.
What is StyleTTS2?
StyleTTS2 is a diffusion text-to-speech model designed for faster inference. It also controls the duration of the generated audio using a text aligner, which is important for timing synchronization. The pitch extractor helps it track F0 and match the speaker’s intonation from the reference audio.
We have built a tool on GitHub that follows the pipeline below.
Figure <number> – Simplified AI dubbing pipeline with StyleTTS2 for one speaker only
We found that many steps could be combined using more advanced models, like WhisperX, which can handle transcription, translation, and time-stamping. However, this approach works only for a single speaker in the video.
The StyleTTS2 Usage for AI Dubbing
The StyleTTS2 AI dubbing process starts with obtaining the original video, which in our case shows a Spanish speaker that needs to be dubbed in English.
¡Llegamos a JAPÓN! | Lo más cool de Tokio 🇯🇵
The extracted audio is then processed through WhisperX, which transcribes and translates the speech, producing both the original text and its English translation with timestamps.
We found that the quality of speech synthesis increases if we use the longest window. Therefore, we combine the text to fill the window as much as possible, enabling long-term iterative generation.
Next, the system cuts the reference audio samples based on the timestamps and pairs them with the translated text.
With no additional hyperparameters, we tested different model checkpoints (LibriSpeech, Vokan, LJSpeech) to get different voice options while keeping the original intonation. People share their checkpoints on HuggingFace.
When generating the English audio, StyleTTS2 processes these paired segments to create natural-sounding speech that matches the rhythm and style of the original. It works in real time, unlike the earlier TorToiSe system, which was much slower and took hours to process short audio clips.
Our Experience with StyleTTS2: Strong and Weak Points
StyleTTS2 operates in real-time, generating speech much faster than alternatives like TorToiSe. It can produce moderate-quality audio in seconds rather than minutes or hours.
The model allows for control over speech duration, which is essential for video dubbing, where timing must match the original footage. It also preserves the intonation patterns from the reference audio, maintaining the speaker’s emotional expression and emphasis.
StyleTTS2 is straightforward to use with minimal hyperparameters to adjust. This makes it accessible for developers without requiring extensive technical knowledge to achieve good results.
However, StyleTTS2 lacks true voice cloning capabilities. While it can mimic intonation, it cannot fully reproduce the original speaker’s unique timbre and vocal characteristics.
The voice selection is limited to pre-trained checkpoints. Users must choose from available voice models rather than generating a customized voice based on the original speaker.
The quality of results is described as “moderate,” suggesting there’s room for improvement in the naturalness and authenticity of the generated speech compared to more resource-intensive approaches.
Future Development of Speech-to-speech Translation Solutions
AI dubbing field is advancing rapidly with two promising developments on the horizon. Pyannote’s new model combines source separation and speaker diarization into a single process, streamlining the pipeline while improving overall quality. This integration means fewer models are needed, making the system more efficient and potentially more accurate.
pyannote/speech-separation-ami-1.0
Even more exciting is the upcoming StyleTTS-ZS model, created by StyleTTS2’s author. This new iteration specifically targets zero-shot voice cloning capabilities, addressing one of StyleTTS2’s key limitations. Though not yet released, the model’s GitHub repository has already garnered 160 stars, indicating strong community interest. When available, developers could simply swap StyleTTS2 with StyleTTS-ZS to achieve significant performance improvements in the voice cloning field.
These developments suggest speech-to-speech translation technology will soon deliver more natural-sounding, voice-matched dubbing with a simpler implementation.
Conclusion
Speech-to-speech translation with AI voice cloning is changing how we communicate across languages. We’ve seen how TorToiSe offers quality but requires time, while StyleTTS2 works faster but with less voice accuracy. Soon, new models like StyleTTS-ZS will likely bridge this gap.
These tools help video bloggers connect with global audiences and keep their personal speaking style. As technology improves, we’ll be able to speak in our natural voices to anyone, no matter their language. The future of communication isn’t just about translating words – it’s about keeping our unique voices too.
If you’re looking for a custom AI voice cloning solution tailored to your needs, Trembit is here to help. Let’s bring your voice to the world—seamlessly and authentically!
