Video Retalking – neural network speaks for you. Another AI that can change the articulation and words of people on video at the user’s request. Just upload the original video with speech and you get a video with a different text.
However, here the synthesized voice obviously differs from the original, but this can be corrected with the help of other services. The open source code of Video Retalking is available on GitHub. (https://github.com/OpenTalker/video-retalking)
What is VideoReTalking?
VideoReTalking is a cutting-edge system designed to edit the facial expressions of individuals in real-world talking head videos based on input audio. What’s phenomenal is its ability to produce high-quality, lip-syncing output videos that can even display different emotions!
The VideoReTalking system breaks down its process into three sequential tasks:
- Face Video Generation with a Canonical Expression: Every frame of the provided talking-head video is modified to match a standardized expression template using the expression editing network. The result is a video that maintains a consistent expression throughout.
- Audio-Driven Lip-Sync: This modified video, along with the provided audio, is passed through a lip-sync network. The outcome? A video that perfectly syncs the lip movements with the input audio.
- Face Enhancement for Photo-realism: The final stage focuses on refining the quality of the synthesized faces. Through an identity-aware face enhancement network and additional post-processing, the visual quality of the faces in the video is significantly improved, ensuring photo-realistic results.
One of the standout features of VideoReTalking is its entirely learning-based approach. All three stages utilize advanced machine learning techniques, and the entire process can be executed sequentially without any manual intervention from the user.
How to Use VideoReTalking?
Getting started with VideoReTalking is straightforward. The steps are as follows:
- Clone the repository from GitHub.
- Set up the environment. This involves creating a dedicated Python environment and installing the necessary dependencies. Specific commands are provided for ease.
- Download the pre-trained models and save them in the ‘checkpoints’ directory.
- Run the inference script! This script includes data preprocessing steps and supports testing any talking face videos without manual alignment. For added flexibility, users can even control the expression of the video subject by utilizing pre-defined templates like “neutral” or “smile.” There’s even the capability to modify the upper face expression with options such as “surprise” or “angry.”
Implications and Acknowledgments
VideoReTalking owes its incredible capabilities in part to prior work in the field. Several other projects and codebases, including Wav2Lip and PIRenderer, have been instrumental in shaping its development.
On the academic front, the technology behind VideoReTalking has been presented at the SIGGRAPH Asia 2022 Conference Track, with contributions from researchers from Xidian University, Tencent AI Lab, and Tsinghua University.
Disclaimer and Usage Rights
It’s essential to understand that VideoReTalking is not an official product of Tencent. Those wishing to utilize the code must adhere to the stipulated open-source license and any intellectual property declarations related to the code. The open-source code operates entirely offline, ensuring that no personal data or other types of information are collected. Users intending to offer services based on this code to end-users must ensure full compliance with relevant laws and regulations.
Furthermore, without express written permission from Tencent, users are not authorized to use legally owned names or logos, like “Tencent.” Lastly, the code’s use for any unlawful activities or those that may harm the legitimate rights of others is strictly prohibited.
In Conclusion
VideoReTalking is a significant leap forward in the world of talking head video editing. Its potential applications span diverse sectors, from the entertainment industry to online education. With its user-friendly setup and profound capabilities, VideoReTalking is poised to redefine the standards of video content editing.
