| Author Name | Affiliation | Postcode | | Haiquan Fang* | School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China | 310023 | | Shuya Shi | School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China | 310023 | | Hanlin Lan | Zhejiang Institute of Administration, Hangzhou 311121, China | 311121 | | Peile Lin | School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China | 310023 | | Yangyang Lin | School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China | 310023 | | Zhongtian Chen | School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China | 310023 | | Yu Lin | School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China | 310023 |
|
| Abstract: |
| In order to reduce the deployment cost of the artificial intelligence model, the model needs to be lightweight. Due to the large size of the Wav2Lip384 model weight file and the high GPU VRAM consumption, we applied TensorRT to perform FP16 quantization on the Wav2Lip384 model. The test was conducted on a computer equipped with an RTX 4090 GPU 24GB VRAM. The results showed that the size of the quantized model weight file was reduced by 50.16%. In terms of inference performance, compared to the original model, the quantized model preserved satisfactory performance, with LSE-D increasing by 0.63%, LSE-C decreasing by 1.32%, and FID increasing by 5.29%. We further applied the quantized Wav2Lip384 model in LiveTalking to build a real-time streaming digital human. Testing revealed that the VRAM consumption decreased by 80.77%. Using the proposed model, more digital humans could be deployed in cases of limited computational resources, thus effectively reducing costs and improving economic benefits in engineering applications. Additionally, we made improvements to LiveTalking. We integrated ASR, LLM, TTS, and the quantized Wav2Lip384 model into LiveTalking to build a lightweight model for a real-time high-definition voice interactive digital human. This provides a reference solution for engineering applications. |
| Key words: digital human quantized model Wav2Lip TensorRT LiveTalking |
| DOI:10.11916/j.issn.1005-9113.26008 |
| Clc Number:TP39 |
| Fund: State Key Program of National Social Science Foundation of China (Grant No. 22AKS019), and Humanities and Social Sciences Pre Research Fund Project of Zhejiang University of Technology (Grant No.SKY-ZX-20220207) |
|
| Descriptions in Chinese: |
| 针对Wav2Lip384模型权值文件体积大、GPU显存占用高的问题,我们应用TensorRT对Wav2Lip384模型进行FP16量化。在RTX 4090 GPU 24GB VRAM计算机上进行测试,结果发现,量化模型权值文件的体积减少50.16%。在模型推理效果方面,相对于原模型,量化模型的推理效果维持在较好的水平,LSE-D增加0.63%、LSE-C减少1.32%、FID增加5.29%。进一步,把Wav2Lip384量化模型应用于LiveTalking,构建实时流式数字人,经过测试发现,显存占用降低80.77%。在有限的计算资源下,可以部署更多数字人,在工程应用上可以有效降低成本,提高经济效益。最后,我们对LiveTalking进行了改进。我们把语音识别、聊天大模型、语音合成、Wav2Lip384量化模型集成到LiveTalking,构建实时高清语音交互数字人轻量化模型,这可为工程应用提供一个参考方案。 |