Please submit manuscripts in either of the following two submission systems

    ScholarOne Manuscripts

  • ScholarOne
  • 勤云稿件系统

  • 登录

Search by Issue

  • 2026 Vol.33
  • 2025 Vol.32
  • 2024 Vol.31
  • 2023 Vol.30
  • 2022 Vol.29
  • 2021 Vol.28
  • 2020 Vol.27
  • 2019 Vol.26
  • 2018 Vol.25
  • 2017 Vol.24
  • 2016 vol.23
  • 2015 vol.22
  • 2014 vol.21
  • 2013 vol.20
  • 2012 vol.19
  • 2011 vol.18
  • 2010 vol.17
  • 2009 vol.16
  • No.1
  • No.2

Supervised by Ministry of Industry and Information Technology of The People's Republic of China Sponsored by Harbin Institute of Technology Editor-in-chief Yu Zhou ISSNISSN 1005-9113 CNCN 23-1378/T

期刊网站二维码
微信公众号二维码
Related citation:
【HTML】   【PDF download】   【View/Add Comment】
Back Issue    Advanced Search
This paper has been browsed  85 times  ,  downloaded  89 times  
Shared by: Wechat More
Research on Lightweight of Digital Human Wav2Lip384 Model
Author NameAffiliationPostcode
Haiquan Fang* School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China 310023
Shuya Shi School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China 310023
Hanlin Lan Zhejiang Institute of Administration, Hangzhou 311121, China 311121
Peile Lin School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China 310023
Yangyang Lin School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China 310023
Zhongtian Chen School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China 310023
Yu Lin School of Public Administration, Zhejiang University of Technology, Hangzhou 310023, China 310023
Abstract:
In order to reduce the deployment cost of the artificial intelligence model, the model needs to be lightweight. Due to the large size of the Wav2Lip384 model weight file and the high GPU VRAM consumption, we applied TensorRT to perform FP16 quantization on the Wav2Lip384 model. The test was conducted on a computer equipped with an RTX 4090 GPU 24GB VRAM. The results showed that the size of the quantized model weight file was reduced by 50.16%. In terms of inference performance, compared to the original model, the quantized model preserved satisfactory performance, with LSE-D increasing by 0.63%, LSE-C decreasing by 1.32%, and FID increasing by 5.29%. We further applied the quantized Wav2Lip384 model in LiveTalking to build a real-time streaming digital human. Testing revealed that the VRAM consumption decreased by 80.77%. Using the proposed model, more digital humans could be deployed in cases of limited computational resources, thus effectively reducing costs and improving economic benefits in engineering applications. Additionally, we made improvements to LiveTalking. We integrated ASR, LLM, TTS, and the quantized Wav2Lip384 model into LiveTalking to build a lightweight model for a real-time high-definition voice interactive digital human. This provides a reference solution for engineering applications.
Key words:  digital human  quantized model  Wav2Lip  TensorRT  LiveTalking
DOI:10.11916/j.issn.1005-9113.26008
Clc Number:TP39
Fund: State Key Program of National Social Science Foundation of China (Grant No. 22AKS019), and Humanities and Social Sciences Pre Research Fund Project of Zhejiang University of Technology (Grant No.SKY-ZX-20220207)
Descriptions in Chinese:
  针对Wav2Lip384模型权值文件体积大、GPU显存占用高的问题,我们应用TensorRT对Wav2Lip384模型进行FP16量化。在RTX 4090 GPU 24GB VRAM计算机上进行测试,结果发现,量化模型权值文件的体积减少50.16%。在模型推理效果方面,相对于原模型,量化模型的推理效果维持在较好的水平,LSE-D增加0.63%、LSE-C减少1.32%、FID增加5.29%。进一步,把Wav2Lip384量化模型应用于LiveTalking,构建实时流式数字人,经过测试发现,显存占用降低80.77%。在有限的计算资源下,可以部署更多数字人,在工程应用上可以有效降低成本,提高经济效益。最后,我们对LiveTalking进行了改进。我们把语音识别、聊天大模型、语音合成、Wav2Lip384量化模型集成到LiveTalking,构建实时高清语音交互数字人轻量化模型,这可为工程应用提供一个参考方案。

LINKS