基于单阶段框架的语音到图像生成方法
CSTR:
作者:
作者单位:

(空军工程大学 空管领航学院,西安 710051)

作者简介:

张振兴(1993—),男,博士,讲师;杨任农(1969—),男,教授,博士生导师

通讯作者:

张滢,2207621676@qq.com

中图分类号:

V247

基金项目:

国家自然科学基金(4,4);陕西省自然科学基金(2021JQ370)


Speech-to-image generation method based on a single-stage framework
Author:
Affiliation:

(School of Air Traffic Control and Navigation, Air Force Engineering University, Xian 710051, China)

Fund Project:

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    为解决多阶段语音到图像生成过程中生成器缺乏对图像进行集成和细化的过程,框架训练多个对抗器网络而增加复杂性、训练成本等问题,提出一种单阶段语音到图像生成对抗网络(single-stage speech-to-image generative adversarial network,Single-S2IGAN)框架。该框架在条件生成对抗网络基础上,引入视觉-语音融合模块(visual speech fusion module,VSFM),VSFM由批归一化(batch normalization,BN)、像素注意力模块(pixel-attention module,MPA)、语音调制模块(speech modulation module,MSM)和加权融合模块(weighted fusion module,MWF)组成。其中,BN用于稳定网络训练;MPA对空间像素区域间的语义关联进行建模,并为关键位置分配更高的权重;MSM利用语音向量中的细粒度语言线索调制视觉特征映射;MWF以自适应的方式捕捉MPA和MSM在通道级别的语义重要性。在CUB bird、Oxford-102、Flickr8k和Places-subset 4个基准数据集上进行一系列实验,结果表明:Single-S2IGAN能够生成高分辨率且与输入语音描述语义一致的图像;与现有其他单阶段框架和多阶段框架方法相比,Single-S2IGAN表现出更好的性能,且其性能水平接近文本到图像生成的方法。本文所提Single-S2IGAN框架为语音到图像生成任务研究提供了新的思路。

    Abstract:

    To address the challenges in multi-stage speech-to-image generation process, such as the generators lack of integration and refinement of images, as well as the increased complexity and training costs associated with training multiple adversarial networks, we propose a single-stage speech-to-image generative adversarial network framework (Single-S2IGAN). This framework builds upon conditional generative adversarial networks by introducing a visual speech fusion module (VSFM), which consists of batch normalization (BN), a pixel-attention module (MPA), a speech modulation module (MSM), and a weighted fusion module (MWF). BN is used to stabilize the training of the network; MPA models the semantic relationships between spatial pixel regions and assigns higher weights to key positions. MSM modulates visual feature maps using fine-grained linguistic cues from the speech vector; and MWF adaptively captures the semantic importance of MPA and MSM at the channel level. A series of experiments were conducted on four benchmark datasets: CUB bird, Oxford-102, Flickr8k, and Places-subset. The results indicate that Single-S2IGAN can generate high-resolution images that are semantically consistent with the input speech descriptions. Compared to both single-stage and multi-stage frameworks, Single-S2IGAN demonstrates superior performance, with levels of effectiveness approaching those of text-to-image generation methods. The proposed Single-S2IGAN framework offers a new perspective for research into speech-to-image generation tasks.

    参考文献
    相似文献
    引证文献
引用本文

张振兴,杨任农,张滢,李永林,赵雨,左家亮.基于单阶段框架的语音到图像生成方法[J].哈尔滨工业大学学报,2026,58(9):115. DOI:10.11918/202403057

复制
分享
相关视频

文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2024-03-25
  • 最后修改日期:
  • 录用日期:
  • 在线发布日期: 2026-10-11
  • 出版日期:
文章二维码