| 引用本文: | 张振兴,杨任农,张滢,李永林,赵雨,左家亮.基于单阶段框架的语音到图像生成方法[J].哈尔滨工业大学学报,2026,58(9):115.DOI:10.11918/202403057 |
| Zhang Zhenxing,Yang Rennong,Zhang Ying,Li Yonglin,Zhao Yu,Zuo Jialiang.Speech-to-image generation method based on a single-stage framework[J].Journal of Harbin Institute of Technology,2026,58(9):115.DOI:10.11918/202403057 |
|
| 摘要: |
| 为解决多阶段语音到图像生成过程中生成器缺乏对图像进行集成和细化的过程,框架训练多个对抗器网络而增加复杂性、训练成本等问题,提出一种单阶段语音到图像生成对抗网络(single-stage speech-to-image generative adversarial network,Single-S2IGAN)框架。该框架在条件生成对抗网络基础上,引入视觉-语音融合模块(visual speech fusion module,VSFM),VSFM由批归一化(batch normalization,BN)、像素注意力模块(pixel-attention module,MPA)、语音调制模块(speech modulation module,MSM)和加权融合模块(weighted fusion module,MWF)组成。其中,BN用于稳定网络训练;MPA对空间像素区域间的语义关联进行建模,并为关键位置分配更高的权重;MSM利用语音向量中的细粒度语言线索调制视觉特征映射;MWF以自适应的方式捕捉MPA和MSM在通道级别的语义重要性。在CUB bird、Oxford-102、Flickr8k和Places-subset 4个基准数据集上进行一系列实验,结果表明:Single-S2IGAN能够生成高分辨率且与输入语音描述语义一致的图像;与现有其他单阶段框架和多阶段框架方法相比,Single-S2IGAN表现出更好的性能,且其性能水平接近文本到图像生成的方法。本文所提Single-S2IGAN框架为语音到图像生成任务研究提供了新的思路。 |
| 关键词: 语音到图像生成 单阶段框架 生成对抗网络 注意力机制 融合模型 |
| DOI:10.11918/202403057 |
| 分类号:V247 |
| 文献标识码:A |
| 基金项目:国家自然科学基金(4,4);陕西省自然科学基金(2021JQ370) |
|
| Speech-to-image generation method based on a single-stage framework |
|
Zhang Zhenxing,Yang Rennong,Zhang Ying,Li Yonglin,Zhao Yu,Zuo Jialiang
|
|
(School of Air Traffic Control and Navigation, Air Force Engineering University, Xian 710051, China)
|
| Abstract: |
| To address the challenges in multi-stage speech-to-image generation process, such as the generators lack of integration and refinement of images, as well as the increased complexity and training costs associated with training multiple adversarial networks, we propose a single-stage speech-to-image generative adversarial network framework (Single-S2IGAN). This framework builds upon conditional generative adversarial networks by introducing a visual speech fusion module (VSFM), which consists of batch normalization (BN), a pixel-attention module (MPA), a speech modulation module (MSM), and a weighted fusion module (MWF). BN is used to stabilize the training of the network; MPA models the semantic relationships between spatial pixel regions and assigns higher weights to key positions. MSM modulates visual feature maps using fine-grained linguistic cues from the speech vector; and MWF adaptively captures the semantic importance of MPA and MSM at the channel level. A series of experiments were conducted on four benchmark datasets: CUB bird, Oxford-102, Flickr8k, and Places-subset. The results indicate that Single-S2IGAN can generate high-resolution images that are semantically consistent with the input speech descriptions. Compared to both single-stage and multi-stage frameworks, Single-S2IGAN demonstrates superior performance, with levels of effectiveness approaching those of text-to-image generation methods. The proposed Single-S2IGAN framework offers a new perspective for research into speech-to-image generation tasks. |
| Key words: speech-to-image generation single-stage framework generative adversarial network attention mechanism fusion model |