Abstract:To address the challenges in multi-stage speech-to-image generation process, such as the generators lack of integration and refinement of images, as well as the increased complexity and training costs associated with training multiple adversarial networks, we propose a single-stage speech-to-image generative adversarial network framework (Single-S2IGAN). This framework builds upon conditional generative adversarial networks by introducing a visual speech fusion module (VSFM), which consists of batch normalization (BN), a pixel-attention module (MPA), a speech modulation module (MSM), and a weighted fusion module (MWF). BN is used to stabilize the training of the network; MPA models the semantic relationships between spatial pixel regions and assigns higher weights to key positions. MSM modulates visual feature maps using fine-grained linguistic cues from the speech vector; and MWF adaptively captures the semantic importance of MPA and MSM at the channel level. A series of experiments were conducted on four benchmark datasets: CUB bird, Oxford-102, Flickr8k, and Places-subset. The results indicate that Single-S2IGAN can generate high-resolution images that are semantically consistent with the input speech descriptions. Compared to both single-stage and multi-stage frameworks, Single-S2IGAN demonstrates superior performance, with levels of effectiveness approaching those of text-to-image generation methods. The proposed Single-S2IGAN framework offers a new perspective for research into speech-to-image generation tasks.