Neural Networks Taxonomy - 25. Audio, Speech and Music Networks

25. Audio, Speech & Music Networks

Audio networks turn waveforms or spectrograms into text, or text into audio. The pipeline in Fig 27 encodes the signal, models temporal context, and decodes to the target — reusing convolutional, recurrent, and Transformer machinery specialised for the time-frequency structure of sound.

Origins and rise

WaveNet generated raw audio sample by sample with dilated causal convolutions — the same structure as the TCN.^[87] Tacotron 2 synthesised natural speech by predicting mel-spectrograms and then vocoding them.^[331] For recognition, the Conformer fused convolution with self-attention,^[332] and Whisper reached robust multilingual transcription by training on very large weakly-labelled datasets.^[333]

Speech recognition

End-to-end networks replaced hand-engineered pipelines for converting speech to text. Deep Speech 2 used a deep recurrent or convolutional stack trained with CTC to map spectrograms to characters,^[334] the Jasper / QuartzNet family distilled the recipe into compact streaming convolutional architectures, and the Conformer interleaved convolutions with Transformer self-attention to capture both local and long-range acoustic cues.^[332] Whisper scaled weak supervision on 680 000 hours of web audio to a multilingual model that transcribes and translates without task-specific tuning.^[333]

Speech synthesis

On the generation side, Tacotron introduced attention-based end-to-end text-to-spectrogram synthesis^[331] and FastSpeech replaced the fragile autoregressive attention with a feed-forward Transformer that predicts mel-spectrogram durations in parallel,^[335] enabling fast and controllable synthesis. Neural vocoders such as WaveNet and HiFi-GAN then turn the spectrogram into audible waveform.

Music generation and source separation

Music raises longer-range structure than speech. The Music Transformer adapted relative self-attention to capture motifs and repetitions over thousands of notes,^[336] while MuseGAN took a GAN-based approach to multi-track symbolic generation. For separating instruments from a mixture, Conv-TasNet operates directly on the waveform with a learned encoder-decoder and masking network,^[337] and Demucs extends the idea to high-fidelity music separation with a U-Net-style structure.

Applications

• Speech recognition and text-to-speech synthesis.
• Music generation and source separation.
• Speaker verification and audio understanding.

Strengths and limitations