Audio Models
Audio multimodal models process speech or audio input (waveforms or spectrograms) alongside text, enabling speech-to-text, speech commands, and audio question answering.
What is Audio Models?
Audio multimodal models process speech or audio input (waveforms or spectrograms) alongside text, enabling speech-to-text, speech commands, and audio question answering.
Audio multimodal models process speech or audio input (waveforms or spectrograms) alongside text, enabling speech-to-text, speech commands, and audio question answering.
Where is it used?
Whisper (OpenAI) does speech-to-text; Qwen2-Audio and AudioPaLM handle audio-in text-out; GPT-4o has native audio I/O; `transformers` supports these via `AutoModelForSpeechSeq2Seq`.
How to build it
Load `WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")`, pass `input_features` from `WhisperProcessor(audio=waveform, return_tensors="pt")`, and call `.generate()`.