11.3 Specialized Models (Vision, Audio)
While Large Language Models (LLMs) excel at text-based tasks, the frontier of AI is increasingly multimodal. Specialized foundation models are designed to understand and process data beyond text, including images, audio, and video. These models are crucial for building applications that can see, hear, and interact with the world in more human-like ways.
Interactive Modality Explorer
The visualization below shows several prominent specialized models, grouped by their primary modality. Click on a model to learn more about its capabilities and use cases.
{{ ctrl.selectedModel.name }}
Modality: {{ ctrl.selectedModel.modality }}
{{ ctrl.selectedModel.description }}
Key Specialized Models
Vision Models
- CLIP (Contrastive Language–Image Pre-training): Connects images and text, enabling powerful zero-shot image classification and image search based on natural language descriptions.
- SAM (Segment Anything Model): An advanced image segmentation model from Meta AI that can identify and "cut out" any object from any image with high precision.
Audio Models
- Whisper: An open-source speech-to-text model from OpenAI that achieves state-of-the-art performance in transcription across multiple languages.
- Wav2Vec 2.0: A framework for self-supervised learning of speech representations from raw audio, foundational for many speech processing tasks.
Image Generation
- Stable Diffusion: A powerful text-to-image latent diffusion model that can generate high-quality, detailed images from textual prompts. Its open nature has led to a vibrant ecosystem of tools and applications.