Modern AI Engineering

Lesson 16.1 · 23 min · video lesson

Multimodal AI: Perceiving Text, Images, and Audio Together

A customer sends our support bot a photo of a cracked blender jar and a voice note saying 'it leaks here' — how can one model understand both at once?

In short: Multimodal AI is AI that takes in or produces more than one kind of data, such as text, images, audio and video. It works by turning each kind of data into vectors with its own encoder, mapping those vectors into a shared space, and letting one model reason over all of them together. It is what powers photo-aware chatbots, text-to-image tools, image search by description and video understanding.

What you will learn

  • The Big Picture
  • What is a Modality?
  • Unimodal AI vs Multimodal AI
  • Why Multimodal AI?
  • How Multimodal AI Works
  • Three Common Types of Multimodal AI
  • Real Examples of Multimodal AI
  • Use Cases of Multimodal AI
  • Common Mistakes to Avoid
  • Quick Summary

In this lesson

  1. The big picture
  2. What is a modality?
  3. Unimodal AI vs multimodal AI
  4. Why multimodal AI?
  5. How multimodal AI works
  6. Three common types of multimodal AI
  7. Code: teaching two modalities to share a space
  8. Real examples of multimodal AI
  9. Use cases of multimodal AI
  10. Common mistakes to avoid
  11. Worked example, step by step
  12. Practice: try it yourself
  13. Quick summary

Key terms in this lesson

Modality
A type of data with its own structure, such as text, images, audio or video.
Multimodal model
A model that takes in and/or produces more than one modality.
Encoder
A network that turns raw data of one modality into vectors.
Projection (connector)
A small learned layer that maps one encoder's vectors into another model's embedding space.
Fusion
The step where information from different modalities is combined: early, late or intermediate.
Contrastive learning
Training that pulls matching pairs (an image and its caption) together and pushes mismatched pairs apart.
Visual hallucination
When a model describes image content that is not actually there.

This lesson is part of the Generative AI Engineering and Complete AI Engineer tracks. See plans and pricing to open the full lesson, its video, labs and quiz, or try the free lesson first (free account needed).

← 15.3 LLM Watermarking: Embedding Invisible Signatures in AI Text · 16.2 Vision Transformers: Applying Self-Attention to Image Patches →