Multimodal model

A multimodal model processes or produces more than one kind of data, such as text, images, audio, video, or sensor signals.

Plan for this page

Understand one concept in plain language and connect it to nearby ideas.

You will leave with: It may translate between modalities, combine them in one representation, or use one to guide another. Supporting several modalities does not mean they are understood equally well or aligned perfectly.
Time: 3–8 min
Before you begin: No prerequisite. Keep one real example in mind.
Do this now: Read the explanation, then compare the example with the common misunderstanding.

Prepared next stepExplore the map of AI modalities

Multimodal modelA multimodal model processes or produces more than one kind of data, such as text, images, audio, video, or sensor signals.

Model
Generative AI
Synthetic media

Plain explanation

It may translate between modalities, combine them in one representation, or use one to guide another. Supporting several modalities does not mean they are understood equally well or aligned perfectly.

Example

A user supplies a chart image and a written question, and the model returns a textual interpretation of visible trends.

Why it matters

Multimodal systems enable accessibility and richer interaction but add failure modes involving perception, timing, provenance, privacy, and cross-modal mismatch.

Common misunderstanding

Misconception: multimodal means the model experiences the world like a person. Correction: it processes several encoded data types through learned computational relationships.

Related concepts

Follow relationships, not only the alphabet

ModelA model is a learned or designed representation that maps inputs to outputs for a defined task or family of tasks.Generative AIGenerative AI produces new text, images, audio, video, code, or other content in response to context and controls.Synthetic mediaSynthetic media is text, imagery, audio, video, or other media created or materially altered by computational generation.

Explore the map of AI modalities

Sources and scope

What supports this object

Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence ProfileU.S. National Institute of Standards and Technology

Addresses risks across generated text, image, audio, video and related modalities; capabilities and evaluations remain model-specific.

Open source

Loading page content...

…