What Is Multimodal AI? (And Why It Rarely Means One Model)

"Multimodal" is used for two different things, and confusing them leads to real disappointment.

One meaning is a model that can read more than one kind of input — text, pictures, audio, video. The other is a system that can produce more than one kind of output. Almost no single model does both, and the gap between those two facts explains why building a "multimodal app" usually means wiring several models together.

Figures last checked: August 26, 2026.

What a modality is

A modality is a type of data: text, image, audio, video. A model is multimodal when it handles more than one.

The useful question is never "is it multimodal" but "which modalities, in which direction" — because reading and writing are entirely different capabilities.

What today's large models actually accept

Read across, not down. Every one of these outputs text only.

Two things jump out.

Gemini reads audio and video; Claude and GPT do not. That is the single clearest capability difference between the big families right now, and it is not a quality judgement — it is a list of accepted input types. If your input is a recorded meeting or a video clip, that difference decides your model before anything else does.

Every family in that table writes text and nothing else. Gemini can watch a video and describe it. It cannot make one.

So who makes the pictures and videos?

Different models entirely. In Atlas Cloud's catalogue alone, the split is:

Nearly all of those image and video models do one job: text in, one kind of media out. They are not conversational, they do not read documents, and they are not the models in the table above.

This is the practical shape of multimodal AI in 2026: a text model that can read several input types, plus specialist generators for each output type, joined by your code. Not one model that does everything.

You can see the split on the models index — chat, image and video are separate lists because they are separate model families.

What multimodal input actually costs

Images and documents are not free to read. They are converted into tokens like everything else, and Anthropic publishes the counts:

• One image plus a short instruction: 1,028 input tokens
• One PDF plus a short instruction: 2,188 input tokens

So a request with twenty screenshots costs about 20,000 input tokens before your prompt. On Claude Sonnet 5 at $2.00 per million that is four cents a request — trivial once, real at fifty thousand requests a month.

Video is heavier still, which is part of why fewer providers accept it.

Where multimodal genuinely earns its place

• Reading documents that are pictures. Scans, screenshots, receipts, handwritten forms. A text model with vision replaces an OCR pipeline and handles layout it was never trained on.
• Audio and video understanding. Meeting summaries, call analysis, describing clips. Today this points at Gemini, because it is the family that accepts those inputs.
• Grounded generation. Show a model your brand photos and ask for copy that matches them.
• Interface work. Give a model a screenshot of a broken page and ask what is wrong.

Where it does not

• When you need media out. Reach for a specialist generator. That is what the 196 video and 121 image models are for.
• When plain text would do. A vision model reading a PDF you could have parsed is a thousand tokens you did not need to spend.
• When "multimodal" is doing marketing work. A model that reads images is not automatically better at text. Check the modality list, not the adjective.

How to choose

1. Write down the direction. What goes in, what comes out. Two lists.
2. Pick the reader by input type. If audio or video is in the input list, the Gemini family is currently the answer among the big three.
3. Pick each generator separately. Image, video and audio models are chosen on their own merits and their own prices.
4. Budget the inputs. A thousand tokens per image, two thousand per PDF, multiplied by your real request volume.

The short version

• Multimodal describes what a model reads, what it writes, or both — always ask which.
• Gemini reads audio and video. Claude and GPT read text, images and files. All of them write text only.
• Media is produced by separate specialist models: 196 video, 121 image and 20 audio models at one provider alone.
• A real multimodal application is usually one reader plus one or more generators, joined by your code.
• Images and PDFs cost about 1,000 and 2,000 input tokens respectively. Multiply by your volume before you design around them.

Explore more on ElliSekiz

See the families side by side. The models index splits chat, image, video and audio, each with current prices.

Compare the readers on price. Token rates for every text model are on the compare page, in the unit each provider bills in.

Sources

• OpenRouter models API, https://openrouter.ai/api/v1/models (input and output modalities, read 2026-08-26)
• Atlas Cloud model catalogue API, https://api.atlascloud.ai/api/v1/models (model type counts)
• Anthropic — Token counting (image and PDF token counts)