Multimodal AI
AI that can take in or produce more than text, such as photos, screenshots, PDFs, voice, audio and video, so you can show it the problem as well as describe it.
Also called: vision · image input · voice mode · talk to AI · image generation
Connections
Understand first
Ideas this one builds on.
Related
Often used together or easily confused.
Resources that teach it
Competency
Find multimodal ai on the commons map ↗
You may know it from
- customer support sending a photo of the problem instead of describing it Support teams learned long ago that a picture removes a round of back-and-forth.
In products
The same idea, under different names in different tools.
| Provider | What they call it |
|---|---|
| OpenAI | ChatGPT voice and image input |
| Gemini Live; NotebookLM Audio Overviews | |
| Anthropic | Claude image and PDF understanding |
In plain words
Multimodal means the AI works across different kinds of material. You can upload a photo of a whiteboard and ask for the action items, share a screenshot of a confusing error, hand over a scanned form, talk through an idea out loud while walking, or ask for an image or an audio summary back.
For many people this is the easiest way in. Instead of carefully writing out what a chart shows, you show the chart. Instead of typing meeting notes, you speak them. The AI still needs to know what you want from the material, so a short instruction alongside the picture or recording still matters.
The same cautions apply as with text. AI can misread a blurry number in a photo or mishear a name, and it will report the mistake just as confidently.
Example
A facilities manager photographs a cluttered storage room and asks for a labelled layout plan with what to keep, move and recycle. She corrects two items it misidentified, then prints the plan for the team.
How it connects
- It extends prompting: the prompt can be a picture, a file or your voice.
- Outputs like diagrams and pages connect to artifacts.
- Misread images are a source of hallucination, so verification still applies.
Try it
Photograph something messy, like your desk or a whiteboard, and write down three things you expect the AI to pull out of the picture. Upload it, ask, and compare its list with yours.
Read this card as plain Markdown: /concepts/con-multimodal.md. Know a parallel from your own field, or a product that implements this? Add it.