Lesson 14

Why Multimodal Prompting Changes What You Can Ask For

Most people treat AI chatbots as text-only tools. They type a question, get an answer, and stop there. But the most powerful models today — GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro — are multimodal. They can process images, audio, and text together in a single conversation. When you learn to prompt with images and text, you unlock capabilities that text alone cannot reach: describing what is visually happening in a photo, extracting data from a screenshot, generating alt text for accessibility, or turning a product shot into a marketing post.

The real problem this solves is context loss. If you ask a text-only model to "describe this product," it has no idea what the product looks like. It will invent plausible details — wrong colors, wrong layout, wrong text on the screen. A multimodal prompt eliminates that guesswork. You give the model the actual pixels, and it reasons from what it sees, not from what it assumes.

This chapter is the foundation for the rest of this lesson series. You will learn what multimodal models are, how to structure a prompt that combines an image with text instructions, and how to verify that the model actually understood the image before you trust its output.

What a Multimodal Model Actually Does

A multimodal model is trained on data that pairs images with text descriptions. During training, it learns to map visual features — edges, shapes, colors, object relationships — to language tokens. When you upload an image and write a prompt, the model does not "see" the image the way a human does. It converts the image into a sequence of visual tokens, then processes those tokens together with your text tokens in the same neural network.

This means the model can answer questions that require joint reasoning across both modalities. For example:

• Describe: "What is happening in this photo? Be specific about the setting, the people, and the lighting."
• Analyze: "This is a screenshot of a dashboard. List every metric you can read, and identify any anomalies in the trend lines."
• Generate: "Write alt text for this image that is under 120 characters and describes the key visual elements for a screen reader user."
• Transform: "Turn this product photo into a social media caption that highlights the texture and the color of the fabric."

The key distinction from a text-only model: the image is not a decorative attachment. It is input data that the model reasons over. Your text prompt tells the model what to do with that data. The quality of your result depends on how precisely you specify the task, the output format, and the constraints.

Concrete Worked Example: From Product Photo to Social Media Post

Let us walk through a realistic scenario. You work for a small ceramics studio. You have a photo of a handmade mug — a matte sage-green glaze with a visible speckled texture, sitting on a wooden table next to a window with soft morning light. You need a social media caption for Instagram, plus a set of hashtags.

Here is the weak prompt most people would write:

Describe this image and write a caption for Instagram.

That prompt gives the model almost no guidance. It will produce something generic like "Beautiful handmade mug" — technically correct, but useless for marketing. The model does not know your brand voice, your audience, or the specific visual details you want highlighted.

Here is the strong prompt:

You are a social media copywriter for a small ceramics studio.
Look at the attached image of a handmade mug.

Write an Instagram caption that:
- Is 3 to 5 sentences long.
- Highlights the matte sage-green glaze and the speckled texture.
- Mentions that it is wheel-thrown and food-safe.
- Uses a warm, conversational tone, as if a friend is showing off their work.
- Ends with a call to action asking followers to comment their favorite mug color.

Then list 8 relevant hashtags, mixing broad terms (e.g., #ceramics) with niche ones (e.g., #wheelthrownceramics).

Notice what changed. You gave the model a role (social media copywriter), specific visual details to reference (sage-green, speckled), constraints (sentence count, tone, call to action), and a structured output (caption, then hashtags). The model can now check the image against your description and confirm the details are accurate before writing.

Step-by-Step: How to Run This Yourself

You can follow along with any current multimodal model. For this example, we will use ChatGPT (GPT-4o) or Claude, but the steps are identical in Gemini or Microsoft Copilot.

1. Open a new conversation. Do not reuse an old chat with unrelated context — that can bias the model.
2. Upload the image. In ChatGPT, click the paperclip icon (or plus sign) in the message box, then select the image file from your computer. In Claude, click the plus icon in the bottom-left corner of the input box. In Gemini, click the image icon in the prompt bar.
3. Wait for the image to attach. You should see a thumbnail preview appear in the input area. If you do not see it, the image did not upload — retry before proceeding.
4. Paste the strong prompt above into the text field, directly below or after the image attachment.
5. Send the message. The model will process the image and your text together.
6. Verify the output. Read the caption. Does it mention the sage-green glaze? Does it reference the speckled texture? If the model hallucinated a detail that is not in the image — for example, claiming the mug has a handle when it does not — correct it with a follow-up prompt: "The mug has no handle. Rewrite the caption without mentioning a handle."

This verification step is critical. Multimodal models are powerful, but they can still misread an image, especially with low resolution or unusual angles. Always cross-check the model's visual claims against the actual image before publishing anything.

Generating Alt Text: A Practical Use Case

Alt text is a perfect example of a task that is nearly impossible with text-only prompting and trivial with multimodal prompting. You need to describe an image for someone who cannot see it. The description must be concise, factual, and focused on what is meaningful in context.

Here is a prompt template you can reuse:

Write alt text for the attached image.

Requirements:
- Maximum 120 characters.
- Describe only what is visually present.
- Do not interpret or add emotional language.
- Lead with the main subject, then the setting, then the most important detail.

Example format:
"A [subject] in [setting], with [key detail]."

If the image is a chart or graph, add a requirement to read the data:

This is a screenshot of a bar chart. Write alt text that:
- States the chart title.
- Names the axes.
- Gives the value of the tallest bar and the shortest bar.
- Does not exceed 150 characters.

This turns a vague "describe the image" request into a precise data-extraction task. The model will read the numbers from the chart and report them accurately — but only if you tell it to. Without that instruction, it may summarize the chart qualitatively ("sales increased over time") instead of giving the actual figures.

Common Mistakes Beginners Make

• Uploading a low-resolution image. If the image is blurry or smaller than about 512×512 pixels, the model may misread details. Use the original file, not a compressed screenshot.
• Writing the prompt before attaching the image. Some interfaces treat the first message differently. Always attach the image first, then write your instruction, so the model knows the image is part of the current request.
• Assuming the model sees what you see. You know the mug is sage-green. The model only knows what the pixels tell it. If the lighting is warm, the glaze might look yellow-green to the model. State the expected color in your prompt and ask the model to confirm or correct it.
• Not specifying the output format. "Describe this image" produces a paragraph. "List five bullet points describing the image" produces a structured list. Decide what you need before you prompt.
• Trusting the first output. Multimodal models make visual errors. Always verify factual claims — especially numbers, text, and colors — against the image.