Can AI watch a video? We tested Claude, ChatGPT, Gemini and Codex (November 2026)

Checked and updated 10 November 2026

Only some AI can watch a video: Gemini takes a video file with its sound and understood every change asked for in our test recordings, while Claude, ChatGPT's GPT models and Codex can't take video at all.

That matters most for coding agents. Claude Code, Codex, Cursor and Copilot all run on models that can't play a video, so when you hand one a screen recording it sees, at best, still frames with no sound. In the recordings people actually send (a bug, a design review, "change this button"), almost everything that matters is said out loud.

We measured it. Here is what each AI did with the same two screen recordings, and what to do instead.

What we tested

Two narrated screen recordings of this website, each asking for five changes, so we knew the right answers:

• Plain (47 seconds): every change said in full. "This Upgrade to Pro button should say Start Pro instead. That one is urgent."
• Pointing (25 seconds): changes said the way people talk while pointing. "Change this to say: show your AI what to fix." "We can drop this one completely." The words only make sense with the screen at that moment.

Every AI got the same request: list every change I asked for, what to change, where, and the new value. A separate model marked each of the five changes as hit (right place and right change), partial or missed.

The recordings are simple: still pages, a highlighted pointer and a synthetic voice. Real recordings scroll, ramble and mumble, which makes every method below harder, not easier.

The results

Given only frames, Claude saw where the pointer stopped every time (it got all five places right, just none of the changes) and said so honestly: "The frames have no sound, so I can see where you pointed but not what you said." GPT and Qwen declined to guess at all. That is the right behaviour, and it still leaves you with nothing to act on.

Frames do get expensive. At one frame every two seconds, a 47-second recording was already about 32,000 tokens of images. A ten-minute walkthrough at the same rate is around 400,000, more than many models accept at once.

What Claude Code actually did with a video file

We gave Claude Code a project and a 47-second recording, feedback.mp4, and said: "I recorded a quick video of the changes I want on the site. Please make them."

It did what a careful engineer would. It checked the file with ffprobe, pulled frames out with ffmpeg and looked at them, noticed the requests were in the voice-over, then looked for a speech-to-text tool on the computer (Whisper, faster-whisper, Vosk). There wasn't one. It stopped and wrote back:

I can't make these changes yet, because the requests are in your voice-over and I can't hear it.

It offered three ways forward: paste a transcript, give it an API key for a transcription service, or install a local Whisper model. Each of those is work for you before any work gets done.

We then ran the same request again with the Tellscreen skill installed. This time Claude Code sent feedback.mp4 to Tellscreen with one command, got the change list back, and found the file and line for all five changes in 76 seconds. (We ran it in read-only mode for the test, so it listed the edits instead of making them.)

So does turning a video into text help?

It depends on which AI you're handing it to.

• Gemini, with the file: the raw video works. If you live in Gemini, upload the file and ask.
• Claude, ChatGPT, Codex, Cursor, Copilot: the video has to become text and pictures first. Frames alone don't work. Frames plus a transcript do, if your agent can make both, and it costs 25 to 50 times more than it needs to.
• Any of them: a list of the changes, each with the words said, where on screen and a screenshot of that moment, is the cheapest input and the easiest to act on. It also leaves a record: the changes are numbered, so you can tick them off.

That list is what Tellscreen writes.

Give your agent a video it can use

If you already have the recording (a QuickTime or Loom download, a phone video), one command turns it into the change list. Your agent can run it too:

curl -fsS -T recording.mov "https://tellscreen.com/api/v1/convert?wait=110"

It prints Markdown: a summary, every change with the words said, where on screen, how to check it's done and a screenshot link of that moment, then the transcript with times. The video is private; the answer includes a link to it. How to send a video to Claude Code, Codex or Cursor has the agent-side setup, and agents can read tellscreen.com/llms.txt.

If you haven't recorded yet, record with Tellscreen: in the browser, from the Mac and Windows app (press ⌥⇧R anywhere and draw on the screen while you talk), or from the Record screen button the Claude Code plugin adds. The change list lands in your prompt. See Tellscreen for Claude Code and Codex.

Loom link instead of a file? How to give a Loom video to Claude or ChatGPT.

How the test was run

The recordings were made from this site's own pages with a synthetic voice, then sent to each model through OpenRouter on 10 November 2026: the MP4 itself to models that accept video, and JPEG frames every 2 seconds at 1280 pixels wide to models that don't. The transcript in the frames-plus-transcript runs came from Whisper large-v3. Tellscreen's list came from its normal pipeline. Each answer was graded by Claude Opus 5.5 against the five known changes; we read every answer ourselves as well. The Qwen runs were repeated with more room to answer, because its first answers were spent thinking; the second time it said plainly that it had received "silent frames only". We kept every raw answer; ask at hello@tellscreen.com if you'd like them.

Sources

• Which models take video, images or audio: OpenRouter model list, read 10 November 2026
• How video is sent to models that accept it: OpenRouter, video inputs
• Claude's supported inputs (text, images and PDFs): Anthropic, vision