Can AI watch videos? Video understanding options for agents

Four ways to give an agent video

An agent can't open a video file and look at it the way it reads a page. It gets video in one of four ways:

1. Transcripts. The words spoken, as text. Cheap and easy to search, but blind to anything that isn't said. For other people's YouTube videos, YouTube's own captions API requires permission to edit the video.
2. A model that watches one video. Gemini reads frames at one per second plus the audio and answers questions about that video, with timestamps. It accepts uploaded files and public YouTube links.
3. An index of your own video. You upload your library to a service such as Twelve Labs or Memories.ai, which indexes it privately so you can search it, analyze it and get embeddings.
4. A shared index of public video. Someone has already indexed the video, and you search it by what happens. UpRock's index holds notes on public videos (transcripts, model-written descriptions and timestamps), not copies, and each result links back to the original at the moment.

If you need X, use Y

Checked 25 Oct 2026 against each vendor's documentation and price page.

Where each option runs out

• Gemini works on the videos you pass it. Its YouTube URL feature "is in preview and is available at no charge", and Google says pricing and rate limits are likely to change. It takes public videos only, at most 10 per request on Gemini 2.5 and later, and the free tier allows 8 hours of YouTube video a day. Models with a 1M-token context handle about 3 hours of video per request at the default resolution, and Google notes that sampling one frame per second can miss detail in fast action.
• Twelve Labs starts from an asset, "a file you upload once". Its docs say its models work on one video at a time, even when you search an index of many, while its Jockey agent reasons across a whole knowledge store. Plans are sized by storage: 3 GB free, 50 GB for $29 a month, 200 GB for $99.
• Memories.ai prices indexing per minute and lists imports from YouTube, TikTok and Instagram URLs on its price list. Its docs say stored data is deleted 30 days after your balance reaches zero, and its pricing page and docs list different search prices ($0.10 and $8 per 1,000).
• UpRock finds only what has been indexed. In our 28 searches on 25 Oct 2026, travel and food queries matched best and "changing a bike tire" found nothing on topic. video_search returns moments with a transcript excerpt, not an answer. Scores rank results rather than prove them, so the agent should treat each hit as a lead and cite it with its link.

A tested agent function: search public video, cite the second

find_video_evidence is shaped like an agent tool: it takes the question, searches UpRock's public index, drops weak matches and returns citations with the platform, the time range and a link that opens at the moment. A failed search comes back as an error string rather than an exception, so the model can read it and carry on.

# pip install "mcp>=2.3"   env: UPROCK_API_KEY# python video_evidence.py "a waterfall in a jungle"import asyncioimport jsonimport osimport sysimport timefrom urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit​import httpx2from mcp import Clientfrom mcp.client.streamable_http import streamable_http_client​# The key goes in an Authorization header, set on the HTTP client the MCP client uses.UPROCK = "https://mcp.uprock.ai/mcp"HEADERS = {"Authorization": f"Bearer {os.environ['UPROCK_API_KEY']}"}YOUTUBE = {"youtube.com", "www.youtube.com", "m.youtube.com", "youtu.be"}PLATFORMS = {"youtube": "YouTube", "tiktok": "TikTok", "instagram": "Instagram"}​​def at_second(url: str, seconds: float) -> str:    """Open YouTube at the moment; TikTok and Instagram links open the post."""    parts = urlsplit(url)    if parts.hostname not in YOUTUBE:        return url    params = dict(parse_qsl(parts.query)) | {"t": f"{int(seconds)}s"}    return urlunsplit(parts._replace(query=urlencode(params)))​​def clock(seconds: float) -> str:    h, rest = divmod(int(seconds), 3600)    m, s = divmod(rest, 60)    return f"{h}:{m:02}:{s:02}" if h else f"{m}:{s:02}"​​async def find_video_evidence(    question: str, min_score: float = 0.55, limit: int = 5) -> dict:    """Agent tool: public video moments that match `question`, as citations."""    async with (        httpx2.AsyncClient(headers=HEADERS, timeout=httpx2.Timeout(30, read=300)) as http,        Client(streamable_http_client(UPROCK, http_client=http)) as client,    ):        res = await client.call_tool("video_search", {"query": question})    text = next((c.text for c in res.content if c.type == "text"), "")    if res.is_error:        return {"question": question, "citations": [], "error": text}    citations = []    for m in json.loads(text).get("results", []):        if m["score"] < min_score or len(citations) == limit:            break  # results come best first        said = " ".join((m.get("transcript") or "").split())        citations.append(            {                "id": len(citations) + 1,                "title": m["title"],                "platform": PLATFORMS.get(m["source"], m["source"]),                "start": clock(m["t0"]),                "end": clock(m["t1"]),                "url": at_second(m["url"], m["t0"]),                "said": said[:200] or None,  # transcript excerpt; None when silent                "score": round(m["score"], 3),                "published": time.strftime("%Y-%m-%d", time.gmtime(m["published_at"])),            }        )    return {        "question": question,        "citations": citations,        "note": "Matched by meaning on what is said and shown. Open the url to check, "        "and cite as [id] with the platform and start time.",    }​​def as_sources(evidence: dict) -> str:    """The citations as lines a model can paste under its answer."""    if not evidence["citations"]:        return evidence.get("error") or "No moments above the score threshold."    return "\n".join(        f"[{c['id']}] {c['title']} ({c['platform']}, {c['start']} to {c['end']}) "        f"{c['url']}"        for c in evidence["citations"]    )​​if __name__ == "__main__":    question = sys.argv[1] if len(sys.argv) > 1 else "a waterfall in a jungle"    evidence = asyncio.run(find_video_evidence(question))    print(json.dumps(evidence["citations"][:1], indent=2, ensure_ascii=False))    print(as_sources(evidence))

Run on "a waterfall in a jungle" on 25 Oct 2026, it returned five citations. The first one's said field quotes the speaker: "There's a waterfall here". The source lines it printed:

[1] Surviving 24 Hours In The Jungle! (YouTube, 2:55 to 3:25) https://www.youtube.com/watch?v=5An-7Q7s7zg&t=175s[2] Video by australia (Instagram, 0:00 to 0:20) https://www.instagram.com/reel/DT_B6qIEvbC/[3] Video by natgeotravel (Instagram, 0:00 to 0:30) https://www.instagram.com/reel/DRudyuliFJX/[4] Video by emirates (Instagram, 0:00 to 0:11) https://www.instagram.com/reel/DBbItfrS4hA/[5] Video by tourismthailand (Instagram, 0:00 to 0:11) https://www.instagram.com/reel/DOfcu64gcoL/

The 0.55 cutoff is a starting point from our runs, not a calibrated threshold: on-topic top results scored about 0.6 to 0.75, but "changing a bike tire" returned an off-topic moment at 0.625. Keep the link in every citation so a person, or the agent with another tool, can check the moment.

Giving it to your agent

If your agent already connects to UpRock's MCP server (https://mcp.uprock.ai/mcp with an Authorization: Bearer header, or https://mcp.uprock.ai/<API_KEY>/mcp), it has video_search without any of this code; the function above is for when you want the filtering and citation format in your own tool layer. Register find_video_evidence as a tool in your framework and return its dict as the tool result. The integration guides for Claude Code, Cursor, LangChain and the Vercel AI SDK show the MCP setup for each.

For the other options, start from the vendors' own documentation: Gemini video understanding, Twelve Labs and Memories.ai. We haven't written code for them here because we didn't run it.

Tested 25 Oct 2026 against the live UpRock MCP server (Python 3.11, mcp 2.3.0): we extracted the function from this page and ran it on "a waterfall in a jungle" (five citations), on "changing a bike tire" (three citations above the cutoff, all off topic, which is why the link matters) and with an empty query, which came back as {"error": "missing required parameter: query"}. We didn't connect it to a model, so registering it as a tool is untested. On 6 Nov 2026 we ran it again with an UpRock API key in an Authorization header.