VIDEOContext

Give AI agents context they can reason over from video.

Timestamped, multimodal, evidence-grounded — not just a transcript. Runs locally.

Website · Install · Use with AI agents · CLI · MCP · Architecture · .vctx Format

Install

pipx install "videocontent[agent] @ git+https://github.com/AAGAM17/VIDEOContext"   # the CLI
videocontent init-agent                                                            # teach your agents

You also need FFmpeg (brew install ffmpeg / apt install ffmpeg); Tesseract adds on-screen text (brew install tesseract / apt install tesseract-ocr). No API key, no cloud account, no Docker. uv tool install … or pip install … work the same way as pipx.

The videocontent release on PyPI (0.1.0) predates the agent integration; until the next release, install from GitHub as shown.

Use VIDEOContext with AI coding agents

Open Claude Code, Codex or any Agent Skills–compatible agent in a folder with a video and ask:

Analyze demo.mp4 and tell me what happened after the error.

or invoke it explicitly: /videocontent analyze demo.mp4 (Claude Code) · $videocontent analyze demo.mp4 (Codex).

The agent inspects the video, analyzes it once (locally), asks a temporal question and answers with evidence. Abridged from a real Claude Code run on this repository's demo video:

### Answer
At 00:00:48.600 a terminal runs `pytest -q tests/` and one test fails with
`ConnectionError: refused on port 5432`. After that, the terminal closes at ~00:00:57, the
presenter says "That concludes the review", and the video returns to the title slide.
Nothing in the recording shows the error being fixed.

### Evidence
00:00:48.600 — [observed · on-screen text] "E ConnectionError: refused on port 5432"
00:00:57.080 — [observed · speech] "That concludes the review, thank you."
00:00:57.375 — [detected · event] scene boundary

### Source
demo.mp4 (demo.vctx)

### Trace
inspect → not_analyzed → analyze → ask "what happened after the ConnectionError" → 5 spans

What the agent can do with it: find when something happened, ask what happened before or after a moment, list every occurrence of an error or concept, compare two recordings, and build context packages (evidence, UI states, events, frame images with timestamps) for rebuilding a UI, writing Playwright tests, debugging, or writing docs. See examples/agents.

Agent	Shortest path	Details
Claude Code	`videocontent init-agent` — or `/plugin marketplace add AAGAM17/VIDEOContext` then `/plugin install videocontent@videocontent`	docs/agents/claude-code.md
Codex	`videocontent init-agent --agent codex`	docs/agents/codex.md
Other agents / any MCP client	`videocontent init-agent --agent agents` · MCP server: `videocontent mcp`	docs/agents/generic-agents.md

Safety by default: extracted video text is treated as untrusted data, never instructions; credential-shaped strings are redacted; nothing is processed without being asked and nothing leaves the machine unless you configure a remote provider.

CLI in 30 seconds

videocontent analyze demo.mp4                                   # process once, summarize
videocontent ask demo.mp4 "What happened after the error?"      # evidence-backed answer
videocontent search demo.mp4 "ConnectionError"                  # timestamped matches
videocontent timeline demo.mp4 --from 0:40 --to 1:00            # what happened in a range
videocontent context demo.mp4 "Recreate this UI in React"      # budgeted package for a task
videocontent compare before.mp4 after.mp4                       # what changed

Every command accepts the video or its .vctx, and has --json (stable script output) and --agent (bounded, provenance-labelled, redacted JSON for AI agents).

Why VideoContext?

Modern AI systems can understand images and, increasingly, video. What is still needed is an infrastructure layer that turns video into structured, reusable context.

Video contains multiple kinds of information at the same time:

What was said
What appeared on screen
What objects were visible
What events occurred
When each piece of information occurred

A transcript alone loses visual information.

OCR alone loses speech.

Individual frame descriptions can lose the temporal relationship between information.

VideoContext processes video into timestamped context that can be searched, queried, exposed through APIs, and made available to AI agents.

flowchart LR

    RAW["🎬 Raw Video"]

    RAW --> PROCESS["Process Once"]

    PROCESS --> VCTX["📦 .vctx<br/>Reusable Context"]

    VCTX --> SEARCH["Search"]
    VCTX --> ASK["Q&A"]
    VCTX --> API["REST API"]
    VCTX --> MCP["MCP"]
    VCTX --> AGENTS["AI Agents"]
    VCTX --> APPS["Applications"]

The goal is simple:

Make video information as searchable and reusable as text while preserving when and where that information occurred.

Features

🎬 Video Processing

VideoContext includes infrastructure for processing video and extracting temporal information.

The project contains components for:

Video ingestion
FFmpeg media inspection
Metadata extraction
Frame sampling
Scene detection
Speech processing
OCR
Vision processing
Object information
Event extraction
Temporal segmentation

📦 Temporal Context

Processed information is represented using the .vctx format.

The core model is:

Information
+
Timestamp
+
Modality
+
Evidence Reference

This means extracted information remains connected to the part of the video it originated from.

flowchart TB

    VIDEO["🎬 Video"]

    VIDEO --> PROCESS["Processing"]

    PROCESS --> DOC["📦 .vctx"]

    DOC --> TRANSCRIPT["Transcript"]
    DOC --> OCR["OCR"]
    DOC --> VISION["Vision"]
    DOC --> OBJECTS["Objects"]
    DOC --> EVENTS["Events"]
    DOC --> SCENES["Scenes"]
    DOC --> SEGMENTS["Segments"]

    TRANSCRIPT --> CONTEXT["Reusable Temporal Context"]
    OCR --> CONTEXT
    VISION --> CONTEXT
    OBJECTS --> CONTEXT
    EVENTS --> CONTEXT
    SCENES --> CONTEXT
    SEGMENTS --> CONTEXT

🔎 Search and Retrieval

VideoContext provides retrieval over processed video context.

The project includes support for:

Lexical retrieval
Modality filtering
Time range filtering
Timestamp lookup
Evidence spans
Search results with traceable source information

flowchart TB

    Q["User Query"]

    Q --> RET["Retrieval"]

    RET --> LEX["Lexical Matching"]
    RET --> TEMP["Temporal Filtering"]
    RET --> MOD["Modality Filtering"]

    LEX --> MERGE["Evidence Selection"]
    TEMP --> MERGE
    MOD --> MERGE

    MERGE --> RESULT["Timestamped Evidence"]

AI Integration

VideoContext includes question answering over retrieved video evidence.

flowchart LR

    Q["Question"]

    Q --> SEARCH["Retrieve Relevant Evidence"]

    SEARCH --> EVIDENCE["Timestamped Evidence"]

    EVIDENCE --> LLM["LLM"]

    LLM --> ANSWER["Answer"]

    EVIDENCE --> OUTPUT["Answer + Evidence"]
    ANSWER --> OUTPUT

The ask() workflow is:

Question
   ↓
Search
   ↓
Select Evidence
   ↓
Build Context
   ↓
LLM
   ↓
Answer + Evidence

The implementation instructs the LLM to answer only from the retrieved evidence and not invent timestamps or facts.

Installation

Core package

pip install videocontent

The core package is intentionally lightweight.

Speech-to-text

pip install "videocontent[asr]"

OCR

pip install "videocontent[ocr]"

Vision adapters

pip install "videocontent[vision]"

Vector retrieval

pip install "videocontent[vectors]"

Embeddings

pip install "videocontent[embeddings]"

REST API

pip install "videocontent[api]"

MCP server

pip install "videocontent[mcp]"

Main optional processing stack

pip install "videocontent[all]"

For development:

pip install -e ".[dev]"

Requirements

VideoContext requires:

Python 3.10 or newer
FFmpeg available on your PATH

For OCR, Tesseract must also be installed.

For example:

brew install ffmpeg
brew install tesseract

On Debian or Ubuntu:

sudo apt install ffmpeg
sudo apt install tesseract-ocr

Check your environment:

videocontent doctor

The doctor command checks the available runtime capabilities, including:

FFmpeg
Tesseract
faster-whisper
FAISS
sentence-transformers
Registered providers

Quick Start

Process a video

from videocontent import Video

video = Video("demo.mp4")

video.process()

video.save()

By default, save() writes:

demo.mp4
   ↓
demo.vctx

You can also process and save in one workflow:

import videocontent

video = videocontent.process(
    "demo.mp4",
    output=True,
)

Process from any supported source

Video / videocontent.open accept a local path, a file:// URL, a direct http(s) media URL, or a VideoSource / VideoAsset:

import videocontent

video = videocontent.open("https://example.com/talk.mp4")
video.process()
video.save("talk.vctx")

videocontent process https://example.com/talk.mp4 -o talk.vctx

Supported today:

Source	Status
Local file	✅ supported
Direct `http(s)` media URL	✅ supported
YouTube / Vimeo / platforms, S3/GCS/Azure, catalogs, live streams	⬜ planned (adapter extension point, not implemented)

Remote URLs are fetched through a security boundary (SSRF/DNS/redirect validation, size and timeout limits) into a temp file, then processed by the same pipeline — the stages never know where the bytes came from. Check access without downloading:

videocontent source inspect https://example.com/talk.mp4
videocontent source resolve https://example.com/talk.mp4 --json

info = videocontent.inspect_source("https://example.com/talk.mp4")
print(info.accessible, info.media)

Search a Video

from videocontent import Video

video = Video("demo.mp4")

video.process()

results = video.search(
    "pricing",
    top_k=5,
)

for hit in results.spans:
    print(hit.timecode)
    print(hit.modality)
    print(hit.text)

A search result contains timestamped evidence.

For example:

00:18:21
transcript
The speaker begins discussing competitor pricing.

00:18:42
ocr
Competitor Pricing

Search Specific Modalities

Restrict retrieval to particular sources.

Search speech and OCR:

results = video.search(
    "competitor",
    modalities=[
        "transcript",
        "ocr",
    ],
    top_k=5,
)

Search OCR only:

results = video.search(
    "pricing",
    modalities=["ocr"],
)

Search within a time range:

results = video.search(
    "pricing",
    start=600,
    end=900,
)

Inspect a Moment in the Video

Use at() to retrieve information associated with a particular point in time.

snapshot = video.at("03:21")

for span in snapshot.spans:
    print(span.timecode)
    print(span.modality)
    print(span.text)

You can also use seconds:

snapshot = video.at(201.0)

Or include nearby information:

snapshot = video.at(
    "03:21",
    window=5,
)

Load an Existing `.vctx`

Once a video has been processed, the original video is not required for retrieval.

from videocontent import load

video = load("demo.vctx")

results = video.search("pricing")

for hit in results.spans:
    print(hit.timecode, hit.text)

This allows .vctx documents to be shared and queried independently of the original video.

Ask Questions About a Video

VideoContext can retrieve evidence and use an LLM to answer questions.

from videocontent import Video

video = Video("lecture.mp4")

video.process()

answer = video.ask(
    "What was the revenue mentioned in the video?"
)

print(answer.answer)
print(answer.confidence)

for evidence in answer.evidence:
    print(
        evidence.timecode,
        evidence.modality,
        evidence.text,
    )

You can restrict the modalities used for evidence:

answer = video.ask(
    "What pricing was shown?",
    modalities=[
        "transcript",
        "ocr",
    ],
    top_k=5,
)

The answer object contains:

question
answer
confidence
evidence
spans
trace

trace records the query plan (temporal or plain search), retrieval stats, and the LLM provider used — the debugger's entry point for "why this answer".

Temporal questions are planned explicitly:

answer = video.ask("What happened before the error?")
print(answer.trace["query_plan"])
# {'anchor': 'the error', 'relation': 'before', ...}

Supported temporal patterns: before X, after X, between A and B, while X, when did X first appear, and bare timecodes (pricing at 12:20).

Temporal Intelligence

Video is treated as a temporal multimodal world, not a bag of facts. All views below are pure functions over the .vctx document — no reprocessing, no network — and every timestamp is copied from the underlying evidence.

video.timeline(600, 900)   # everything known about a range, in order
video.chapters()           # extractive chapters (titles marked derived)
video.entities()           # Stripe in OCR + "Stripe" in speech → one linked entity
video.changes()            # text turnover, speech transitions, scene boundaries
video.receipt()            # processing receipt: how this context was generated
video.plan("What error appeared?")  # stages needed + coverage against this doc

Entities carry occurrence timestamps, linked modalities, and uncertainty (ambiguous=True when the link is weak). Chapters group scenes with frequency-ranked keyword titles, always inferred=True.

Evidence graph

The derived view agents traverse — never storage, always explainable:

graph = video.graph()                     # 296 nodes, 901 edges on the demo fixture
graph.neighbors("evt_0076")               # [(node, edge)] with construction rules
graph.path("utt_0001", "segment_0001")    # bounded BFS over rule-cited edges
graph.supporting_evidence("evt_0076")     # observed facts behind a derived node
graph.explain("edge_00042")               # the rule, in words, with provenance

video.explain("evt_0076")                 # node view: evidence + neighbors
video.entity_timeline("ConnectionError")  # every sighting in time order
video.query_plan("what happened after the error?")
# → {intent: temporal_after, entities: [...], retrieval_strategy: [...],
#    graph_operations: [...], coverage: {...}, fallback_strategy: ...}

Every edge cites exactly one construction rule (OBSERVED_IN from segment membership, DERIVED_FROM from event refs, PRECEDES from timeline order…). Temporal adjacency is labeled TEMPORAL_SEQUENCE, never causation.

Collections: many videos, one index

from videocontent.collection import CollectionIndex, compare

index = CollectionIndex({"demo-a": doc_a, "demo-b": doc_b})
result = index.search("OAuth login")  # spans tagged with video_id
index.occurrences("ConnectionError")  # every sighting across videos
index.videos_with_all("OAuth", "checkout")  # videos containing both
index.link_entities()                 # collection identity, local evidence kept
diff = compare("demo-a", doc_a, "demo-b", doc_b)
# added / removed / changed / unchanged / uncertain + structure

Collections never merge files — each .vctx keeps its identity and provenance. Timestamps stay video-local; no clock is shared unless you explicitly align one.

Minimal context packages

ctx = video.context(
    "What caused the checkout error?",
    max_spans=5,      # cap evidence spans
    max_seconds=120,  # cap evidence duration
    max_frames=3,     # cap frames
    expand_s=5.0,     # pull ±5 s of co-occurring evidence around each match
)
print(ctx.token_estimate, ctx.budget_notes)

The pipeline is query → retrieval → temporal expansion → dedup → budget → package. budget_notes records every cut, so minimization is auditable.

For agents, video.context_package(task) builds the full first-class package — planned query, budgeted evidence, entities/events/changes in play, frames, graph summary, provenance, warnings, and what was omitted — serializable to JSON, compact text, or Markdown, with opt-in secret redaction that never mutates the original.

CLI

VideoContext provides two CLI entry points:

videocontent

and:

vctx

Process a video

videocontent process demo.mp4

This creates:

demo.vctx

Specify an output location:

videocontent process demo.mp4 \
  --output output.vctx

Write compressed output:

videocontent process demo.mp4 \
  --gzip

Machine-readable output:

videocontent process demo.mp4 \
  --json

Inspect a `.vctx`

videocontent inspect demo.vctx

Inspect the transcript:

videocontent inspect demo.vctx \
  --transcript

Inspect OCR:

videocontent inspect demo.vctx \
  --ocr

Inspect events:

videocontent inspect demo.vctx \
  --events

Inspect segments:

videocontent inspect demo.vctx \
  --segments

Inspect everything:

videocontent inspect demo.vctx \
  --all

Machine-readable output:

videocontent inspect demo.vctx \
  --json

Search

videocontent search \
  demo.vctx \
  "pricing"

Restrict results:

videocontent search \
  demo.vctx \
  "competitor" \
  --modality transcript \
  --modality ocr \
  --top-k 5

Search a time range:

videocontent search \
  demo.vctx \
  "pricing" \
  --from 10:00 \
  --to 15:00

JSON output:

videocontent search \
  demo.vctx \
  "pricing" \
  --json

Inspect a Moment

videocontent at \
  demo.vctx \
  03:21

Include nearby evidence:

videocontent at \
  demo.vctx \
  03:21 \
  --window 5

Ask Questions

videocontent ask \
  demo.vctx \
  "What was the revenue?"

Restrict the evidence:

videocontent ask \
  demo.vctx \
  "What pricing was discussed?" \
  --modality transcript \
  --modality ocr \
  --top-k 5

JSON output:

videocontent ask \
  demo.vctx \
  "What was the revenue?" \
  --json

Show the query plan and trace:

videocontent ask \
  demo.vctx \
  "What happened before the error?" \
  --explain

Timeline, Events, Entities, Changes, Chapters

videocontent timeline demo.vctx --from 10:00 --to 15:00
videocontent events demo.vctx --type error_shown
videocontent entities demo.vctx
videocontent changes demo.vctx
videocontent chapters demo.vctx

All support --json. Temporal search planning:

videocontent search demo.vctx "what happened before the error" --temporal --json
# includes "query_plan": {"anchor": ..., "relation": "before", ...}

Context Packages

videocontent context demo.vctx "What caused the checkout error?" \
  --max-spans 5 \
  --max-frames 3 \
  --expand-s 5 \
  --explain

One-shot Analysis and Agent Setup

videocontent analyze demo.mp4                 # reuse demo.vctx or process once; summarize
videocontent analyze demo.mp4 --no-process    # report only; never process
videocontent analyze demo.mp4 --profile ui_design --json
videocontent init-agent                       # install the skill for detected agents
videocontent init-agent --check               # where is it installed, is it current
videocontent mcp                              # MCP server on stdio

Every query command below accepts the video file as well as its .vctx, and --agent switches output to the bounded agent envelope (schema: videocontent.agent/1).

Agent Commands

videocontent graph demo.vctx --entity ConnectionError
videocontent entity-timeline demo.vctx ConnectionError
videocontent plan demo.vctx "what happened after the error?"
videocontent explain demo.vctx evt_0076
videocontent compare a.vctx b.vctx
videocontent collection search "OAuth login" a.vctx b.vctx
videocontent collection entities a.vctx b.vctx

All support --json. The agent workflow is iterative: inspect a video, search for candidates, explain the evidence IDs, entity-timeline the names, context for the final package — never one giant request.

Check Your Environment

videocontent doctor

Machine-readable output:

videocontent doctor \
  --json

The `.vctx` Format

.vctx is the central temporal context artifact used by VideoContext.

Conceptually:

flowchart TB

    VIDEO["🎬 Raw Video"]

    VIDEO --> PROCESS["Processing Pipeline"]

    PROCESS --> VCTX["📦 .vctx"]

    VCTX --> TRANSCRIPT["Transcript"]
    VCTX --> OCR["OCR"]
    VCTX --> VISION["Vision"]
    VCTX --> OBJECTS["Objects"]
    VCTX --> EVENTS["Events"]
    VCTX --> SCENES["Scenes"]
    VCTX --> SEGMENTS["Segments"]
    VCTX --> FRAMES["Frames"]

A .vctx document can contain information such as:

{
  "vctx_version": "1.0",

  "video": {
    "duration": 62.4,
    "fps": 30.0,
    "width": 1280,
    "height": 720
  },

  "transcript": [],

  "ocr": [],

  "vision": [],

  "objects": [],

  "events": [],

  "scenes": [],

  "segments": [],

  "frames": []
}

The exact contents depend on the processing stages and providers used.

Read the complete specification:

.vctx Format Specification

Processing Architecture

flowchart TB

    VIDEO["🎬 VIDEO"]

    VIDEO --> MEDIA["Media Processing"]

    MEDIA --> META["Metadata"]
    MEDIA --> AUDIO["Audio"]
    MEDIA --> FRAMES["Frames"]

    AUDIO --> ASR["Speech Processing"]

    FRAMES --> OCR["OCR"]
    FRAMES --> VISION["Vision"]
    FRAMES --> OBJECTS["Objects"]
    FRAMES --> SCENES["Scene Detection"]

    ASR --> TEMP["Temporal Processing"]
    OCR --> TEMP
    VISION --> TEMP
    OBJECTS --> TEMP
    SCENES --> TEMP

    TEMP --> EVENTS["Events"]
    TEMP --> SEGMENTS["Segments"]

    EVENTS --> VCTX["📦 .vctx"]
    SEGMENTS --> VCTX

    VCTX --> RETRIEVAL["Retrieval"]

    RETRIEVAL --> SDK["Python SDK"]
    RETRIEVAL --> CLI["CLI"]
    RETRIEVAL --> API["REST API"]
    RETRIEVAL --> MCP["MCP"]

The .vctx document is the shared representation between the processing and retrieval layers.

REST API

VideoContext includes a FastAPI application under:

apps/api

Install API dependencies:

pip install "videocontent[api]"

Run the API:

uvicorn apps.api.main:app \
  --host 0.0.0.0 \
  --port 8000

The API provides:

POST   /v1/videos
GET    /v1/videos/{video_id}

POST   /v1/videos/{video_id}/process
GET    /v1/videos/{video_id}/status

GET    /v1/videos/{video_id}/download

POST   /v1/videos/{video_id}/search
POST   /v1/videos/{video_id}/ask

GET    /v1/videos/{video_id}/timeline
GET    /v1/videos/{video_id}/segments
GET    /v1/videos/{video_id}/frames

GET    /health
GET    /ready

Health Check

curl http://localhost:8000/health

Readiness:

curl http://localhost:8000/ready

Upload a Video

curl -X POST \
  http://localhost:8000/v1/videos \
  -F "file=@demo.mp4"

The response includes a video_id.

Process a Video

curl -X POST \
  http://localhost:8000/v1/videos/VIDEO_ID/process \
  -H "Content-Type: application/json" \
  -d '{}'

Processing configuration can also be supplied in the request body.

Check Processing Status

curl \
  http://localhost:8000/v1/videos/VIDEO_ID/status

Search Through the API

curl -X POST \
  http://localhost:8000/v1/videos/VIDEO_ID/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "pricing",
    "top_k": 5
  }'

Restrict modalities:

curl -X POST \
  http://localhost:8000/v1/videos/VIDEO_ID/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "competitor",
    "modalities": [
      "transcript",
      "ocr"
    ],
    "top_k": 5
  }'

Ask a Question Through the API

curl -X POST \
  http://localhost:8000/v1/videos/VIDEO_ID/ask \
  -H "Content-Type: application/json" \
  -d '{
    "question": "What pricing was discussed?",
    "top_k": 5
  }'

The response contains:

Question
Answer
Confidence
Evidence

Get Timeline Information

curl \
  http://localhost:8000/v1/videos/VIDEO_ID/timeline

Restrict the time range:

curl \
  "http://localhost:8000/v1/videos/VIDEO_ID/timeline?start=60&end=120"

Get Entities, Changes, Chapters, Receipt

curl http://localhost:8000/v1/videos/VIDEO_ID/entities
curl http://localhost:8000/v1/videos/VIDEO_ID/changes
curl http://localhost:8000/v1/videos/VIDEO_ID/chapters
curl http://localhost:8000/v1/videos/VIDEO_ID/receipt

Agent Intelligence

curl -X POST http://localhost:8000/v1/videos/VIDEO_ID/plan \
  -H "Content-Type: application/json" \
  -d '{"question": "what happened after the error?"}'
curl "http://localhost:8000/v1/videos/VIDEO_ID/entity-timeline?name=ConnectionError"
curl "http://localhost:8000/v1/videos/VIDEO_ID/evidence?ref=evt_0076&ref=ocr_0001"
curl "http://localhost:8000/v1/videos/VIDEO_ID/explain?ref=edge_00042"

Collections (timestamps stay video-local):

curl -X POST http://localhost:8000/v1/collections \
  -H "Content-Type: application/json" \
  -d '{"video_ids": ["A", "B"]}'
curl -X POST http://localhost:8000/v1/collections/COLLECTION_ID/search \
  -H "Content-Type: application/json" \
  -d '{"query": "OAuth login"}'
curl http://localhost:8000/v1/collections/COLLECTION_ID/entities
curl -X POST http://localhost:8000/v1/collections/compare \
  -H "Content-Type: application/json" \
  -d '{"video_a": "A", "video_b": "B"}'

MCP Server

Recommended: videocontent mcp — built into the package, no extra dependencies, ten bounded tools (videocontent_inspect, _analyze, _search, _ask, _timeline, _entities, _changes, _context, _compare, _explain) that take a video path directly.

claude mcp add videocontent -- videocontent mcp     # Claude Code
codex mcp add videocontent -- videocontent mcp      # Codex
videocontent init-agent --mcp                        # both, via their own CLIs

Any other MCP client: command videocontent, args ["mcp"], stdio transport. See docs/agents/generic-agents.md.

Legacy MCP server (`apps/mcp`)

The original server below takes video_ids and requires the mcp 1.x SDK (its API was removed in mcp 2.x, hence the <2 pin in the extra):

apps/mcp

pip install "videocontent[mcp]"

The MCP server exposes processed video context to MCP-compatible AI applications and agents.

flowchart LR

    AGENT["🤖 AI Agent"]

    AGENT --> MCP["VideoContext MCP Server"]

    MCP --> SEARCH["Search"]
    MCP --> TRANSCRIPT["Transcript Search"]
    MCP --> OCR["OCR Search"]

    MCP --> EVENTS["Event Lookup"]
    MCP --> OBJECTS["Object Lookup"]

    MCP --> SEGMENTS["Segment Retrieval"]
    MCP --> FRAMES["Frame Retrieval"]
    MCP --> TIMELINE["Timeline"]

    MCP --> ASK["Video Q&A"]

    SEARCH --> VCTX["📦 .vctx"]
    TRANSCRIPT --> VCTX
    OCR --> VCTX
    EVENTS --> VCTX
    OBJECTS --> VCTX
    SEGMENTS --> VCTX
    FRAMES --> VCTX
    TIMELINE --> VCTX
    ASK --> VCTX

The available tools are:

inspect_video

search_video
search_transcript
search_ocr

find_event
find_object

get_entities
get_entity_timeline
get_events
get_evidence
find_changes
get_chapters

get_segment
get_frame
get_timeline

ask_video
get_context
explain_evidence
explain_relation
find_before
find_after
compare_videos

register_collection
search_collection

`search_video`

Search across available modalities.

Example input:

{
  "video_id": "VIDEO_ID",
  "query": "pricing",
  "top_k": 10
}

`search_transcript`

Search speech only.

{
  "video_id": "VIDEO_ID",
  "query": "revenue",
  "top_k": 10
}

`search_ocr`

Search on-screen text.

{
  "video_id": "VIDEO_ID",
  "query": "pricing",
  "top_k": 10
}

`find_event`

Find events by type.

{
  "video_id": "VIDEO_ID",
  "event_type": "slide_changed"
}

`find_object`

Find detected objects.

{
  "video_id": "VIDEO_ID",
  "label": "person"
}

`get_segment`

Retrieve a segment.

{
  "video_id": "VIDEO_ID",
  "segment_id": "SEGMENT_ID"
}

`get_frame`

Retrieve frame metadata.

{
  "video_id": "VIDEO_ID",
  "frame_id": "FRAME_ID"
}

`get_timeline`

Retrieve timeline information.

{
  "video_id": "VIDEO_ID",
  "start": 60,
  "end": 120
}

`ask_video`

Ask a question about processed video context.

{
  "video_id": "VIDEO_ID",
  "question": "What was the main pricing discussion?",
  "top_k": 5
}

The MCP server returns the answer together with evidence from the video context.

`inspect_video`

Describe a processed video: duration, fact counts per modality, stage statuses.

{
  "video_id": "VIDEO_ID"
}

`get_entities`

Timestamp-grounded entities with uncertainty flags.

{
  "video_id": "VIDEO_ID",
  "top_k": 20
}

`find_changes`

What changed between adjacent regions, with evidence.

{
  "video_id": "VIDEO_ID"
}

`get_chapters`

Extractive chapters (titles are derived keywords, ranges are factual).

{
  "video_id": "VIDEO_ID"
}

Agent iteration tools

Designed for inspect → search → inspect evidence → refine → context:

Tool	Purpose
`get_entity_timeline`	Every occurrence of an entity in time order
`get_events`	Events in a range, optionally by type
`get_evidence`	Exact facts behind reference IDs
`get_context`	Budgeted package, Markdown or JSON
`explain_evidence`	Supporting evidence for a node/fact
`explain_relation`	Construction rule behind an edge
`find_before` / `find_after`	Temporal lookup around an anchor
`compare_videos`	Added/removed/changed/uncertain
`register_collection` + `search_collection`	Multi-video search with video IDs

{
  "video_id": "VIDEO_ID",
  "entity": "ConnectionError"
}

{
  "collection_id": "COLLECTION_ID",
  "query": "OAuth login",
  "top_k": 10
}

All MCP tools cap result counts so agent context stays small and traceable.

Web Application

VideoContext includes a web application under:

apps/web

The web application is built with:

React
TypeScript
Vite
Tailwind CSS

Install dependencies:

cd apps/web

npm install

Run the development server:

npm run dev

Build for production:

npm run build

flowchart LR

    USER["User"]

    USER --> WEB["React Web Application"]

    WEB --> API["VideoContext REST API"]

    API --> PROCESS["Processing"]

    API --> RETRIEVAL["Retrieval"]

    PROCESS --> VCTX["📦 .vctx"]

    RETRIEVAL --> VCTX

Configuration

VideoContext uses ProcessingConfig.

from videocontent import ProcessingConfig, Video

config = ProcessingConfig()

video = Video(
    "lecture.mp4",
    config=config,
)

video.process()

Configuration can also be provided through CLI configuration and overrides.

Example:

videocontent \
  --set sampling.mode=adaptive \
  process demo.mp4

The CLI also supports a YAML configuration file:

videocontent \
  --config config.yaml \
  process demo.mp4

Provider Architecture

VideoContext separates the core representation from the implementations used to process and query video.

flowchart TB

    CORE["VideoContext Core"]

    CORE --> ASR["Speech Providers"]

    CORE --> OCR["OCR Providers"]

    CORE --> VISION["Vision Providers"]

    CORE --> EMB["Embedding Providers"]

    CORE --> LLM["LLM Providers"]

    CORE --> RET["Retrieval"]

The project contains provider and registry infrastructure that allows different implementations to be used without changing the central temporal representation.

Project Architecture

flowchart TB

    ROOT["VIDEOContext"]

    ROOT --> APPS["apps"]

    APPS --> API["api"]
    APPS --> MCP["mcp"]
    APPS --> WEB["web"]

    ROOT --> SRC["src/videocontent"]

    SRC --> CLI["cli"]
    SRC --> EMB["embeddings"]
    SRC --> JOBS["jobs"]
    SRC --> LLM["llm"]
    SRC --> MEDIA["media"]
    SRC --> PROC["processing"]
    SRC --> RET["retrieval"]
    SRC --> SCHEMA["schema"]
    SRC --> STORAGE["storage"]

    ROOT --> DOCS["docs"]
    ROOT --> TESTS["tests"]
    ROOT --> SCRIPTS["scripts"]

The applications and core library are separated.

The main package lives under:

src/videocontent

Applications live under:

apps/

Development

Clone the repository:

git clone https://github.com/AAGAM17/VIDEOContext.git

cd VIDEOContext

Install development dependencies:

pip install -e ".[dev]"

Run the test suite:

pytest

The repository uses:

pytest
ruff
mypy

Run Ruff:

ruff check src tests

Run MyPy:

mypy

Documentation

The repository includes additional technical documentation.

Architecture

docs/ARCHITECTURE.md

Describes the project architecture and design decisions.

`.vctx` Specification

docs/VIDEO_CONTEXT_SPEC.md

Describes the temporal context document format.

Roadmap

docs/ROADMAP.md

Describes the planned development direction.

Project Status

Version: 0.1.0

VideoContext is currently in the Alpha stage.

The current repository includes:

Core Python package
Video processing pipeline
Universal source layer (local files + direct URLs, inspect/resolve, SSRF-safe ingestion)
.vctx temporal context format (with source provenance)
Speech processing infrastructure
OCR infrastructure
Vision infrastructure
Object information
Event information
Scene information
Temporal segmentation
Retrieval
Python SDK
CLI
LLM-backed question answering
REST API
MCP server (built-in videocontent mcp, plus the legacy apps/mcp)
Agent integration: provider-neutral skill, Claude Code and Codex plugins, init-agent
React web application
Tests
Documentation
Development tooling

The project is actively evolving.

See the Roadmap for planned development.

Contributing

Contributions are welcome.

If you would like to contribute:

Fork the repository.
Create a branch for your changes.
Make the changes.
Add or update tests where appropriate.
Run the existing test suite.
Open a pull request describing your changes.

Useful areas for contributions include:

Video processing
Speech-to-text
OCR
Vision providers
Object detection
Event extraction
Embeddings
Retrieval
LLM integrations
MCP integrations
API development
Web application development
Performance
Testing
Documentation

For larger architectural changes, opening an issue or discussion first can help align the implementation with the existing project structure.

License

VideoContext is licensed under the Apache License 2.0.

See LICENSE.

flowchart LR

    V["🎬 RAW VIDEO"]

    V --> P["MULTIMODAL<br/>PROCESSING"]

    P --> E["TIMESTAMPED<br/>EVIDENCE"]

    E --> C["📦 .VCTX"]

    C --> R["SEARCH<br/>Q&A<br/>API<br/>MCP"]

    R --> A["🤖 AI APPLICATIONS"]

Name		Name	Last commit message	Last commit date
.claude-plugin		.claude-plugin
.codex-plugin		.codex-plugin
.github/ISSUE_TEMPLATE		.github/ISSUE_TEMPLATE
.vctx-work/demo/frames/pass		.vctx-work/demo/frames/pass
apps		apps
benchmarks		benchmarks
docker		docker
docs		docs
examples/agents		examples/agents
scripts		scripts
skills/videocontent		skills/videocontent
src/videocontent		src/videocontent
tests		tests
website		website
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
benchmark_results.json		benchmark_results.json
pyproject.toml		pyproject.toml
retrieval_benchmark.json		retrieval_benchmark.json
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

VIDEOContext

Give AI agents context they can reason over from video.

Install

Use VIDEOContext with AI coding agents

CLI in 30 seconds

Why VideoContext?

Features

🎬 Video Processing

📦 Temporal Context

🔎 Search and Retrieval

AI Integration

Installation

Core package

Speech-to-text

OCR

Vision adapters

Vector retrieval

Embeddings

REST API

MCP server

Main optional processing stack

Requirements

Quick Start

Process a video

Process from any supported source

Search a Video

Search Specific Modalities

Inspect a Moment in the Video

Load an Existing .vctx

Ask Questions About a Video

Temporal Intelligence

Evidence graph

Collections: many videos, one index

Minimal context packages

CLI

Process a video

Inspect a .vctx

Search

Inspect a Moment

Ask Questions

Timeline, Events, Entities, Changes, Chapters

Context Packages

One-shot Analysis and Agent Setup

Agent Commands

Check Your Environment

The .vctx Format

Processing Architecture

REST API

Health Check

Upload a Video

Process a Video

Check Processing Status

Search Through the API

Ask a Question Through the API

Get Timeline Information

Get Entities, Changes, Chapters, Receipt

Agent Intelligence

MCP Server

Legacy MCP server (apps/mcp)

search_video

search_transcript

search_ocr

find_event

find_object

get_segment

get_frame

get_timeline

ask_video

inspect_video

get_entities

find_changes

get_chapters

Agent iteration tools

Web Application

Configuration

Provider Architecture

Load an Existing `.vctx`

Inspect a `.vctx`

The `.vctx` Format

Legacy MCP server (`apps/mcp`)

`search_video`

`search_transcript`

`search_ocr`

`find_event`

`find_object`

`get_segment`

`get_frame`

`get_timeline`

`ask_video`

`inspect_video`

`get_entities`

`find_changes`

`get_chapters`

`.vctx` Specification