Composing Capability

A lot of my research begins with video. When a talk is worth studying closely, I usually extract its transcript with yt-dlp, pass it to a text model, and turn the result into structured notes.

That workflow looked good enough until I began checking those notes against the original videos. The notes looked complete. They were not.

Speakers often complete spoken ideas with something visible on screen. They rely on phrases like “this,” “right here,” and “as you can see,” whose meaning depends on that visual target. Once a video is flattened into text, the target disappears.

The model could not inspect what was being discussed. More importantly, the failure did not look like a failure. The notes could supply a plausible interpretation without marking that the visual evidence was missing. They sounded complete even when the source representation was not.

In a research workflow, that is more dangerous than an obvious error. An unsupported inference can quietly enter the record as though it came from the source.

I rebuilt the workflow around that failure.

I built video-context-mcp to process a video into an aligned working set of transcript segments, OCR-derived screen text, keyframes, and timestamps.

Through the Model Context Protocol (MCP), an agent can search that material and inspect the relevant moment.

Instead of receiving one flattened transcript, the agent can investigate speech and screen evidence as distinct but connected channels—all while continuing to operate through text.

The text model had not learned to watch video. It remained bounded by its native capabilities.

What changed was the system around it.

What started as an attempt to improve a video-ingestion workflow ended up moving part of the research capability outside the model.

I call this evidence-path engineering: preparing sources, preserving retrieval state, and maintaining verification outside the connected model so that each claim retains a path back to its evidence.

Video is an ideal stress test because meaning is distributed across speech, visuals, and time. The broader question is which practical capabilities can be composed around a model boundary—and what remains stable when the model or harness changes.

A model boundary is not a source boundary

The failure became easier to reason about once I separated two boundaries that transcript-first workflows tend to collapse.

The interaction boundary determines what kinds of information can reach the model during a particular run. A text-based model can receive language but cannot directly inspect a frame. An image-capable model can inspect selected frames. A model and harness with native-video input can accept a larger temporal sequence.

The source boundary determines what remains available around and after that run. Treating a video as a source requires more than accepting it as an attachment. Its evidence should remain queryable across questions, its claims should resolve to specific moments, and another person or agent should be able to inspect the same material while the prepared working set exists.

Those notes failed first at the interaction boundary: the transcript crossed into context, but the slide did not. They also exposed a source-boundary problem. Once the transcript had been summarized, the resulting claims no longer resolved to the speech and screen moments from which they had been assembled.

That diagnosis matters because “the model cannot do this” can conceal several different failures. The model may lack the required modality. The harness may not deliver a frame. The source representation may omit the relevant evidence. The retriever may select the wrong moment. The final claim may have no resolvable source handle. Each failure occurs at a different boundary and calls for a different remedy.

Changing models is one remedy. Direct video input may be the simplest choice for a one-off question, especially when motion or temporal relationships carry the answer.

But model choice is rarely determined by modality alone. A research workflow may already depend on a harness’s tools, permissions, state, privacy boundary, or local deployment. On constrained hardware, practical native-video inference may not be available. Switching models can solve the immediate input mismatch while requiring the surrounding workflow to be rebuilt.

Even when it is available, native input does not automatically make a source reusable, selectively searchable, or verifiable across later questions. Conversely, a prepared evidence path can provide those properties while still losing motion, gesture, or visual structure that direct video analysis would preserve.

The design question is not which path is universally superior. It is which evidence the task requires, which boundary currently blocks it, and whether an alternative representation can cross that boundary without losing the decisive meaning.

Building the evidence path

The answer was not to dump more material into the prompt. An entire transcript plus hundreds of frames can quickly gather noise. The source needed to be prepared broadly and disclosed narrowly.

video-context-mcp turns speech, OCR, keyframes, metadata, and timestamps into an aligned evidence working set. A question begins with lightweight search results and identifiers. The agent can then request a transcript range, inspect a frame, or crop a region when the current evidence is insufficient. The complete artifact remains outside the model’s immediate context.

Five boundaries make the intervention reusable beyond this one video.

Representation asks which features of the source can become forms the system can manipulate. In this case, speech becomes a timestamped transcript, visible language becomes OCR text, and selected screen states become keyframes. None is the video itself. Each preserves something and removes something.

Context asks whether the system can select a sufficient evidence bundle for the current question. Indexing the whole source makes later discovery possible; retrieving only the relevant speech-and-screen neighborhood keeps the current interaction focused.

Action asks whether the agent can discover and sequence the operations that expose missing evidence. Search, transcript retrieval, frame inspection, cropping, and follow-up queries let the agent investigate the source instead of waiting for a person to anticipate every necessary screenshot.

State asks what survives across turns and retries. A stable video identifier, prepared index, and resolvable evidence handles prevent every question from starting with ingestion again. This is reusable working state, not human-like memory, and its lifetime and cleanup behavior remain part of the system contract.

Verification asks whether a claim can be checked against the original artifact and the task’s success criteria. Timestamps and channel labels create a route back to the video. They do not make an interpretation correct by themselves; a citation-shaped timestamp is useful only when it resolves to evidence that supports the claim.

These boundaries form a loop rather than a one-way enrichment pipeline. The model interprets the available evidence, decides whether another action is needed, observes the result, and interprets again. Deterministic infrastructure extracts, indexes, filters, and locates. The connected model chooses actions and connects meanings. The user defines the research question and verifies consequential or ambiguous results.

MCP contributes an action and delivery contract; another protocol or application could implement the same evidence architecture. An MCP-compatible harness may still discover different tools, deliver different result modalities, or preserve state differently.

The intervention therefore degrades rather than failing as an all-or-nothing feature. A text-only model can use transcript, OCR, search results, and timestamps while remaining unable to interpret a diagram’s geometry. An image-capable model can inspect selected frames and crops. A native-video model may not need this decomposition for one question, but it can still benefit from repeatable retrieval and stable verification handles when the source must be used again.

What the trace proves

The strongest available evidence is a deterministic replay of a real investigation of Coyle’s talk. The explanation depends on joining three source moments:

• At 18:14, speech distinguishes structural input checks from ontology-level result checks.
• At 18:43, the screen supplies the compact formulation “Pydantic in, ontology out.”
• At 19:11, the screen grounds the distinction in a concrete rule: an order can be refunded at most once; a second refund is the double-payout bug.

The replay shows the tool sequence retrieving those moments, distinguishing speech from screen evidence, and presenting them as one timestamped bundle. The observable change is narrower than “the model became better at video.” Before the intervention, the working representation contained only the speech projection. After it, the system could locate and expose evidence from both channels while keeping the source moments resolvable.

That demonstrates capability composition: the mechanism exists, and the assembled system can perform an evidence-retrieval sequence that the transcript-only path does not provide.

It does not yet demonstrate capability amplification, meaning a measured difference in answer quality between a named baseline and the assembled system. The replay is deterministic, not a live comparative model run. A proper paired test must hold the model, harness, video, prompt, and answer requirements constant: one condition receives the relevant transcript only; the other can retrieve the speech-and-screen bundle. Both outputs must then be checked against the same claims, channel attributions, and timestamps.

Until that trace is captured, the defensible result is architectural and observable: a previously absent evidence channel became retrievable and inspectable. One demonstration should not be inflated into a benchmark.

That proof discipline is part of the strategy. Attribute every success to the model, and we cannot tell whether the gain came from reasoning, retrieval, a tool contract, harness delivery, or hidden human work. Attribute every success to the tool, and we erase the model’s interpretation and the harness’s action loop. A trace makes the division of labor visible.

The system gained the capability

What, exactly, gained the capability?

The model is the underlying model and its supported interface. The harness is the runtime that manages tools, state, and result delivery; an agent is the model acting through that harness. The system is the complete assembly of model, harness, tools, evidence, state, environment, and supervision. Model capability describes what the first can do through its interface. System capability describes what the full assembly can accomplish on a defined task under stated conditions.

A source-model mismatch can be addressed at three layers. The model can change so the source becomes a native input. The source can be transformed into supported representations. Or the harness can give the model an iterative way to request and inspect those representations. These interventions are not equivalent, but all three can change what the assembled system can practically do.

This sits inside the shift toward compound AI systems: useful behavior emerging from models combined with retrieval, tools, and control logic. Harness engineering makes those environments legible to an agent; context engineering selects the small, high-signal set the model receives.

Evidence-path engineering applies that concern for legibility to the source itself. It asks how an artifact becomes selectively inspectable by a harness, how task-relevant evidence crosses the model’s supported interface, and how the resulting claims remain connected to the source.

The same pattern appears in semantic-code-mcp. Rather than feeding an entire private codebase into a prompt, external infrastructure builds an addressable semantic index. The agent retrieves narrow code slices, and the result retains links back to the source files. The model is not retrained; the evidence around it is represented and selected differently.

Under this view, capability belongs to a task-system pair under stated conditions. Model capability is not automatically the upper bound of system capability. System reliability, however, remains limited by the weakest necessary interface.

What survives when the harness changes?

Better research notes were the first payoff. A second became visible because I move among different harnesses depending on the work: why should the expensive preparation of one source have to restart whenever the reasoning environment changes?

If the video evidence is trapped inside one conversation, source format quietly dictates model and harness choice. If the prepared evidence exists outside that conversation, another compatible agent can begin from the same source working set without automatically discarding the investigation that preceded it.

The tempting claim is that a shared protocol makes the capability portable. It does not. A shared protocol can make an interface portable. The full capability still depends on tool discovery, permissions, result delivery, context management, model modality, reasoning, and supervision.

Portability is layered. Source continuity means the agents refer to the same artifact. Representation continuity means they can begin from the same transcript, OCR, keyframes, and metadata while that working set remains available. Evidence continuity means their claims can resolve to the same source moments. Interface continuity means compatible harnesses can request the same operations. Workflow continuity means they can attempt the same pattern of search, inspection, follow-up, and verification.

Outcome continuity is different. Two agents can receive equivalent evidence and still select different actions, interpret a frame differently, stop at different points, or produce answers of different quality. Outcome continuity must be measured rather than inferred from protocol support.

This is one reason stable interfaces matter. Anthropic’s Managed Agents architecture, for example, separates the model and harness, tool-execution environment, and session record so those layers can evolve independently. Evidence-path engineering does not solve general model switching, but it offers one similar seam: source preparation and verification handles can live outside a single model’s conversational history.

The current video implementation provides a reusable working set but not durable institutional memory. Long-lived organizational use would require persistence, export, access control, lifecycle policy, and evidence handles that continue to resolve over time.

The precise claim is smaller and more useful: evidence access can be more portable than the capability outcome.

Where composition stops

Composition is not a universal substitute for native capability. It works only when decomposition preserves enough evidence for the task.

Video supplies obvious failure cases. A question about the exact sequence of keypresses in a coding demonstration, for example, may depend on a button that flashes red for a single frame. A keyframe extractor configured around scene changes or fixed intervals may omit the event; the transcript may never mention it, and OCR may never capture it. The composed evidence path would be blind to the very thing the question requires. Similar losses occur when meaning depends on gesture, motion, spatial structure, or metadata the harness does not deliver.

In those cases, the system should expose its insufficiency rather than produce a fluent approximation. The correct next step may be another inspection, human review, direct continuous-video analysis, or a different system.

Composition also creates liabilities. Every retriever, cache, tool, permission, and control loop can add latency, cost, synchronization failures, debugging burden, and attack surface. A sophisticated workaround can become dead weight when a newer model supports the task directly. Evidence and verification infrastructure should remain only where it continues to earn its complexity.

The lesson is not that every missing model feature can be reconstructed with tools. It is that the model boundary is not always the capability boundary.

My original notes looked complete because the missing evidence had disappeared before the model could object to its absence. The system became more useful only when that loss became visible, inspectable, and recoverable.

Explore video-context-mcp

The implementation is available in the video-context-mcp repository and on npm.