A video enters a harness in one place, the page that holds the file, and becomes one attachment in one command. The page decodes, because that is where a codec is. The host never decodes video and never parses a container a stranger dropped in.
The four steps
ingestVideo from @lloyal-labs/ui runs them:
- Ask. The page sends
video:describeand gets back how this host composes a video: its sampling, the longest clip it takes, and the limits a soundtrack window must meet when anything the host runs hears. A clip longer than the host takes is refused from the container's own duration, before a byte is pushed. - Push the original as a blob, in chunks, over the OCI distribution blob upload API the host serves at
/v2/…/blobs/uploads/. A dropped chunk resumes from what the host reports it holds, and the blob is admitted only under the digest of the whole file. It is kept: it is what the reader brought. - Sample and ingest. Frames are sampled at the host's rate and each enters the store as an image. When something the host runs hears, the soundtrack is cut into windows and each enters as a recording.
- Compose. One
video:composenames every frame, every soundtrack window and the original. The host checks each reference, transcribes each window when a transcription service is bound, and commits one attachment.
import { ingestVideo, useHarness, useVideoCapabilities } from "@lloyal-labs/ui";
import { ingestMedia, isVideoContainer } from "@lloyal-labs/media";
const { bridge } = useHarness();
const video = useVideoCapabilities();
const root = isVideoContainer(new Uint8Array(await file.slice(0, 64).arrayBuffer()))
? (await ingestVideo({ origin, bridge, file, capabilities: video, onProgress })).root
: await ingestMedia(origin, new Uint8Array(await file.arrayBuffer()));Serve the commands on the host, with the harness's config reader, so each video is described and composed under attachments.video. The video keys are set in harness.yml and fixed for the run, since the page samples by the policy it was described. The group answers only once it is served with the others:
import { serveCommands, serveDefaults, useVideoCommands } from "@lloyal-labs/rig";
const video = yield* useVideoCommands({ wire, config: runner.config });
yield* serveCommands<Command>(commands, [brief, library, voice, video /* , … */], serveDefaults({ wire, run }));Sampling
- Times. Frames are
fpsapart, from the picture's first frame, up tomaxFrames; a longer clip is sampled evenly across its length instead. Each frame is stamped with the time the decoder showed it, on the file's own clock. A frame shown for several sample times is kept once. - Size. Each frame is scaled so its longer edge is at most
maxEdgePx, and encoded as JPEG. - Sound. The channels are mixed to mono, at a rate every hearing service admits. Sound that starts after the picture is placed at its own time, the gap silent.
- Windows. The soundtrack is cut into windows of the length the host asks for,
attachments.video.soundtrackWindowSeconds, five seconds by default, each ending at the quietest point near its end and no longer than every service that hears admits. The length doesn't follow the frames' time marks: a transcript note joins the first mark at or after it, however the soundtrack was cut. When nothing the host runs hears, no soundtrack is cut at all.
The settings and their bounds are on harness.yml. Configuring video is what each one trades.
Transcription
With a transcription service bound, the host transcribes each soundtrack window when the video is composed. A silent window is skipped. A window the service fails on is reported back as untranscribed, with its reason. The video keeps that window's recording; only its transcript is missing. Without a transcription service the video still arrives, with no transcript and no error. When an embedding model that hears is bound, the soundtrack is still cut and kept, and each window comes back untranscribed, because transcription is not configured.
Swapping the decoder
What a decoder does, and all it does. A decoder is media's ClipDecoder: duration, picture, frames, sound and dispose. frames yields, for each time asked, the frame shown at that time, encoded as a video frame, with the time it is shown from. sound yields decoded sound in order, each run at its own time.
What stays the same whichever decoder reads the file. prepareVideo is the procedure, written once against that contract: the times sampled, how frames are stamped and sized, the soundtrack's mix, its rate, its placement and its cut, and the compose it produces. A decoder swap cannot change what a model receives beyond the pixels and samples the decoder hands over.
The decoders that exist. In the page, mediabunnyDecoder reads the container with Mediabunny and decodes through WebCodecs. Where the platform's decoder refuses a stream, it reads on in software from where the platform stopped.
Reading a clip from any runtime. Ask video:describe, read through your decoder with prepareVideo, and send the compose it gives you:
import { prepareVideo } from "@lloyal-labs/media";
const prepared = await prepareVideo(decoder, capabilities, ingest, { signal, onProgress });
send({ type: "video:compose", requestId, ...prepared, source });On a host with no canvas, encodeVideoFrame from @lloyal-labs/media/node encodes a decoded frame as the page does, at the same quality. The shipped host still never decodes video; choosing a decoder for it is not done yet.
Refusals
Each is said in a sentence the reader sees, and none leaves a half-composed video behind:
- a clip longer than the host takes, before any byte is pushed;
- more frames than the host samples;
- a reference that is not an admitted root, a soundtrack window that is not a recording, runs past the clip or overlaps the one before it, before anything is transcribed;
- a chunk whose range does not describe its body, and a whole whose digest does not match.
Cancelling
Aborting ingestVideo stops the decoder and ingests nothing more. Aborted as composing begins, the page sends nothing; aborted after, it sends video:cancel, and the host stops mid-transcription and composes nothing. A video already composed is not cancelled.
Progress
ingestVideo reports uploading with how much of the original is pushed, sampling, ingesting with how many of the frames are in, then composing once it sends the compose. While the host transcribes, it reports each window as video:progress, which ingestVideo passes on as transcribing.
Live capture
captureFrames reads frames from a live track, such as a camera, in groups. Each group is ingested and composed as a live capture: a video with no original and no transcript. It needs a browser that can read frames from a track, which Chromium and Electron can.
Related
- Content-addressed video storage: what the compose commits.
- Video protocol reference: every command and event.