videoTokenizer · RTX 3090 ·

Sparse Tokenizer Bench

The tokenizer cuts each 128 × 128 frame into an 8 × 8 grid of 16 px cells. A router decides which cells carry an event this frame; a fovea gives a few of them high resolution. A temporal router then groups frames into level-1 and level-2 tokens. This page tracks, run by run, whether each of those choices lands where the video actually changes.

event cell (router) high-res cell (fovea) level-1 temporal token level-2 temporal token pixels changed (ground truth)

Model walkthrough · rs_base on a held-out real clip

How the tokenizer works, step by step

rs_small, rs_base and rs_large are one fixed design with learned decisions inside it; the three differ only in widths and in the depth of the top-level network. The video below runs rs_base once on held-out and draws every intermediate tensor: the masks, scores, attention maps, tokens and losses are the model's own. Nothing in it is mocked up.

Fixed by hand (the scaffold)

128 × 128 frames cut into an 8 × 8 grid of 16 px cells
an always-on cheap path: each cell downsampled to 4 × 4 px (48 numbers)
a heartbeat that forces each cell to update once every 64 frames, one cell per frame along a diagonal
caps per frame: at most 12 routed cell events, exactly 2 high-res cells
two temporal stages, each aiming for 1 token per 4 inputs; heartbeats every 64 and 256 frames
how actions enter: a_t−1 is added to the frame vector (dropped half the time in training); a_t conditions the next-frame model

Learned

when a cell fires: the router's W_q, W_k, threshold β and temperature τ
which event cells get high resolution: the fovea score, trained to predict how much high resolution would lower the action loss
when frames become level-1 and level-2 tokens: one router per temporal stage
all embeddings, the spatial and temporal transformers, the pooling queries, the memory read and the heads

Tensor shapes

B = 8 clips and T = 256 frames per training step. Parameter counts include the main network once (the checkpoint stores it under two names).

Loss (pixel objective, as rs_base, rs_small and rs_large were trained)

Real video · long-video memory experiment

Can the model recall a hidden part of a scene after 27 minutes?

Recall example: cue, masked query, truth and each model's reconstruction of the hidden block

Masked revisit (real/train_mem.py). A camera is shown for 10 s (cue), then G frames of other robots, then the same camera returns with a 4×4 block of cells (a quarter of the frame) blacked out on every query frame. The model must paint the hidden block. Orange box: the hidden block. Each model's reconstruction is pasted in; mse is over the block in this frame.

Error on the hidden block vs gap (held-out, lower is better)

Solid: the true cue. Dashed: control cue (same camera, a different time) with identical filler and query. Grey dotted: copying the block from the true cue's last frame; grey dash-dot: from the control cue's last frame.

Memory benefit: error(control cue) − error(true cue)

Positive = the specific episode is remembered. ± 1 standard error over pairs.

Where the cell memory looks (cellmem, true cue)

Share of the recall head's attention mass on memory entries from the cue, the filler (other robots) and the query frames themselves.

Training: hidden-block error

Smoothed over 10 logged points; the gap grows from 1K to 16K frames over the first half of training.

Metrics

Diagnostic: is the cue's cell state enough, given perfect retrieval?

Real video · long-term memory probe

How far back does the model remember?

Filmstrip of one probe pair: the matching and mismatched stream share filler and query

History swap (real/probe_memory.py). Two streams differ only in their first 200 frames: a cue from the query's own camera (match) or from another robot (mismatch). Then G frames of other held-out robots, then later frames of the cue's camera. Everything after the cue is identical, so any difference in the model's state at the query is memory of the cue. Measured on the first 32 query frames, previous actions zeroed, 16 pairs per gap.

Influence: how much the cue still changes the state h at the query

‖h_match − h_mismatch‖ / ‖h_match‖, mean over pairs. Log scale. Solid: full model. Dashed: frame-rate memory read switched off. Dotted: main network output and memory zeroed.

Benefit: does the matching cue make the query easier to predict?

Error with the mismatched cue minus error with the matching cue (positive = memory helps), full model, ± 1 standard error over pairs.

Next: matched comparison against RoboTTT

Real robot video · held-out episodes

Watch it tokenize

Left to right: frame t with rs_base's event and high-res cells; the same for the pixel-change heuristic rs_heur; rs_base's prediction of frame t+1; the true frame t+1. Strip: the last 96 frames of motion, events per frame and temporal tokens. The clip is the 150-frame window with the most motion among the first 1,500 held-out frames, at the stream's own frame rate.

Probe 1 · temporal router

Does the temporal router fire on real video?

Level-1 tokens per second divided by the heartbeat rate (one forced token every 64 frames). 1× means the router never chose a boundary itself. Bar = all frames; ● = the 25% most moving frames, ○ = the 25% stillest. Log scale.

Probe 2 · spatial router and fovea

Where do the events land?

All 17 held-out streams at once, 200 frames each from their most active stretch, played at 10 fps (5 fps streams run 2× fast, 15–20 fps ones slower). Per cell: magenta corners = pixels changed since the last frame; yellow outline = the router chose an event; thin grey outline = an event forced by the heartbeat, which updates each cell once every few frames, staggered across the grid; cyan = the fovea gave the cell high resolution. Counters per tile: router events, changed cells, hits (router events on changed cells), high-res cells.

Averaged over 3,000 frames

Each square is one held-out stream (a frame from its middle), tinted by how often each of the 64 cells is changed / gets an event / gets high resolution, over up to 3,000 held-out frames. Numbers: event recall and precision against changed cells.

Probe 3 · what the tokens are good for

Next-frame prediction and actions from vision

Next-frame PSNR minus copying frame t (dB)

Right of the line: better than repeating the last frame.

Action error from vision ÷ predicting zero

Previous actions zeroed, so the model must read the action from the video. Left of the line: better than predicting zero.

All runs, mean over 17 streams

One seed per configuration. rs_d1 and rs_d4 never saw most of these streams in training, so their all-stream means are not a data-scaling result; see the next table.

Data scaling, on the streams the small-data runs were trained on

Action error from vision (lower is better; predicting zero shown for reference) and PSNR gain over copying. All runs: 8,000 steps, B 8, T 256.

Toy stream · JEPA objective

Does a latent objective stop paying for the noisy TV?

Held-out toy stream, 10 fps. Tags: A arm, I item, R rule symbol (shown for 5 frames per flash), TV the noisy TV, which changes every frame but never matters for the task. Strip: events on the TV cell, the rule cell (dark marks = frames where the rule is shown) and the arm + item cells.