The Generator Is the Tracker

Haiyu Yang, Miel Hostens · arXiv:2607.17120

The Generator Is the Tracker: multi-object tracking by painting persistent identity colours

Trackers keep identity outside the video: track buffers, motion models, appearance embeddings. A video generator already has to know which pixels belong to the same person from one frame to the next, or the people in its videos would not stay themselves. We ask whether that knowledge can be read out directly, by asking the generator to paint each person a colour that stays the same over time.

We fine-tune a 22B text-to-video model (LTX-2.3) to turn an RGB clip into an ID-map clip: every person painted a flat, distinct colour on black, same colour, same person. Long videos are generated as chained windows, and a short continuation fine-tune teaches the model to extend a colouring it is handed. Tracks are read off the generated pixels. There is no separately trained detector, no motion model and no association module.

A full DanceTrack validation video (dancetrack0065, 702 frames, five dancers in near-identical costumes) and the ID-map video the system generated from it, in 18 chained windows of 49 frames. Each colour is one identity, carried only by the generated pixels from window to window. The five main colours persist, with short-lived fragments: over the whole video the chain scores IDF1 0.47 with 30 predicted track ids for 5 people. This is the development video, on which design decisions were made. Input video: DanceTrack (Sun et al., 2022).
40.3 HOTAon the DanceTrack test server, from generated video alone, in one frozen submission
44.1 AssAassociation above every tracker of the 2020–22 benchmark suite; detection (DetA 37.6) is the deficit
34.8 vs 15.9–16.8validation HOTA with chaining and continuation training together, against either one alone (18.2 with neither)
42 %of gap events the chain held on both sides (86 of 205) got their own colour back, against 23–24 % for classical associators on the same boxes

The idea: identity as colour

Tracking has two parts. Binding: the same region must get the same label from frame to frame. State: a person who is hidden must still be remembered when they come back. Conventional trackers solve both with machinery beside the video. A video diffusion model attends to a whole clip at once, so binding is something it does anyway; the question is whether it can be made to show it, and to carry state from one clip to the next.

The interface is an image: each person painted with one of the 47 identity colours of a 48-colour palette (index 0, black, is background), chosen to maximise pairwise separation in CIELAB (closest pair ΔE 26.4). Decoding needs no model: every pixel snaps to its nearest palette colour, each colour's connected regions give a box, and the palette index is the track id. The codec round-trips masks at IoU ≥ 0.99 with every identity preserved, and ID-maps are only ever stored losslessly, because compression smears flat colours.

The 47 identity colours. Index 0, black, is background. In training each ground-truth identity is given one colour for the whole video; at inference the model chooses colours itself.

How it works

System overview: the RGB window and the cleaned tail of the previous window enter the video diffusion transformer, which generates an ID-map window; the codec decodes it into tracks and the cleaned map becomes the prefix for the next window.
Each 49-frame window is generated from the aligned RGB window and a 9-frame prefix: the decoded, corrected and re-encoded tail of the previous window. The cleaned map gives the tracks and seeds the next window.

Translate

LTX-2.3 is fine-tuned with its native in-context LoRA: the RGB clip enters as clean reference tokens with positions aligned to the target, so attention learns a positionally locked translation from RGB to ID-map, aligned at the level of latent tokens (32×32-pixel patches). Rank 32, 0.74 % of the parameters, on 777 clips from the 40 DanceTrack training videos with silhouettes from SAM 3. The whole fine-tune fits on one 80 GB A100.

Chain

A video is split into 49-frame windows at stride 40. Each window's first 9 frames are fixed to the previous window's tail after decode, correct, re-encode: snapped to palette colours, speckles removed, encoded again. The condition is always a valid ID-map, never raw generation, and it is the only state the system has.

Continue

Chaining at inference alone fails: the model was trained to invent colourings, not to continue one, so past the overlap it repaints. A short continuation fine-tune hands it the target's own first 9 frames half of the time. On the development video full-video IDF1 rises from 0.17 to 0.47.

Where the identity comes from

Identity could in principle come from linking windows after the fact. To separate the two ingredients we ran a 2×2 factorial on all 25 full-length validation videos with identical window geometry (640 windows per cell): the checkpoint before or after continuation training, run as independent windows (linked afterwards) or as a chain.

The same development video, three ways, raw generated colours with no linking. Independent windows repaint the dancers at almost every window boundary. Chained without continuation training, the colours survive each boundary, because the prefix fixes them, but drift inside the windows. Chained after continuation training they persist far longer, though not perfectly: full-video IDF1 0.47 on this video, against 0.17 without continuation training. Input video: DanceTrack (Sun et al., 2022).
translation-only checkpointafter continuation training
Validation HOTA by generation mode and checkpoint: independent windows 18.2 and 15.9, chained 16.8 and 34.8. 01020 3040 HOTA, 25 validation videos Independent windows, translation-only (B): 18.2 HOTA 18.2 Independent windows, after continuation training (D): 15.9 HOTA 15.9 Chained, translation-only (C): 16.8 HOTA 16.8 Chained, after continuation training (A, ours): 34.8 HOTA 34.8 Independent windows Chained windows

Neither ingredient helps on its own; together they double HOTA. Chaining is worth +18.9 HOTA [15.2, 22.9] for the continuation-trained checkpoint (24 of 25 videos) and nothing for the translation-only one. Re-linking the chain's own windows afterwards changes nothing (34.7), so the colours already carry everything overlap linking can recover.

On identical boxes the trained chain's colours beat classical association. We drop the colour ids from the chain's own boxes and re-associate them with a Kalman/IoU tracker and with a DINOv3 appearance-and-motion bank. Detection is then the same, and association falls from 38.3 AssA to 21.6 and 24.3, with 3,079 and 2,408 identity switches against 1,950. Fast dance motion breaks frame-to-frame geometry and identical costumes defeat appearance; the generated colour stays put. Without continuation training (cell C) the colours are no better than Kalman/IoU on the same boxes.

CellCheckpointGenerationHOTADetAAssAIDF1AssA, same boxes: Kalman/IoUDINOv3 bank
A (ours)continuation-trainedchain34.832.338.338.321.624.3
Ctranslation-onlychain16.828.710.115.510.412.6
Dcontinuation-trainedindependent15.917.415.115.710.711.2
Btranslation-onlyindependent18.219.717.717.312.714.3

DanceTrack validation, TrackEval, sampling seed 0. Validation was also the development set. Independent windows are linked by a pinned overlap-IoU associator, selected on the 5-person development video; part of their deficit is detection, because that associator merges crowded, identically dressed dancers. Read by raw colour instead, independent windows keep their detections, and at the same continuation-trained checkpoint the chain still leads by +19.8 HOTA in all 25 videos. In two further seeds the chain scores 37.1 and 37.8 HOTA, and its lead on identical boxes holds in every seed.

On the test server

One frozen submission to the official DanceTrack server, with no test-side iteration, scores 40.3 HOTA. That is not state of the art: 2025–26 specialists reach 66–76. What no other entry shows is the profile. To our knowledge it is the only submission that runs neither a box detector nor an association or propagation module, and the only one whose association exceeds its detection.

Scatter of AssA against DetA on the DanceTrack test set. Every detector-based tracker lies below the diagonal AssA equals DetA; ours lies above it, at DetA 37.6 and AssA 44.1.
DanceTrack test: every detector-based tracker sits below the AssA = DetA diagonal; ours (star) is the only point above it. Dashed curves are iso-HOTA.
MethodYearDetectorHOTADetAAssAIDF1
ByteTrack2022separate47.771.032.153.9
MOTR2022end-to-end54.273.540.251.5
OC-SORT2023separate55.180.338.354.6
Deep OC-SORT2023separate61.382.245.861.5
MOTRv2+2023separate + end-to-end69.983.059.071.7
SAM2MOT2026separate + SAM 275.879.772.283.9
Ours (generation only)2026none40.337.644.145.2

A selection of the paper's Table 1, which lists 21 trackers with their sources. +Uses CrowdHuman for its proposal detector. AssA is computed over matched detections only, so a tracker that finds the easy people and keeps them whole scores high AssA at low DetA. Our recall is low and lowest in crowds; the like-for-like comparison of association is the one on identical boxes above.

Keeping a hidden person's colour

The hardest identity problem is a person who disappears, behind someone else or out of frame, and comes back. We mined every ground-truth gap of at least 9 frames in the 25 validation videos: 383 events, 360 occlusions and 23 exits. Of the 205 events the chain tracked on both sides of the gap, it gave 86 (42 %) back their own colour, 22 % of all 383 events. A Kalman/IoU tracker and the appearance bank, run on the same boxes and events, managed 47 and 50.

dancetrack0019, at half speed. The ring follows one dancer; it is dashed while the dancer is hidden (its position then is interpolated). The dancer's colour is painted through the gap and comes back with them. This is one of the paper's qualitative examples, a success; 119 of the 205 events are failures. Input video: DanceTrack (Sun et al., 2022).
Gap length (frames)9–2425–4849–150All
Chain (ours)60/12719/507/2686/205 (42 %)
Kalman/IoU on the same boxes33/12712/502/2647/205 (23 %)
Appearance + motion bank on the same boxes39/12710/501/2650/205 (24 %)

Events tracked before and re-detected after the gap, on the boxes TrackEval scores. "All" includes the 2 events longer than 150 frames, which no system re-acquired.

The window is 49 frames long, yet gaps longer than that still come back 7 times in 26. The colour is carried through the gap: in all 86 recoveries of the reported seed (and all but one in each of the other two seeds) the pre-gap colour appears in at least one hidden frame. But it is also painted in 110 of the 119 failures, so carry-over is necessary, not sufficient. What tends to separate the two is where the colour stays: in the median recovery it covers the whole gap and stays near the hidden person (median IoU 0.35 with their interpolated position), while in failures it covers less of the gap (0.90) and drifts away (0.19). An identity the model stops painting is lost.

A second benchmark

To test the recipe rather than one model, we re-ran it end to end on SportsMOT (basketball, football and volleyball; validation only), re-tuning nothing. On the 45 validation videos the colours reach 26.9 HOTA, and the ordering on identical boxes repeats: AssA 28.2 for the colours against 20.7 for the appearance bank and 17.1 for Kalman/IoU. Long-gap re-acquisition does not carry over, for an instructive reason: 369 of the 374 gap events are players leaving the frame, and a person outside the frame cannot be kept alive in pixels.

Limits

Two scatter plots over the 25 validation videos: AssA and DetA both fall as the number of people in the video grows.
Per validation video: association (left) and detection (right) against the number of people in the video.

How we got here

The project started with image generators, and their failure is what pointed to video. Gemini's image model, used with no training and a memory canvas, kept colours from one frame to the next but lost people across long gaps. A LoRA-tuned Cosmos3-Nano became a good per-frame painter (validation mask IoU 0.585), but three ways of making it carry identity between frames, through the previous map, a text-slot bank and a cross-attention bank, did not bind identity. The diagnosis was a weak colour-binding signal in the generator's training loss, plus drift whenever the generator was chained on its own outputs. A video model, which attends to a whole window at once, was the next bet rather than a proven remedy, and continuation training turned out to be the piece that made chaining work. These trials are kept in image_generators/ with their write-up.

Code and records

FolderWhat is in it
core/The shared library (gentrack package and tests): ID-map codec and palette, MOT format, metrics, and the invariant tests (codec round trip first)
video_generator/The paper's method: training pairs, LTX-2.3 fine-tuning, chained generation, evaluation, baselines, the test-server entry
datasets/Download scripts for DanceTrack, SportsMOT and MOT17, SAM 3 silhouette labels, occlusion mining
experiments/Job records: the parameters of the paper's runs
paper/The ICLR and arXiv versions of the paper, the scripts that build every table and figure, and the run records they read
image_generators/The earlier image-generator trials and their findings
pip install -e ".[dev]"
pytest -x                 # core/tests: the codec round-trip invariant runs first

Training and generation run on Databricks with one 80 GB A100; video_generator/README.md gives the pipeline in order. The paper's experiments used about 224 A100-hours, not counting the earlier image-model pilots.

Licences and data

ItemTerms
Code in this repositoryMIT
LoRA checkpoints and generated ID-mapsNon-commercial research only, following the datasets they come from; the LoRA is a derivative of LTX-2.3 and also subject to the LTX-2 Community License Agreement
LTX-2.3 base modelLightricks' licence for LTX-2; see its model card
SAM 3 (silhouette labels)SAM License
DanceTrack, SportsMOTTheir non-commercial research licences. The videos on this page show one full DanceTrack validation video (dancetrack0065) and a 107-frame excerpt of another (dancetrack0019) for reference, credited to the dataset's authors (Sun et al., 2022); we do not redistribute the dataset. Get it from its authors.

Citation

If you use this work, please cite the paper.

@article{yang2026generator,
  title   = {The generator is the tracker: Multi-object tracking by painting
             persistent identity colours},
  author  = {Yang, Haiyu and Hostens, Miel},
  journal = {arXiv preprint arXiv:2607.17120},
  year    = {2026},
  doi     = {10.48550/arXiv.2607.17120}
}

The numbers on this page follow the November 2026 revision of the paper (source in paper/, for both the ICLR version and arXiv v2); arXiv v1 (July 2026) predates it.

Data and models this work builds on

@inproceedings{sun2022dancetrack,
  title     = {{DanceTrack}: Multi-object tracking in uniform appearance and diverse motion},
  author    = {Sun, Peize and Cao, Jinkun and Jiang, Yi and Yuan, Zehuan and Bai, Song and
               Kitani, Kris and Luo, Ping},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2022}
}