These are the contracts between the tools in What to build. Times are seconds of song time unless noted. Keep the names, and any tool written against them, by you or by an agent, will fit the others.
The project folder
<film>/
project.json the settings (below)
BRIEF.md the director's ask, verbatim
DECISIONS.md dated decisions, verbatim; later entries supersede earlier ones
RUNS.md ids and arguments of long or delegated runs, for resuming
STATUS.md one status, rewritten in place
ledger.jsonl one line per paid call
audio/song.wav the master as delivered (44.1 kHz stereo)
audio/stems/ vocals.wav, drums.wav, bass.wav, other.wav
audio/windows/<clip>.wav|.mp3 clean vocal windows for sung clips
lyrics/lyrics.txt the sung words, spelled as sung, one line per lyric line
lyrics/display.txt the on-screen tokens, one line per lyric line
analysis/words.json timed words
analysis/beats.json the beat map and sections
analysis/hits.json the sound events
analysis/report.md a readable summary and the ear-check list
board/shots.json the storyboard: the source of truth for the edit
board/audit.json the audit's results
board/board.pdf the board sheet for review
plates/prompts.md the plate prompt book
plates/cands/<plate>/<k>.jpg every candidate
plates/picks.json {plate: {file, cand, shots}}
clips/<clip>.json one file per clip (below)
clips/<clip>/take-N.mp4 the takes, with take-N-sync.png beside them
edit/edl.json the edit list
engine/ the renderer's per-film data and chapter code
render/frames/ frame i = song time i/24 s
out/ review.mp4, master.mp4, vertical.mp4The lyric files share one parser. Blank lines and every line starting with [ (section tags, notes) are skipped, and
the Nth remaining line of display.txt shows the Nth remaining line of lyrics.txt.
project.json
{
"name": "my-song", "title": "My Song", "scale": "quick | studio | full",
"song": "audio/song.wav", "fps": 24, "size": [1920, 1080],
"lead": {
"canon": "<her identity text, full form>",
"far": "<the short form for full length and wider>"
},
"look": {"mode": "native | clean-grade | print", "accent": "#D97757",
"fonts": {"lyric": "…", "micro": "…", "display": "…"}, "hero_plate": null},
"mj": {"params": "--ar 16:9 --v 8.2 --raw --hd", "profile": "<your own code; never published>"},
"models": {"video": "bytedance/seedance-2.5", "edit": "google/gemini-3-pro-image"},
"budget": {"openrouter_usd": 100, "higgsfield_credits": 1000, "confirm_over_usd": 5,
"motion_s": 200, "frozen": false},
"banned": ["words", "never", "to", "use"],
"audio": {"tempo_range": [129, 136], "downbeat_phase": null,
"hits": {"add": [[170.7, 0.8]], "drop": []},
"name_map": {"<heard>": ["<sheet>", "<words>"]}},
"board": {"lead_min": 0.7, "max_clips": 45},
"gates": {"board_approved": false, "prompts_approved": false, "v1_approved": false}
}budget.frozen is the stop switch: while it is true, nothing may spend. banned is checked by the board audit and
belongs in every agent's prompt.
Timed words
analysis/words.json:
{"words": [{"i": 0, "w": "ladies", "line": 0, "t0": 9.32, "t1": 9.64, "conf": 0.91, "src": "w2v"}],
"lines": [{"line": 0, "text": "Ladies. Gentlemen. Agents.", "t0": 9.32, "t1": 10.72}],
"gaps": [{"t0": 18.8, "t1": 26.3, "after_word": 12}]}src is w2v, whisper, interp or manual. A manual word carries a note with the evidence, and re-runs keep it.
The beat map
analysis/beats.json. The tempo varies; never derive beats from one BPM.
{"duration": 306.4,
"beats": [0.452, 0.908],
"downbeats": [0.452, 2.276],
"tempo": [{"t": 0.0, "bpm": 131.55}, {"t": 5.0, "bpm": 131.6}],
"downbeat_phase": 0, "downbeat_method": "harmony | kick | manual",
"sections": [{"name": "intro", "t0": 0.0, "t1": 18.96, "energy": 0.31, "bar0": 0, "bar1": 9,
"source": "lyrics-tag | set | heuristic"}]}Bars count from 0: bar k starts at downbeats[k]. Eighth notes are the beats plus the midpoints between them.
Hits
analysis/hits.json:
{"kick": [0.45], "snare": [0.91], "hat": [0.68],
"hits": [{"t": 10.72, "score": 0.81}],
"risers": [{"t0": 9.83, "t1": 10.72, "land": 10.72}],
"fills": [{"t0": 42.3, "t1": 44.1}],
"vocal_entries": [9.32],
"env24": {"drums": [0.0], "vocals": [0.0], "bass": [0.0]}}Display tokens
lyrics/display.txt has one line per lyric line. TEXT|n shows TEXT for the next n sung words, and n defaults to 1.
Ladies. Gentlemen. Agents.
+25%|3 =|1 −17%|2
Feel the AGI|5The markers are *x* accent, ~x~ strike-through and _x_ mono; |0 rides on the next token. A line must consume
exactly its aligned words.
The board
board/shots.json, top level:
{"version": 1, "duration": 306.4, "chapters": [], "shots": [], "plates": []}A chapter is {id, name, t0, t1, sections, accent, set}. A plate is {plate, set, prompt, aspect, used_by: [shot ids]}.
| Field | Type | Rule |
|---|---|---|
id |
"s001" … |
stable once the director has seen them |
t0, t1 |
seconds | on beats (within 0.03 s) or declared hits; shots tile the song |
section |
name | a section of the beat map |
lines |
[int] | the lyric lines whose words fall in the shot |
set |
text | the place, concretely |
lead_on_screen |
bool | true only when she is recognisable |
framing |
ECU · CU · MCU · MS · MWS · WS · FS · EWS · OTS | |
action |
text | what happens and the reaction, with an end state |
camera |
text | one move, tied to a sound |
motion_tier |
still · plate3d · motion · sung | |
plate |
key or null | null only for pure motion design |
clip |
id or null | for motion and sung shots |
prompt |
text | the plate's prompt, framing first, ending "no text, no letters, no logos" |
type |
[{line, mode, text, placement}] |
mode: subtitle · coverline · masthead · stack · mass |
devices |
[{kind, anchor, params, key, line}] |
key: W · B · H · F · T; the explaining graphic is one of these |
treatment |
text or null | a 2.5D family; never the same on two 2.5D shots in a row |
transition_in |
text | cut, or the motivated device |
chrome |
text | what the always-on devices show here |
accent |
hex | the chapter's accent |
hit |
seconds | optional: a declared hit this shot starts on |
notes |
text | the director's notes, verbatim |
ESCAPE VELOCITY's first shot, in this schema. The prompt is written with today's canon text; the film's own predates the adult wording.
{"id": "s001", "t0": 0.0, "t1": 2.111, "section": "intro", "lines": [],
"set": "the back seat of a driverless robotaxi on US-101 at night", "lead_on_screen": true, "framing": "CU",
"action": "Frame 1 is her face in the dark back seat, eyes closed. A red lamp slides across her face left to right and lights the clay streak and the star clip. On the song's first hit (1.12) her eyes open straight into the lens and stay there.",
"camera": "locked, very slow push in (3%)", "motion_tier": "motion", "plate": "waymo-cu", "clip": "SD01",
"prompt": "close-up of a 28-year-old Caucasian American woman with a grown-up angular face, defined cheekbones and a strong jawline, pale skin and light freckles, a glossy black blunt jaw-length bob with heavy straight bangs and one clay-orange streak through the bangs, a small flat clay-orange eight-pointed star hair clip, a thin headset microphone at her cheek, sitting upright alone in the back seat of a driverless robotaxi at night, her face in the centre of the frame, eyes closed, a single red lamp sliding across one cheek through the side window, dark leather seat, the lower third of the frame dark and empty, blue-black night, fog, one small red lamp as the only warm note, fine silver-gelatin grain, cinematic 35mm film still, no text, no letters, no logos",
"type": [],
"devices": [
{"kind": "flap", "anchor": {"slot": "bottom"}, "key": "F",
"params": {"rows": [{"text": "18 MONTHS TO ESCAPE THE PERMANENT UNDERCLASS", "prev": "", "at": 0.0}], "cw": 36, "gap": 3}},
{"kind": "sheet", "anchor": {"box": "a face"}, "key": "B",
"params": {"title": "EXPRESSION · MODEL 01 · LIVE", "rows": [["CALM", ".98"], ["WORRY", ".02"], ["THRESHOLD · WORRY > 0.50", "NOT MET", "hot"]]}},
{"kind": "cv.bracket", "anchor": {"box": "a face"}, "key": "B", "params": {"label": "FACE · MODEL 01", "conf": 0.99}}],
"treatment": null, "transition_in": "cut",
"chrome": "the split-flap board across the lower third; clatters from frame 1, settles by 1.5 s",
"accent": "#D97757",
"notes": "The hook. At 1.0 s: her face, the streak and a readable board line on a muted phone."}The idea behind the explaining graphic: a face that is not worried under the year's most-quoted threat, and an expression meter that says so in numbers.
Clips
clips/<clip>.json:
{"id": "W02", "shot": "s031", "kind": "sung",
"status": "draft | locked | submitted | delivered | declined | failed",
"provider": "openrouter | higgsfield", "model": "bytedance/seedance-2.5",
"start": "plates/cands/siren-cu/2.jpg",
"song_t0": 8.56, "needed": [8.85, 12.4], "words": "Why would I deceive you?",
"window": {"t0": 8.56, "t1": 12.56, "mode": "clean", "file": "audio/windows/W02.wav", "url": "https://…"},
"body": {"prompt": "…", "duration": 4, "resolution": "720p", "aspect_ratio": "16:9"},
"takes": [{"n": 1, "provider": "higgsfield", "job": "<provider job id; never published>", "status": "delivered",
"file": "clips/W02/take-1.mp4", "credits": 28,
"sync": {"lag_s": 0.7063, "ncc": 0.22, "mode": "copy", "synced": [[8.85, 10.25]],
"diverge_at": 10.31, "cut_at": 10.2458}}],
"covers": ["W02-c1"],
"parent": null,
"why": "cover for <root>: it diverges from our audio at song <t>; this window starts in the word gap at <t0>"}The time maps: song time = song_t0 + clip time − lag_s, and clip time = song time − song_t0 + lag_s.
The edit list
edit/edl.json maps each shot to its sources:
{"s001": {"segs": [{"t0": 0.0, "clip": "SD01-t1", "in": 0.0}]},
"s031": {"segs": [{"t0": 8.85, "clip": "W02-t1", "in": 0.996}, {"t0": 10.2458, "clip": "W02-c1-t1", "in": 0.31}]},
"s002": {"plate": "p002"}}in is the clip time at the segment's start: song time − song_t0 + lag_s.
The logs
DECISIONS.md:
## <date> · director · board
"s034-s036 could use more creativity, kind of a boring moment."
→ s034–s036 rethought (replay loop, tally marks).
## <date> · director · supersedes <date> "keep the halftone"
"keep it clean so it feels like imax, high res rather than dithered"ledger.jsonl, one paid call per line. A real cost is filled in from the provider; the estimate is never quoted as
spend.
{"ts": "…", "provider": "openrouter", "model": "bytedance/seedance-2.5", "kind": "video", "units": 5,
"est_usd": 1.16, "real_usd": 1.155, "request_id": "<private>", "status": "delivered"}RUNS.md gets one line per run: the run id, the script path, the arguments file, what it does and when it started.
A review
What a reviewer returns per chapter. The verdict is computed from it in code.
{"owner": 7.5, "craft": 7, "sync": 8, "story": 7, "pass": true,
"fixes": [{"shot": "s045", "time": 101.2, "severity": "blocking",
"problem": "the coverline sits over her mouth for 6 frames",
"fix": "move the column right (x 196→250) so it runs 40–80 px under her hair"}],
"checks": {"lyric_gate": "132/132 tokens on screen at their time (41.2s)", "sheets": 28,
"ms_per_frame": 820, "flash_max_per_s": 2}}owner is the director's taste, and it counts double.