Claudia.gallery

Data formats

The files every tool reads and writes: the project folder, the timed words, the beat map, the hits, the display tokens, the board, the clips and their sync, the edit list, and the logs.

Markdown  ·  MCP: get_guide("formats")  ·  3 min · 13 sections

These are the contracts between the tools in What to build. Times are seconds of song time unless noted. Keep the names, and any tool written against them, by you or by an agent, will fit the others.

The project folder

<film>/
  project.json            the settings (below)
  BRIEF.md                the director's ask, verbatim
  DECISIONS.md            dated decisions, verbatim; later entries supersede earlier ones
  RUNS.md                 ids and arguments of long or delegated runs, for resuming
  STATUS.md               one status, rewritten in place
  ledger.jsonl            one line per paid call
  audio/song.wav          the master as delivered (44.1 kHz stereo)
  audio/stems/            vocals.wav, drums.wav, bass.wav, other.wav
  audio/windows/<clip>.wav|.mp3   clean vocal windows for sung clips
  lyrics/lyrics.txt       the sung words, spelled as sung, one line per lyric line
  lyrics/display.txt      the on-screen tokens, one line per lyric line
  analysis/words.json     timed words
  analysis/beats.json     the beat map and sections
  analysis/hits.json      the sound events
  analysis/report.md      a readable summary and the ear-check list
  board/shots.json        the storyboard: the source of truth for the edit
  board/audit.json        the audit's results
  board/board.pdf         the board sheet for review
  plates/prompts.md       the plate prompt book
  plates/cands/<plate>/<k>.jpg    every candidate
  plates/picks.json       {plate: {file, cand, shots}}
  clips/<clip>.json       one file per clip (below)
  clips/<clip>/take-N.mp4 the takes, with take-N-sync.png beside them
  edit/edl.json           the edit list
  engine/                 the renderer's per-film data and chapter code
  render/frames/          frame i = song time i/24 s
  out/                    review.mp4, master.mp4, vertical.mp4

The lyric files share one parser. Blank lines and every line starting with [ (section tags, notes) are skipped, and the Nth remaining line of display.txt shows the Nth remaining line of lyrics.txt.

project.json

{
  "name": "my-song", "title": "My Song", "scale": "quick | studio | full",
  "song": "audio/song.wav", "fps": 24, "size": [1920, 1080],
  "lead": {
    "canon": "<her identity text, full form>",
    "far": "<the short form for full length and wider>"
  },
  "look": {"mode": "native | clean-grade | print", "accent": "#D97757",
           "fonts": {"lyric": "…", "micro": "…", "display": "…"}, "hero_plate": null},
  "mj": {"params": "--ar 16:9 --v 8.2 --raw --hd", "profile": "<your own code; never published>"},
  "models": {"video": "bytedance/seedance-2.5", "edit": "google/gemini-3-pro-image"},
  "budget": {"openrouter_usd": 100, "higgsfield_credits": 1000, "confirm_over_usd": 5,
             "motion_s": 200, "frozen": false},
  "banned": ["words", "never", "to", "use"],
  "audio": {"tempo_range": [129, 136], "downbeat_phase": null,
            "hits": {"add": [[170.7, 0.8]], "drop": []},
            "name_map": {"<heard>": ["<sheet>", "<words>"]}},
  "board": {"lead_min": 0.7, "max_clips": 45},
  "gates": {"board_approved": false, "prompts_approved": false, "v1_approved": false}
}

budget.frozen is the stop switch: while it is true, nothing may spend. banned is checked by the board audit and belongs in every agent's prompt.

Timed words

analysis/words.json:

{"words": [{"i": 0, "w": "ladies", "line": 0, "t0": 9.32, "t1": 9.64, "conf": 0.91, "src": "w2v"}],
 "lines": [{"line": 0, "text": "Ladies. Gentlemen. Agents.", "t0": 9.32, "t1": 10.72}],
 "gaps":  [{"t0": 18.8, "t1": 26.3, "after_word": 12}]}

src is w2v, whisper, interp or manual. A manual word carries a note with the evidence, and re-runs keep it.

The beat map

analysis/beats.json. The tempo varies; never derive beats from one BPM.

{"duration": 306.4,
 "beats": [0.452, 0.908],
 "downbeats": [0.452, 2.276],
 "tempo": [{"t": 0.0, "bpm": 131.55}, {"t": 5.0, "bpm": 131.6}],
 "downbeat_phase": 0, "downbeat_method": "harmony | kick | manual",
 "sections": [{"name": "intro", "t0": 0.0, "t1": 18.96, "energy": 0.31, "bar0": 0, "bar1": 9,
               "source": "lyrics-tag | set | heuristic"}]}

Bars count from 0: bar k starts at downbeats[k]. Eighth notes are the beats plus the midpoints between them.

Hits

analysis/hits.json:

{"kick": [0.45], "snare": [0.91], "hat": [0.68],
 "hits": [{"t": 10.72, "score": 0.81}],
 "risers": [{"t0": 9.83, "t1": 10.72, "land": 10.72}],
 "fills": [{"t0": 42.3, "t1": 44.1}],
 "vocal_entries": [9.32],
 "env24": {"drums": [0.0], "vocals": [0.0], "bass": [0.0]}}

Display tokens

lyrics/display.txt has one line per lyric line. TEXT|n shows TEXT for the next n sung words, and n defaults to 1.

Ladies. Gentlemen. Agents.
+25%|3 =|1 −17%|2
Feel the AGI|5

The markers are *x* accent, ~x~ strike-through and _x_ mono; |0 rides on the next token. A line must consume exactly its aligned words.

The board

board/shots.json, top level:

{"version": 1, "duration": 306.4, "chapters": [], "shots": [], "plates": []}

A chapter is {id, name, t0, t1, sections, accent, set}. A plate is {plate, set, prompt, aspect, used_by: [shot ids]}.

Field Type Rule
id "s001" … stable once the director has seen them
t0, t1 seconds on beats (within 0.03 s) or declared hits; shots tile the song
section name a section of the beat map
lines [int] the lyric lines whose words fall in the shot
set text the place, concretely
lead_on_screen bool true only when she is recognisable
framing ECU · CU · MCU · MS · MWS · WS · FS · EWS · OTS
action text what happens and the reaction, with an end state
camera text one move, tied to a sound
motion_tier still · plate3d · motion · sung
plate key or null null only for pure motion design
clip id or null for motion and sung shots
prompt text the plate's prompt, framing first, ending "no text, no letters, no logos"
type [{line, mode, text, placement}] mode: subtitle · coverline · masthead · stack · mass
devices [{kind, anchor, params, key, line}] key: W · B · H · F · T; the explaining graphic is one of these
treatment text or null a 2.5D family; never the same on two 2.5D shots in a row
transition_in text cut, or the motivated device
chrome text what the always-on devices show here
accent hex the chapter's accent
hit seconds optional: a declared hit this shot starts on
notes text the director's notes, verbatim

ESCAPE VELOCITY's first shot, in this schema. The prompt is written with today's canon text; the film's own predates the adult wording.

{"id": "s001", "t0": 0.0, "t1": 2.111, "section": "intro", "lines": [],
 "set": "the back seat of a driverless robotaxi on US-101 at night", "lead_on_screen": true, "framing": "CU",
 "action": "Frame 1 is her face in the dark back seat, eyes closed. A red lamp slides across her face left to right and lights the clay streak and the star clip. On the song's first hit (1.12) her eyes open straight into the lens and stay there.",
 "camera": "locked, very slow push in (3%)", "motion_tier": "motion", "plate": "waymo-cu", "clip": "SD01",
 "prompt": "close-up of a 28-year-old Caucasian American woman with a grown-up angular face, defined cheekbones and a strong jawline, pale skin and light freckles, a glossy black blunt jaw-length bob with heavy straight bangs and one clay-orange streak through the bangs, a small flat clay-orange eight-pointed star hair clip, a thin headset microphone at her cheek, sitting upright alone in the back seat of a driverless robotaxi at night, her face in the centre of the frame, eyes closed, a single red lamp sliding across one cheek through the side window, dark leather seat, the lower third of the frame dark and empty, blue-black night, fog, one small red lamp as the only warm note, fine silver-gelatin grain, cinematic 35mm film still, no text, no letters, no logos",
 "type": [],
 "devices": [
   {"kind": "flap", "anchor": {"slot": "bottom"}, "key": "F",
    "params": {"rows": [{"text": "18 MONTHS TO ESCAPE THE PERMANENT UNDERCLASS", "prev": "", "at": 0.0}], "cw": 36, "gap": 3}},
   {"kind": "sheet", "anchor": {"box": "a face"}, "key": "B",
    "params": {"title": "EXPRESSION · MODEL 01 · LIVE", "rows": [["CALM", ".98"], ["WORRY", ".02"], ["THRESHOLD · WORRY > 0.50", "NOT MET", "hot"]]}},
   {"kind": "cv.bracket", "anchor": {"box": "a face"}, "key": "B", "params": {"label": "FACE · MODEL 01", "conf": 0.99}}],
 "treatment": null, "transition_in": "cut",
 "chrome": "the split-flap board across the lower third; clatters from frame 1, settles by 1.5 s",
 "accent": "#D97757",
 "notes": "The hook. At 1.0 s: her face, the streak and a readable board line on a muted phone."}

The idea behind the explaining graphic: a face that is not worried under the year's most-quoted threat, and an expression meter that says so in numbers.

Clips

clips/<clip>.json:

{"id": "W02", "shot": "s031", "kind": "sung",
 "status": "draft | locked | submitted | delivered | declined | failed",
 "provider": "openrouter | higgsfield", "model": "bytedance/seedance-2.5",
 "start": "plates/cands/siren-cu/2.jpg",
 "song_t0": 8.56, "needed": [8.85, 12.4], "words": "Why would I deceive you?",
 "window": {"t0": 8.56, "t1": 12.56, "mode": "clean", "file": "audio/windows/W02.wav", "url": "https://…"},
 "body": {"prompt": "…", "duration": 4, "resolution": "720p", "aspect_ratio": "16:9"},
 "takes": [{"n": 1, "provider": "higgsfield", "job": "<provider job id; never published>", "status": "delivered",
            "file": "clips/W02/take-1.mp4", "credits": 28,
            "sync": {"lag_s": 0.7063, "ncc": 0.22, "mode": "copy", "synced": [[8.85, 10.25]],
                     "diverge_at": 10.31, "cut_at": 10.2458}}],
 "covers": ["W02-c1"],
 "parent": null,
 "why": "cover for <root>: it diverges from our audio at song <t>; this window starts in the word gap at <t0>"}

The time maps: song time = song_t0 + clip time − lag_s, and clip time = song time − song_t0 + lag_s.

The edit list

edit/edl.json maps each shot to its sources:

{"s001": {"segs": [{"t0": 0.0, "clip": "SD01-t1", "in": 0.0}]},
 "s031": {"segs": [{"t0": 8.85, "clip": "W02-t1", "in": 0.996}, {"t0": 10.2458, "clip": "W02-c1-t1", "in": 0.31}]},
 "s002": {"plate": "p002"}}

in is the clip time at the segment's start: song time − song_t0 + lag_s.

The logs

DECISIONS.md:

## <date> · director · board
"s034-s036 could use more creativity, kind of a boring moment."
→ s034–s036 rethought (replay loop, tally marks).

## <date> · director · supersedes <date> "keep the halftone"
"keep it clean so it feels like imax, high res rather than dithered"

ledger.jsonl, one paid call per line. A real cost is filled in from the provider; the estimate is never quoted as spend.

{"ts": "…", "provider": "openrouter", "model": "bytedance/seedance-2.5", "kind": "video", "units": 5,
 "est_usd": 1.16, "real_usd": 1.155, "request_id": "<private>", "status": "delivered"}

RUNS.md gets one line per run: the run id, the script path, the arguments file, what it does and when it started.

A review

What a reviewer returns per chapter. The verdict is computed from it in code.

{"owner": 7.5, "craft": 7, "sync": 8, "story": 7, "pass": true,
 "fixes": [{"shot": "s045", "time": 101.2, "severity": "blocking",
            "problem": "the coverline sits over her mouth for 6 frames",
            "fix": "move the column right (x 196→250) so it runs 40–80 px under her hair"}],
 "checks": {"lyric_gate": "132/132 tokens on screen at their time (41.2s)", "sheets": 28,
            "ms_per_frame": 820, "flash_max_per_s": 2}}

owner is the director's taste, and it counts double.