goldeneye-pc-port/docs/dev/release-dropin-rom-plan.md

# Release plan — "drop in your ROM and play"

> **Historical record — do not act on status claims in this file.** Part A shipped as the NTSC-U drop-in ROM flow (PR #84, issue #6). The remaining Part B pieces are tracked in the roadmap. Current open work and status: [`docs/ROADMAP.md`](../ROADMAP.md) (2026-09-29).

Status: proposal / not yet scheduled. Owner: TBD.

## Goal

An end user downloads the GitHub release zip, unzips it, drops their
legally-owned GoldenEye 007 N64 ROM (`.z64`, big-endian) into a folder,
double-clicks the exe, and plays. No Python, no MSYS2, no manual
asset-extraction, no command line.

## Where we are today

The Windows bundle produced by `tools_pc/bundle-win.sh` (and uploaded by the
`windows-build` CI job) already contains only: the engine exe, its MinGW /
SDL2 / zlib DLLs, a README, and licenses. It deliberately ships **no** ROM and
**no** game assets. That part is fine.

The gap is everything the runtime needs *alongside* the ROM before a level
will render:

| Dependency | Produced today by | Consumed at runtime by |
|---|---|---|
| `port/src/romassets_<region>.s` — absolute cart-address symbols for every obseg/ramrom/music asset + `ge007.ld` segment markers | `python3 scripts/gen_romassets.py <u\|e\|j>`, compiled **into** the exe | linker; `&symbol` + `romCopy()` in game code |
| `data/pcmodels-<region>/{pcmodels.bin,manifest.csv}` (~1.3 MB) — every model file re-laid-out to PC struct layout, RZ-compressed | `python3 tools_pc/d43_emit.py <region>` | `port/src/pcmodels.c` (`pcmodelsReserveSize`/`LoadSidecars`/`PatchTable`) |
| `data/pccg-<region>/{pccg.bin,manifest.csv}` (~3.6 MB) — every stage `bg/*.seg` + `Tbg_*_stanZ` re-laid-out to PC layout | `python3 tools_pc/d69_emit.py <region>` | `port/src/pccg.c` |
| ROM present at `data/ge007.<region>.z64` (or `baserom.<r>.z64`) | user copies it | `port/src/romdata.c` maps it at `0x10000000` |

Without the two sidecar trees the game shows the intro logos and then crashes
in `modelPromoteNodeOffsetsToPointers` (finding D179).

The full decomp asset-extraction toolchain (`scripts/extract_baserom.u.sh`,
`tools/extractor`, MIPS binutils) is **not** on the runtime path and never
needs to reach an end user — it only regenerates the committed `assets/**`
`.inc.c` / `.bin` metadata that a plain `git clone` already has.

### Important facts established while writing this plan

1. **`gen_romassets.py` does not need the ROM.** It resolves every symbol for
   `u`, `e`, and `j` from committed files (`scripts/filelist.<r>.csv`,
   `assets/obseg/file_resource_table.inc.c`, `assets/ramrom/ramrom.s`,
   `assets/music/*.s`, `ge007.ld`). The ROM is only opened to size the
   trailing `images` segment, and there is already a CSV-derived fallback when
   it is absent. So all three `romassets_<r>.s` files can be generated in CI
   with no ROM. (`docs/building.md` claims PAL/JP need "region-specific
   prop/font offsets from that region's ROM" — the current generator code
   shows no such ROM read; treat that sentence as stale, but verify before
   relying on it — see Open questions.)

2. **`d43_emit.py` / `d69_emit.py` are pure-stdlib Python 3.** They read only
   the ROM plus committed repo files, and their output is a deterministic
   function of the ROM. Both carry extensive built-in round-trip validation
   (re-parse the emitted image, check every field against the N64 source) and
   exit non-zero on any mismatch. `d43_emit.py` also runs a battery of
   buffer-fit cross-checks.

3. **The engine is region-locked at compile time.** `REGION_DEFS` in
   `CMakeLists.txt` differ per region (`VERSION_US/EU/JP`, `BUGFIX_*`,
   `LEFTOVERDEBUG`, …) and are used in 60+ game files. One exe cannot serve
   all three ROMs; the release must ship three exes (or one launcher that
   dispatches to three).

4. **Data/config location is already sane.** `sysResolvePath()` maps `$S/` to
   `./data/` (next to CWD if present, else next to the exe). `ge007.ini`,
   `ge007.eep`, and the sidecar trees all live under `$S/`. First launch
   already writes `ge007.ini` there. The sidecar loaders already search
   `$S/`, `$E/`, and `./`.

5. **ROM identity checking already half-exists.** `port/src/romdata.c`
   validates magic / title / cart-ID / country byte. `tools_pc/romverify.c`
   goes further (recomputes the `n64cksum` 6102 checksum over
   `[0x1000,0x100000)` using `tools/n64cksum.c`). Canonical SHA-1s for all
   three regions are in the README requirements table.

## Recommended approach

**Two-part: (A) fold the ROM-independent piece into CI now, and (B) add a
native first-run converter to the engine so the sidecars are generated on the
user's machine from their own ROM.**

### Part A — ship all three regions, pre-generate all `romassets`

Small, low-risk, unblocks a "download three exes, pick the one for your ROM"
release even before Part B lands.

- CI `windows-build` becomes a 3-way matrix (`ntsc-final`, `pal-final`,
  `jpn-final`). Each leg runs `gen_romassets.py <u|e|j>` then `build-pc.sh`.
- `bundle-win.sh` collects all three exes (`ge007.ntsc-final.exe`,
  `ge007.pal-final.exe`, `ge007.jpn-final.exe`) into one bundle, with shared
  DLLs.
- Add a tiny **launcher** `ge007.exe` (≈150 lines C, its own CMake target):
  reads `data/*.z64` (and `baserom.*.z64`), checks the header country byte +
  SHA-1 against the three known-good hashes, then `CreateProcess`es the
  matching region exe. On no ROM / unknown ROM it shows a message box naming
  the accepted versions and hashes and the folder to drop the ROM in.
  - Alternative if a separate launcher is unwanted: make each region exe, on
    detecting a ROM whose country byte is for a different region, re-exec its
    sibling. Slightly hackier; avoids a fourth binary.

### Part B — native first-run sidecar converter

Port `d43_emit.py` and `d69_emit.py` to C as a new engine module
`port/src/pcconvert/` (files: `pcconvert.h`, `d43.c`, `d69.c`,
`pcconvert.c` orchestrator). On startup, `romdataInit()` (after it has the
mapped ROM and region) calls `pcconvertEnsureSidecars(region)`:

1. If `$S/pcmodels-<region>/pcmodels.bin` **and** `$S/pccg-<region>/pccg.bin`
   exist and their embedded format-version tag matches the build → return
   immediately (the common case after first run).
2. Otherwise show a "Preparing game data from your ROM (one-time, ~5 s)…"
   splash / log line, run the two converters, write the four output files
   into `$S/`, then continue boot. Cache a `pcconvert.stamp` with the ROM
   SHA-1 + converter version so a ROM swap or engine upgrade re-runs it.

The converters need, besides the ROM:

- `scripts/filelist.u.csv`
- `assets/obseg/file_resource_table.inc.c`
- `assets/**/ModelFileHeader.inc.c` (NS/NT per model)
- the stage `bg` / `stan` `.inc.c` metadata `d69` reads

These are committed text/CSV, a few hundred KB total. **Embed them into the
exe at build time** via a CMake codegen step (`file(READ ...)` →
`.byte`/`const char[]`), so the release stays "exe + DLLs" with nothing loose.
A build-file change is required here (new `add_custom_command` + generated
source) — out of scope for this doc, but noted as the one unavoidable
`CMakeLists.txt` edit.

Reuse: `zlib` is already linked (RZ = `0x11 0x72` + raw deflate — the exact
codec `pcmodels`/`pccg` already decompress). The engine already has robust
byte-swap helpers in `romdata.c`. The Python round-trip validators should be
ported too and run in debug builds / behind an env flag.

**Effort estimate:** `d43_emit.py` is ~1000 lines of intricate model-graph
simulation (LOD/SWITCH rewire, BSP splice, marker-expansion bounds, DFS
placement) + validation; `d69_emit.py` ~500 lines. Call it a 2–4 week
focused task with careful diffing of C output against the Python output
byte-for-byte on all three ROMs before trust. This is the single biggest
work item.

### Interim option (faster Part B, if timeline pressure)

Freeze the two emit scripts with **PyInstaller / Nuitka in the Windows CI
job** (the runner has Python) into one `ge007-convert.exe`, ship it in the
bundle, and have the engine `CreateProcess` it on first run when sidecars are
missing. Pure-stdlib scripts freeze into a single clean exe with no user
Python. Downsides: ~8–15 MB added to the bundle, Windows-only, and it is a
second binary to sign / trust. Recommend this **only** as a stopgap that buys
time for the real C port; it satisfies "no user-visible Python/MSYS2" but not
"single native binary" or non-Windows.

### Rejected options

- **Precompute and ship the sidecars.** They are ROM-derived copyrighted game
  data (`data/pcmodels-*`, `data/pccg-*` are gitignored precisely for this).
  Redistribution is not acceptable. Rejected.
- **Region-independent sidecars.** `d43`/`d69` output is a function of the
  ROM; there is no region-independent form to precompute, and even if there
  were it would still be ROM-derived data. Rejected.
- **Bundle CPython + the .py scripts loose.** Meets the letter of "no MSYS2"
  but not "no Python", ages badly, and is bigger than a frozen exe for no
  gain. Rejected in favour of the interim freeze if a stopgap is needed.

## CI / release changes

1. `windows-build` → matrix over the three ROMIDs; each leg generates its
   `romassets_<r>.s` and builds.
2. Commit all three `port/src/romassets_{u,e,j}.s` (or generate in each CI
   leg — committing is simpler and lets non-CI builds skip the generator).
   Add an `e`/`j` regen + `git diff --stat` check to the `validate` job
   mirroring the existing US one.
3. `bundle-win.sh`: bundle all three region exes + the launcher; keep the
   "no ROM / size ≤ 60 MB" guards (raise the limit only if the interim frozen
   converter is included).
4. `release` job: unchanged shape (draft pre-release); update
   `.github/release-notes.md` with the drop-in-ROM instructions + the
   requirements/hash table.
5. Add a CI check that runs `d43_emit.py --check-only` / a `d69` equivalent
   against a **CI-only** ROM if one can be provided via encrypted secret;
   otherwise this stays a manual pre-publish smoke test (the release job is
   already a human-gated draft).
6. Once Part B lands: a CI step that builds the C converter, runs it and the
   Python converter on the same ROM (secret), and `cmp`s the outputs.

## Code to be written

| File(s) | Rough scope | Notes |
|---|---|---|
| `port/src/launcher/ge007_launch.c` + CMake target | ~150 lines | ROM detect, SHA-1 match vs 3 known hashes, `CreateProcess` region exe, message-box on failure |
| `port/src/pcconvert/d43.c` | ~1000–1300 lines | C port of `tools_pc/d43_emit.py` incl. graph sim + validation |
| `port/src/pcconvert/d69.c` | ~500–700 lines | C port of `tools_pc/d69_emit.py` |
| `port/src/pcconvert/pcconvert.c` + `.h` | ~200 lines | orchestrator, stamp file, splash text, wired into `romdataInit()` |
| CMake codegen for embedded metadata | ~40 lines CMake + generated `.c` | embeds `filelist.u.csv`, `file_resource_table.inc.c`, `ModelFileHeader.inc.c`, bg/stan `.inc.c` |
| `tools_pc/bundle-win.sh`, `.github/workflows/ci.yml` | edits | matrix, 3 exes, launcher, notes |
| SHA-1 helper (or reuse a vendored small SHA-1) | ~100 lines | for launcher + `pcconvert.stamp`; `romverify.c`'s n64cksum path is an alternative identity check |

## ROM SHA-1 / region handling

- Accept a ROM at `data/ge007.<region>.z64`, `data/*.z64` (any name — sniff
  the header), or `baserom.{u,e,j}.z64`.
- Verify: N64 magic + "GOLDENEYE" + "GE" + country byte (already done), then
  SHA-1 against the README table. On a header-valid but hash-mismatched ROM
  (bad dump, byte-swapped `.n64`/`.v64`, ROM hack): warn but allow a
  `--skip-rom-check` / ini opt-out; byte-swapped images should be detected and
  the user told to supply a true big-endian `.z64`.
- Region is taken from the country byte (`E`→ntsc-final, `P`→pal-final,
  `J`→jpn-final), which already drives `pcmodelsRegionForCountry()`.

## Config / data location

No change needed. `$S/` = `./data/` (or `<exedir>/data/`). First run creates
`ge007.ini`; Part B additionally creates `pcmodels-<region>/`,
`pccg-<region>/`, and `pcconvert.stamp` there. Document that the user makes a
`data/` folder next to the exe and drops the ROM in it (the launcher can
offer to create it).

## Open questions

1. **Does PAL/JP `romassets` generation truly need nothing from the ROM?**
   `docs/building.md` says it needs region-specific offsets; the code says
   otherwise. Generate `e` and `j` in CI and diff against a
   locally-ROM-generated copy to confirm before committing them.

2. **Does `d43_emit.py` really use `scripts/filelist.u.csv` for all three
   regions?** — *Resolved: yes, and that is a real bug.* Both emitters
   hard-code the NTSC filelist regardless of `REGION`: `FILELIST =
   "scripts/filelist.u.csv"` at `tools_pc/d43_emit.py:47` and
   `tools_pc/d69_emit.py:56`, so `pal-final`/`jpn-final` runs read NTSC byte
   offsets out of the wrong ROM — decompression garbage or `find_row()`
   misses. The per-region CSVs already exist in the repo with the identical
   shape (`rom_offset,size,asset_path,flag,flag`): `scripts/filelist.e.csv`
   (32,843 bytes) and `scripts/filelist.j.csv` (37,947 bytes), just with bare
   names instead of NTSC's `.bin`-suffixed ones — and `find_row()`
   (`d43_emit.py:89-93`) already tries multiple name variants, so it would
   work against any of the three CSVs if pointed at the right one. The fix is
   small: pick `scripts/filelist.{u,e,j}.csv` from `REGION` instead of
   hard-coding `.u`. **Remaining real unknown:** `d43_emit.py` also reads
   every `assets/**/modelFileHeader.inc.c` (512 files, e.g.
   `assets/obseg/chr/armourguard/modelFileHeader.inc.c:2`) to get NS/NT — arg
   4 and arg 8 of the `MODELFILEHEADER(...)` macro — which it uses in
   `build_nodes()` (`d43_emit.py`, ~lines 125-157) to compute the model root
   offset `R0 = 4*NS + 12*NT`. Those headers are committed decomp data, one
   file per model dir, with no region-suffixed variants; whether NS/NT
   actually differ between NTSC/PAL/JP ROMs for the same model cannot be
   verified from files in this repo alone (it would need an actual PAL or JP
   ROM to check). If they don't differ, the CSV fix above makes `d43`/`d69`
   region-correct on its own; if they do, that is a second, currently
   invisible bug.

3. **Where does Part B's converter write, so `pcmodels.c` / `pccg.c` find it?**
   — *Resolved.* Both loaders resolve the region from the ROM country byte at
   offset `0x3E` via the same `Country→region` map (`'E'`→`"ntsc-final"`,
   `'P'`→`"pal-final"`, `'J'`→`"jpn-final"`;
   `pcmodelsRegionForCountry()` at `port/src/pcmodels.c:66-74`, same pattern
   in `port/src/pccg.c:59-67`), then search the prefixes `$S/`, `$E/`, `./`
   in order (`pcmodels.c:79`; `pccg.c` similar) through `sysResolvePath()`
   (`port/src/system.c:207-259`: on Windows `$S/` → `./data` if it exists,
   else `<exedir>/data`; `$E/` → `<exedir>/`). So a native first-run
   converter in `port/src/pcconvert/` must write exactly, for region `R`:
   `data/pcmodels-R/pcmodels.bin` + `data/pcmodels-R/manifest.csv`
   (`pcmodels.c:100,109`) and `data/pccg-R/pccg.bin` +
   `data/pccg-R/manifest.csv` (`pccg.c:92,101`). The manifest is a CSV with
   header `name,offset,size`, decimal-parsed via `strtol`
   (`pcmodels.c:185-186`, `pccg.c:176-177`), row cap 1024 for pcmodels / 256
   for pccg.

4. **Launcher vs self-re-exec vs three shortcuts** — pick one UX.

5. **Splash UI for the first-run convert** — reuse the SDL window / a simple
   progress line, or a pre-SDL Win32 dialog? ~5 s is short enough that a log
   line + busy cursor may suffice.

6. **Non-Windows.** Part B's C converter is portable; the launcher and
   bundling are Windows-only today. Out of scope but the C port keeps the
   door open.

7. **Signing.** Unsigned exes + a "reads your ROM" first-run step will trip
   SmartScreen. Not blocking, but worth a release-notes note.

8. **How much metadata does Part B's CMake codegen actually have to
   generate?** — *Resolved: far less than the original plan assumed.* The
   emitters read three kinds of committed metadata:
   (a) `scripts/filelist.u.csv` (and now `.e`/`.j` per question 2) — plain CSV,
   ~33-41 KB each; readable from disk at runtime, no embedding needed.
   (b) `assets/obseg/file_resource_table.inc.c` (38,056 bytes, 884 lines) —
   used only to extract an ordered filename list via regex; a C port can read
   the already-compiled `file_resource_table[]` symbol directly at link time
   instead of parsing or embedding it.
   (c) `assets/**/modelFileHeader.inc.c`, 512 files — **this is the one real
   codegen artifact**: the C port needs a build-time-generated table of 512
   entries mapping model symbol → (NS, NT), the two args of the
   `MODELFILEHEADER(...)` macro that `d43_emit.py` actually uses. Everything
   else the scripts use — the N64_REC/PC_REC record-size tables, ADDR_OPS
   marker bytes, `d69_emit.py`'s `TILESIZES[11]` array — are small fixed
   constants that can just be hand-transcribed into the C port, not
   codegen'd. Bottom line: the CMake codegen step Part B calls for is
   narrowly the 512-entry nsnt table, not a general metadata-embedding
   system.

## Suggested milestone ordering

1. Part A (3-region matrix + pre-generated `romassets` + launcher) — days.
2. Interim frozen `ge007-convert.exe` wired to first-run — days, optional.
3. Part B C port of `d43`/`d69` + embedded metadata + output diffing — weeks.
4. Drop the interim binary; single-binary drop-in experience complete.

## Part B follow-up backlog (was GitHub issue #85, folded here 2026-09-25)

Follow-up to #6, which shipped in v0.2.0 as an NTSC-U-only drop-in (PR #84). The issue was closed on GitHub and its scope is tracked here now. It has three independent workstreams.

### 1. PAL/JP asset pipeline repair (data + build)

`scripts/filelist.e.csv` and `filelist.j.csv` contain naming defects that make PAL/JP conversion impossible from this tree (D258: no PAL/JP PC build has ever been producible). The repair path has been verified but not applied:

- **PAL:** rename 31 model rows to the exact `file_resource_table.inc.c` symbols. Examples: `Pbridge_console` ×6 → `Pbridge_console1aZ…3bZ`, and `Pchrgrenadelauncher` → `PchrgrenadelaunchZ`. Greedy sequence alignment proves the mapping: 512×512 exact 1:1, offsets non-decreasing, and zero size anomalies against US.
- **JP:** rename the mislabeled font row @3044276 from `fontZurichBold_fontchartable.bin` to `fontBankGothic_fontchartable.bin`. The contiguity invariant confirms it: the Gothic kerning table @3043600+676 ends exactly at this row, and the size matches the US Gothic chartable.
- **Then:**
  - `gen_romassets.py e/j` must run clean without a ROM.
  - Commit `romassets_e.s` and `romassets_j.s`.
  - Replace the hardcoded `filelist.u.csv` in the d43/d69/d88 emitters with a region→suffix map.
  - Remove the NTSC-U-only gate in `prepare-assets.py`.
- **Needs a PAL and/or JP ROM** for final byte-level verification of at least one region. Only NTSC is available locally.

### 2. Multi-region release layout

- Give each region its own OUTPUT_NAME (`ge007-ntsc-final` / `-pal-final` / `-jpn-final`) plus a default alias.
- Add a region re-exec guard in `romdata.c` (~50 lines): if the country byte at 0x3E doesn't match the build, re-exec the sibling exe.
- CI builds all 3 regions sequentially, and the bundle scripts package all 3 exes.

### 3. C port of the emitters (optional, removes PyInstaller)

Port the d43/d69/d88 emit logic to C. That removes the frozen-converter dependency entirely, because the converter becomes part of the engine binary. It's a 2–4 week task, and PD's `filemodel.c` is the closest analogue. It isn't required for PAL/JP: the frozen Python path works for all three regions once #1 lands.

**Acceptance:** a user with a PAL or JP ROM gets the same one-step drop-in experience as NTSC does today.