From bbc32957e1e37b221af0b70b41fd62b217722878 Mon Sep 17 00:00:00 2001 From: James Dansereau Date: Tue, 22 Sep 2026 22:51:21 -0400 Subject: [PATCH] D325-D327: ROM-mod (xdelta) compatibility investigation + settled plan Docs-only. Records the mod-compatibility investigation so it is not re-derived when mod support is picked up. No code changed. Measured two shipped community mods against a vanilla NTSC-U ROM: - Goldfinger 64 v1.0 (12.8 MB xdelta): ROM 12 -> 24 MB, 797/803 vanilla filelist.u.csv slices differ. - Random Eye v1.1 (286 KB xdelta): ROM size byte-identical, yet 96.5% of bytes and 803/803 slices differ; content preserved but shifted (+7216/+7392, one file -229888). Both die at tools_pc/d43_emit.py:273 with a zlib error on a slice taken at a hardcoded vanilla offset. Cause: the file table is compiled in (ob.c -> file_resource_table.inc.c -> romassets_.s) and the converters reuse the same vanilla offsets. ROM size is not the constraint; ROM code is never executed, so ASM hacks are inert regardless. D325 - Goldfinger 64 probe + mod taxonomy. D326 - Random Eye repacks too, so D325's hypothesised in-place tier is empty. D327 - plan settled: reading the table from the ROM is ruled out by measurement (hw_address absent in all four encodings tested, 0 hits for the segment-relative form; two structural fileentry scans found nothing); content-addressed offset remapping with gap inference is the viable route, since obInit differencing proves files are contiguous. Also recorded: GE maps its 0x7F000000 game segment directly from ROM, uncompressed, uniquely signature-findable at ROM 0x34b30. The 1964GEPD fork reviewed in D327 is GPL-2.0 against this MIT tree; no code was copied or vendored, facts re-derived from our own ROM and decomp. --- docs/dev/findings-index.csv | 3 + docs/dev/findings.md | 269 ++++++++++++++++++++++++++++++++++++ 2 files changed, 272 insertions(+) diff --git a/docs/dev/findings-index.csv b/docs/dev/findings-index.csv index 4472b76b..0bb57229 100644 --- a/docs/dev/findings-index.csv +++ b/docs/dev/findings-index.csv @@ -202,3 +202,6 @@ D321,CLOSED,D318 trigger chain fully mapped:,CLOSED — trigger chain mapped and D322,OPEN,Long-session audio degradation:,"OPEN — static triage done (teardown audit, mixer statelessness, D202-coverage check); ranked hypotheses:" D323,OPEN,G_TL_LOD bindings of explicit-LOD textures:,"OPEN / parked — mechanism confirmed in code; census (tools_pc/bg_lod_audit.py) shows 15 levels with explicit-LOD textures bound under TEXTLOD-on classes (stat 378, azt 806 binds heaviest); overwhelmi…" D324,FIXED,"GE's stub declares nothing, so TUs that include it get NO CRT prototypes —",FIXED (2026-09-22) — proven impact: +D325,CLOSED,ROM-mod (xdelta) compatibility:,CLOSED as a question — +D326,CLOSED,Random Eye randomizer repacks the ROM too —,CLOSED as a question — +D327,CLOSED,"Mod support, resolved plan:",CLOSED as a question — diff --git a/docs/dev/findings.md b/docs/dev/findings.md index 967954d0..1be4a427 100644 --- a/docs/dev/findings.md +++ b/docs/dev/findings.md @@ -610,6 +610,9 @@ covers D24–D69; the log continues in §H (D32 procedure, D70–D121). | D322 | **Long-session audio degradation: full campaign on v0.3.0 — audio progressively worsens from Silo, by Caverns/Cradle the OST is inaudible and SFX "come and go"; restarting the game restores it (issue #87, user report, 2026-09-20).** — full `## D322` entry at file tail | OPEN — static triage done (teardown audit, mixer statelessness, D202-coverage check); ranked hypotheses: voice-pool exhaustion/counter drift > queue starvation > evtq saturation. `GE_D322` pool-telemetry probe shipped; needs a campaign capture with `GE_D322=1 GE_D204=1`. | | D323 | **G_TL_LOD bindings of explicit-LOD textures: PC samples driver mips from LOD0, N64 samples the authored mip chain (static census, 2026-09-21).** — full `## D323` entry at file tail | OPEN / parked — mechanism confirmed in code; census (`tools_pc/bg_lod_audit.py`) shows 15 levels with explicit-LOD textures bound under TEXTLOD-on classes (stat 378, azt 806 binds heaviest); overwhelmingly CI8/IA8 (subtle), the only RGBA5551 cut-outs are the D236 set (already fixed). No reported artifact; fix path documented (upload GL mip chains for TEXTLOD-on bindings) but not worth doing speculatively. Also documents GE's OMH field-write encoding (op 0xBA, value at absolute bit position in w1). | | D324 | **GE's stub declares nothing, so TUs that include it get NO CRT prototypes — implicit `int f()` declarations silently corrupt non-int returns: `atof` garbage in every float ini parse (D38 class).** — full `## D324` entry at file tail | FIXED (2026-09-22) — proven impact: `Game.ScreenShakeIntensity` (the only float ini option) silently parsed to garbage whenever a user set it; caught via the D294 `GE_ROOMPOOL` hook. Fix: `port/shim/stdlib.h` now declares atof/strtod/strtol/atexit/getenv unconditionally (K&R form); pc_protos.h gains atof/strtod for game TUs; glass2.c PORT block uses `__builtin_memset`. Residual harmless implicit decls (atoi/floorf/sqrtf/memset builtins, lround) documented. | +| D325 | **ROM-mod (xdelta) compatibility: relocating/total-conversion mods cannot work while the file table is compiled in; in-place data-only mods are one sidecar-staleness fix away (Goldfinger 64 probe, 2026-09-22).** — full `## D325` entry at file tail | CLOSED as a question — mechanism measured (GF64 relocates 797/803 vanilla file slices; 12→24 MB expanded ROM; `d43_emit` dies on a vanilla-offset zlib slice). ASM mods never work; relocating mods need a runtime-parsed file table (rule-2 collision for added levels); in-place data-only mods need only a ROM-identity stamp in `manifest.csv` (`port/src/romconvert.c`). | +| D326 | **Random Eye randomizer repacks the ROM too — the hypothesised "in-place data-only" mod tier is empty in practice; the in-ROM file table is the single keystone for any mod support (2026-09-22).** — full `## D326` entry at file tail | CLOSED as a question — identical ROM size but 96.5% of bytes and 803/803 file slices differ (content preserved, shifted +7216/+7392/−229888); same `d43_emit` zlib failure. D325's cheap tier disproven. Two bounded scans failed to find the in-ROM `fileentry` table (likely inside the compressed code image); that recovery question gates everything. Backlogged (local notes §5, extends T3.6). | +| D327 | **Mod support, resolved plan: reading the file table out of the ROM is ruled out by measurement; content-addressed offset remapping (gap inference over contiguously-packed files) is the viable route (1964GEPD fork review, 2026-09-22).** — full `## D327` entry at file tail | CLOSED as a question — plan settled, no research blocker left. `hw_address` absent from the ROM in all 4 encodings tested (0 hits for the segment-relative form); `file_resource_table.inc.c` is a decomp reconstruction. Viable route: locate unchanged files by content, bound changed ones by neighbour gaps (`obInit` differencing proves contiguity). Verified primitive: GE's `0x7F000000` segment maps uncompressed from ROM `0x34b30`, uniquely signature-findable. 1964 fork is GPL-2.0 — facts only, no code reuse. | Phase 2 replaced the Phase-1 demo loop with the real `mainproc()` on real OS threads, compiled GE's real `src/sched.c`, and brought in PD's fast3d software @@ -12683,3 +12686,269 @@ Full campaign (or at minimum Silo → Caverns/Cradle) with **`GE_D322=1 GE_D204= **Residual implicit declarations (audited, all in the harmless buckets above — warning-only, no action):** `atoi` in `src/game/chr.c`, `src/game/sky.c`, `src/game/bondview2.c`, `port/src/optionsoverlay.c`; `lround` in `optionsoverlay.c` (fine on Windows LLP64 where `long`=32 bits; would truncate on an LP64 Linux build — revisit only if a Linux port is actively maintained); the builtin family (`floorf`/`sqrtf`/`memset`) wherever it appears. **Status:** FIXED (2026-09-22). Build clean; D294 verification battery re-run post-fix (faithful→authored size, user settings→0 failures, `GE_ROOMPOOL=0.25`→honoured + exhaustion on demand). Cross-ref: D38 (original implicit-declaration finding), D294 (where it was caught). +## D325 — ROM-mod (xdelta) compatibility: relocating/total-conversion mods cannot work while the file table is compiled in; in-place data-only mods are one sidecar-staleness fix away (Goldfinger 64 probe, 2026-09-22) + +### Question + +Can the port run classic GoldenEye ROM mods distributed as xdelta patches, and +is a `mods/` folder mechanism worth building? Probe target: **Goldfinger 64 +v1.0** (12.8 MB xdelta, a total-conversion campaign), with the Random Eye +randomizer (286 KB xdelta) as the contrasting light-touch case. + +### What the port actually does with the ROM + +- `port/src/romdata.c` reads the `.z64` and memcpys it to the N64 cart address + `0x10000000`; **all game data is read out of it at runtime**. `romSize` comes + from `fsSize`, so ROM size is not fixed — an *expanded* ROM maps fine. The + only header validation is the country byte at `0x3E` (`romHeaderValid`). +- ROM **code is never executed** — we run the compiled decomp C. Every ASM hack + in a mod is inert by construction. +- **The binding constraint:** the file table is *compiled into our binary*, not + read from the ROM. `src/game/ob.c:29` includes + `assets/obseg/file_resource_table.inc.c`, whose `hw_address` fields are + linker symbols resolved by `port/src/romassets_.s` to **hardcoded + vanilla ROM offsets**. The offline converters do the same at convert time: + `tools_pc/d88_emit.py:96` reads that table, and `scripts/filelist.u.csv` + carries 812 vanilla offset/size rows. +- Sidecars (`pccg-*/`, `pcmodels-*/`) are **presence-checked only** + (`rcSidecarsPresent`, `port/src/romconvert.c`); `manifest.csv` is + `name,offset,size` with **no ROM identity stamp**. Swapping the ROM therefore + leaves stale converted data in place silently. + +### Probe (isolated worktree off `main`; primary checkout untouched) + +1. `xdelta3 -d` of `Goldfinger64_1_0.xdelta` against our vanilla NTSC-U ROM + applied **cleanly** (rc=0). Output **25,165,824 bytes** — a 12 MB → 24 MB + expanded ROM. +2. Header: magic `80371240` unchanged, internal name `GOLDENEYE` → `GOLDFINGER`, + country byte still `E` (so `rcRegionForCountry` maps it to `ntsc-final` and + `romHeaderValid` accepts it), CRC pair changed, boot segment + `0x40..0x1000` byte-identical. +3. **Relocation measured:** slicing all 803 usable `filelist.u.csv` + (offset, size) rows out of both ROMs and comparing — **797 of 803 differ**, + 6 identical. Total content replacement with a rebuilt file layout. +4. **Converter failure signature** (first hard stop, exactly as predicted): + + ``` + File "tools_pc/d43_emit.py", line 273, in process + src = zlib.decompress(rom[addr:addr + size][2:], -15) + zlib.error: Error -3 while decompressing data: invalid block type + ``` + + i.e. a hardcoded vanilla offset now lands on unrelated bytes. `d69`/`d88` + and the runtime path would fail the same way; not run, and no build was + attempted. + +### Taxonomy + verdict + +| Mod class | Verdict | +|---|---| +| **ASM / code hacks** | **Never.** ROM code is not executed here. | +| **Relocating / expanding / total conversions** (Goldfinger 64) | **Out of reach.** Every compiled `hw_address` and every `filelist.u.csv` row is wrong. Would require the file table (and, for added levels, the stage tables) to be **parsed from the ROM at runtime** instead of compiled from `file_resource_table.inc.c` + `romassets_*.s`. That is an architectural project, and the added-level half collides with AGENTS.md rule 2 (new stage-table entries are `src/game` logic, not an ABI/layout edit). Not a `mods/` folder feature. | +| **In-place data-only** (offsets and sizes preserved — e.g. a setup randomizer like Random Eye) | **Hypothesis — DISPROVEN by D326**, which found the Random Eye randomizer repacks the ROM as well (identical size, 803/803 slices shifted); no known mod actually inhabits this tier. The fix below is still worth doing on its own merits, but unlocks no mod by itself. Needs one fix: stamp ROM identity (size + hash) into `manifest.csv` and reconvert on mismatch, in `rcSidecarsPresent` / `port/src/romconvert.c`. Without it a swapped ROM keeps stale sidecars and mis-renders in a way that looks like corruption rather than like a stale cache. | + +A bounded scan of both ROMs for an in-ROM 12-byte `fileentry` array +(`{s32 index, char *filename, u8 *hw_address}`, `src/game/ob.h:16`) found no +convincing ~800-entry candidate (only a 40-entry run at `0x29e24c`, present in +both ROMs, almost certainly a false positive of the heuristic). So **whether +the table is even recoverable from an arbitrary modded ROM is unresolved** and +is its own investigation — it is the gating question for the relocating-mod +refactor above. + +### If a `mods/` loader is ever built + +- xdelta3 is **GPLv2** — invoke it as a separate bundled executable (the + existing `prepare-assets/ge007-convert` spawn pattern in + `port/src/romconvert.c`) or implement VCDIFF (RFC 3284) decode; do not link + it into this MIT tree. +- Never redistribute a patched ROM or a patch; user supplies both, same posture + as the base ROM. +- Sidecar trees must be keyed per ROM hash, which is the same stamp fix as + above. + +**Status:** CLOSED as a question — mechanism established and measured; no code +changed. Follow-up (b) was run and **disproved the cheap tier — see D326**, which +also identifies the real keystone (recovering the file table from the ROM at +runtime) and carries the backlog pointer. Follow-up (a), the `manifest.csv` +ROM-stamp fix in `port/src/romconvert.c`, still stands on its own merits (a +genuine latent bug today for anyone swapping ROMs or regions). Cross-ref: D43/D69/D88 (the converters whose vanilla offsets +this exposes), D179 (missing-sidecar abort path). + +## D326 — Random Eye randomizer repacks the ROM too: the hypothesised "in-place data-only" mod tier is empty in practice; the in-ROM file table is the single keystone for any mod support (2026-09-22) + +### Why this was run + +D325 closed with a hypothesis: mods that preserve every file's offset and size +would work almost for free, needing only a ROM-identity stamp on the sidecars. +The Random Eye randomizer (`Murks-Random-eye-zer-v1.1.xdelta`, 286 KB) was the +obvious candidate — a *setup* randomizer, no new content, and a patch 45× smaller +than Goldfinger 64's. This tests that hypothesis. **It fails.** + +### Measurements (same isolated worktree method as D325) + +- Patch applies cleanly (rc=0) to our vanilla NTSC-U ROM. Output size is + **identical** — 12,582,912 bytes. Internal name still `GOLDENEYE`, country + byte still `E`, boot segment `0x40..0x1000` byte-identical, CRC pair changed. + Every surface signal says "small in-place edit". +- It is not. **96.5% of all ROM bytes differ** (12,144,703 of 12,582,912), and + **803 of 803** usable `filelist.u.csv` slices differ — every single file, + including fonts and the Rareware logo, which a randomizer has no reason to + touch. +- The content is preserved and **shifted**: searching the modded ROM for + vanilla file content finds it intact at a new offset — + `jfont_dl` / `jfont_chardata` / `efont_chardata` / `animationtable_entries` + all at **+7216**, `GwppksilZ` / `PICBMZ` at **+7392**, and + `GwristdartZ` at **−229888** (genuinely rearranged, not merely displaced). +- Converter failure is the D325 signature again: + `tools_pc/d43_emit.py:273` → `zlib.error: Error -3 while decompressing data: + invalid stored block lengths`. + +**Conclusion:** the community toolchain (GE Setup Editor lineage) *repacks* the +ROM. Even a content-neutral randomizer comes out with a rebuilt file layout. So +the D325 taxonomy's cheap tier — "in-place data-only" — has no known real +inhabitants, and the `manifest.csv` ROM-stamp fix, while still worth doing on +its own merits (swapping ROMs/regions today silently serves stale sidecars), +**does not by itself unlock any actual mod**. + +### Revised verdict: one keystone, not a tiering problem + +All ROM-mod support now reduces to a single research question: **can the file +table be recovered from an arbitrary modded ROM at runtime?** Our table is +compiled in (`src/game/ob.c:29` → `file_resource_table.inc.c` → linker symbols +in `port/src/romassets_.s`), and the converters re-use the same vanilla +offsets (`tools_pc/d88_emit.py:96`, `scripts/filelist.u.csv`). + +Two bounded scans failed to locate the table in either ROM: + +1. 12-byte-stride runs of `{s32 index, char *filename, u8 *hw_address}` + (`src/game/ob.h:16`) with ascending index and in-range ascending address — + best candidate was a 40-entry run at `0x29e24c` present *identically in both + vanilla and Goldfinger 64*, i.e. a heuristic false positive, not an ~800-entry + table. +2. Known file offsets encoded as big-endian u32 in raw / `|0x10000000` / + `|0xB0000000` form, over the first 200 `filelist.u.csv` entries: 20 raw hits + across the whole ROM, 1 each for the other two encodings, **no dense cluster + anywhere**. + +**Answered in D327:** the table is provably not in the ROM in any plaintext +encoding (including the segment-relative `0x7F000000` form, 0 hits), so approach +(a) is ruled out — `file_resource_table.inc.c` is a decomp reconstruction, and +the retail table lives in the loaded/compressed code image this port never +unpacks. D327 replaces this open question with a viable route: content-addressed +offset remapping with gap inference over contiguously-packed files. + +**Not fixable regardless:** ASM/code mods (ROM code is never executed here), and +mods that add levels (new stage-table entries are `src/game` logic — AGENTS.md +rule 2, not the ABI/layout exception). + +**Status:** CLOSED as a question — D325's cheap tier disproven, keystone +identified and then **resolved in D327** (table-in-ROM ruled out; content-addressed +remapping specified). No code changed. Backlogged as §5 of the local +`docs/dev/notes/MODERN-PORT-FEATURES-BACKLOG.md` (extends T3.6), sized +large/multi-session and gated on the table-recovery question; correct sequencing +is after first public release. Cross-ref: D325 (Goldfinger 64 probe + the +taxonomy this revises), D43/D69/D88 (the converters carrying vanilla offsets), +D179 (missing-sidecar abort path). + +## D327 — Mod support, resolved plan: reading the file table out of the ROM is ruled out by measurement; content-addressed offset remapping is the viable route (1964GEPD fork review, 2026-09-22) + +### Why this was run + +D326 left one gating unknown — "can the file table be recovered from a modded +ROM at runtime?" — with a guess that it lives inside the compressed code image. +A community fork, `1964GEPD-automatic-mod-compatibility`, advertises automatic +mod compatibility and was reviewed for a technique. This entry settles the +gating question with measurements and replaces the guess with a plan. + +### What the 1964 fork actually does (not what we need) + +Its change is in the emulator's TLB setup (`InitTLBOther`). 1964 had a +GoldenEye **speed hack** that direct-maps the `0x7F000000` game segment with the +ROM offset hardcoded (`0x90034b30`); mods relocate that segment, so the hack +broke. The fork replaces the hardcode with a scan for a unique 5-instruction +signature at the segment's head. Its own code comments note the game *"still +work[s] without hack"* — this is an optimization guard, not mod-loading +machinery. + +**Why it transfers so poorly:** 1964 *executes the ROM's MIPS code*, so it never +needs a file table at all. Mod compatibility is nearly free for an emulator and +structural for us. There is exactly one transferable fact (next paragraph). + +### Transferable fact, independently verified + +GE maps its `0x7F000000` game segment **directly from ROM, uncompressed**, and +the segment head is uniquely identifiable. Re-derived here against our own +vanilla NTSC ROM by scanning for the instruction sequence +`lui $at,0x3f80` / `mtc1 $at,$f0` / `addiu $v0,$zero,-1` / `lui $at,hi` / +`sw $v0,lo($at)`: **exactly one hit, ROM offset `0x34b30`** — matching the +emulator's old hardcoded `0x90034b30`. This locates a *relocated* segment in a +modded ROM in one pass. Useful, but on its own it does not yield the file table. + +### (a) Reading the table out of the ROM — RULED OUT + +`file_resource_table`'s `hw_address` values are not present in the vanilla ROM +in any plaintext encoding. Measured over the first 100 `filelist.u.csv` offsets: + +| Encoding | Hits | +|---|---| +| raw u32 offset | 4 (incidental) | +| `\|0x80000000` | 1 | +| `\|0xB0000000` | 1 | +| segment-relative `0x7F000000 + off − 0x34b30` | **0** | + +Plus two structural scans for a 12-byte `{s32 index, char *filename, +u8 *hw_address}` run (`src/game/ob.h:16`) — one requiring ascending index and +in-range ascending address, one additionally requiring `filename` to point at a +plausible ASCII string. No candidate in either ROM. (The 40-entry run at +`0x29e24c` reported in D325 appears identically in vanilla *and* Goldfinger 64, +confirming it as heuristic noise.) + +This is consistent with `assets/obseg/file_resource_table.inc.c` being +self-labelled `//TODO: Autogenerate me` — a decomp **reconstruction**, not the +retail layout. The retail equivalent lives in the loaded/compressed code image, +which this port never unpacks (we compile the decomp instead). **So D326's +gating question is answered in the negative, and no further scanning is +warranted.** + +### (b) Content-addressed offset remapping — the viable route + +`obInit` derives each file's size by differencing adjacent `hw_address` values +(`src/game/ob.c:126`), which means **files are packed contiguously in the ROM**. +That is the lever: + +1. We know every vanilla file's exact bytes. For a repacked ROM, locate each + *unchanged* file by content search → its new offset. +2. A file the mod actually **changed** cannot be found that way, but because the + layout is contiguous it is **bounded by the gap between its located + neighbours** — offset and size both fall out. +3. Output is a per-mod offset/size table that replaces the compiled + `hw_address` values at runtime and feeds `d43`/`d69`/`d88` in place of + `scripts/filelist.u.csv`. + +One-time cost at the convert step, not a runtime cost. Verified plausible +against Random Eye, whose content is preserved and merely shifted +(+7216 / +7392, one file −229888; D326). + +**Known limits:** a run of *adjacent* changed files cannot be subdivided by gap +inference; files a mod *adds* get no help; and mods that add levels still need +new stage-table entries in `src/game` (AGENTS.md rule 2, not the ABI/layout +exception). ASM/code mods remain impossible — ROM code is never executed here. + +### License posture (important) + +`1964GEPD-automatic-mod-compatibility` is **GPL-2.0**; this is an MIT tree. Do +not vendor it, copy code from it, or paste its functions — the same hygiene call +as the 2026-09-19 removal of the vendored Mouse Injector (`AGENTS.md`, `reference/mouse-injector/README.md`). +The segment-start offset and instruction signature recorded above are facts +about *Rare's* code and were re-derived directly from our own ROM and decomp; +any implementation must be written from our sources, with provenance noted as +"technique observed in a GPL-2.0 project, reimplemented independently". Better +still: the segment start is obtainable from our own link map, so the scan can be +skipped entirely. + +**Status:** CLOSED as a question — mod-support plan settled. Approach (a) ruled +out by measurement; approach (b) specified above with its limits; no runtime +research blocker remains, only implementation effort (large, correct sequencing +is after first public release). No code changed. Cross-ref: D325 (Goldfinger 64 +probe + mod taxonomy), D326 (Random Eye repack — cheap tier disproven), +D43/D69/D88 (converters carrying vanilla offsets), D179 (missing-sidecar abort).