diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index 87a7294e..70afb78d 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -78,15 +78,46 @@ Full workstream breakdown (WS1–WS6) in `docs/PLAN-linear-level-sweep.md`. `HANDOFF.md`, `AGENTS.md`, `CLAUDE.md`): the D121 + D122 entries in `docs/PCPortResearch.md` §F/§H and this file's edits. Commit or fold in. -## Next task — WS4/WS5: drive the 21-level sweep +## Next task — converter write-audit (C3r / C4 / C5 / C6) -**WS4 matrix DONE** — `docs/LEVEL-STATUS.md`. **7 / 21 PASS** (Bunker1 09, -Silo 20, Archives 24, Train 25, Caverns 39, Egypt 32, Cuba 54). 14 crashes -in 7 classes C1–C7. Reusable sweep runner: `tools_pc/level_sweep.sh` -(bare `-level_XX`, `GE_PCDUMP="80-260:40"`, 24 s watchdog, -`taskkill //F //IM ge007.x86_64.exe`; **`export PATH=".../mingw64/bin:$PATH"` +**D125 is LANDED (M-16b).** `tools_pc/d88_emit.py:374` now writes 8 NUL +bytes; all 21 `Usetup*Z` sidecars regen'd; **Aztec `-level_28` PASSES** +(was C3 crash), Bunker1/Silo unregressed, `d125_check.py` 21/21 MATCH. +M-16 probes reverted, scratch deleted, `d125_check.py` kept. Sweep 12 → +13/21. Full write-up: §H D125 M-16b. + +The last 3 sessions' converter bugs are one family — width/endianness/ +stride errors in `tools_pc/d*_emit.py` (D112 `put_f32`, D120 opcode-0x18, +D125 slice width). Pi audited every `out[` write in `d88_emit.py`; +`d43_emit.py` + the BG sidecar emitters never got that pass, and the open +crashes line up with exactly those files: + +- **C3r** Bunker2 — DOOR-tail `linkedDoorOffset`/`linkedDoor` layout in + `d88_propdefs.py` vs compiled PC `DoorRecord` (`offsetof`/`sizeof` + cross-check; D123 read-before-write-id pattern). +- **C6** Surface2 — `PitemZ_entries[modelid]` (D122/D123 cont., + `d43_emit.py` / `d88_propdefs.py`). +- **C4** Depot — BG tile/room table (`prop.c:902`). +- **C5** Control — BG portal `offset_portal` unresolved/BE (`bg.c:5723`). + +Audit `d43_emit.py` + BG sidecar emit paths for slice-width mismatches, +un-byteswapped fields, N64-vs-PC stride math. Then the structural piece: +**C2-GDL** (Runway + Facility) — `docs/BRIEF-C2gdl-model-reloc.md`, its +own focused session. Defer C2m Jungle (matrix), C7 Surface1 (add a narrow +`#ifdef PORT` `sndSetupSound` load guard only — audio is Phase 3). + +Sweep note: `level_sweep.sh` was flaky this session (spurious NO-FRAMES on +known-good levels under machine load / the 24 s watchdog + D117 +nondeterminism). Verify individual levels with a 35–45 s window and +`GE_PCDUMP="60-500:40"` when a sweep row looks wrong. + +Sweep runner: `tools_pc/level_sweep.sh` (bare `-level_XX`, `GE_PCDUMP="80-260:40"`, +24 s watchdog, `taskkill //F //IM ge007.x86_64.exe`; **`export PATH=".../mingw64/bin:$PATH"` before `addr2line`** — the script's own symbolication no-op'd without it). +The M-13 matrix below is partially superseded by M-14 (see §H D125 for the current +12/21 status) — treat §H as authoritative. + | Class | Levels | Site | |---|---|---| | **C1** | Dam 33, Runway 35, Frigate 26, Statue 22, Streets 29, Cradle 41 | `chrIsNotDeadOrShot` chraction.c:4483 — `self` = AI-list rodata ptr | diff --git a/docs/LEVEL-STATUS.md b/docs/LEVEL-STATUS.md index b216b49e..3c8096d8 100644 --- a/docs/LEVEL-STATUS.md +++ b/docs/LEVEL-STATUS.md @@ -27,16 +27,19 @@ Build = `f2beae4b` + `tools_pc/d88_propdefs.py` (D123) + `port/src/gimgfixup.c` | Runway | 35 | CRASH | `import_texture_i8` gfx_pc.cpp:821 (fault 0x721b8ee8) — same as Facility | **C2** | | Facility | 34 | CRASH | `import_texture_i8` gfx_pc.cpp:821 (fault 0x72181ee8) — model-GDL relocation align bug, D124, NOT fixed | **C2** | | Jungle | 37 | CRASH | `gfx_sp_matrix` gfx_pc.cpp:1046 (fault 0x401c68e0) — C2 texture crash fixed (D124), now explosion-DL `G_MTX` (D75/matrix family) | **C2m** | -| Aztec | 28 | CRASH | `door7F054FB4` propobj.c:13601 (`door->model->obj->RootNode->Child->Child`) | **C3** | -| Bunker2 | 27 | CRASH | `door7F054FB4` propobj.c:13523 (door displacement list walk) | **C3** | +| Aztec | 28 | **PASS** 91.0% | — (C3 fixed, D125) | — | +| Bunker2 | 27 | CRASH | `door7F054FB4` propobj.c:13536 (`var_s1->openPosition`, `var_s1`=0xffff.. from `linkedDoor`) | **C3r** | | Depot | 30 | CRASH | `sub_GAME_7F00324C` prop.c:902 (`sp4C->room` after `walkTilesBetweenPoints`) | **C4** | | Control | 23 | CRASH | `sub_GAME_7F0BA2D4` bg.c:5723 (`portal_pts->numPoints`, via chrprop.c:62) | **C5** | | Surface2 | 43 | CRASH | `modelLoad` loadobjectmodel.c:393 (`PitemZ_entries[modelid].header->RootNode`) | **C6** | | Surface1 | 36 | CRASH | `sndSetupSound` snd.c:653 (fault 0x56220001…, ascii-ish) | **C7** | -**12 / 21 PASS.** D123 cleared C1 on all 6 (Dam/Frigate/Statue/Cradle/Streets +**13 / 21 PASS.** D123 cleared C1 on all 6 (Dam/Frigate/Statue/Cradle/Streets PASS; Runway falls through to C2). D124 cleared Jungle's texture crash but -it now hits an explosion-DL matrix crash. 9 crashes remain in 6 classes. +it now hits an explosion-DL matrix crash. D125 (`d88_emit.py:374` 8-byte +slice / 4-byte literal → boundpad-name blob drift) cleared C3-Aztec; +Bunker2 falls through to a separate DOOR-tail `linkedDoor` layout bug +(C3r). 8 crashes remain in 6 classes. Notes: Streets passed this run but the C1 agent saw a `gfx_sp_matrix` crash — likely the same explosion-DL `G_MTX` (C2m) as Jungle, timing- @@ -98,41 +101,44 @@ Full write-up + next-step probe plan: PCPortResearch §F D124-Facility addendum. Likely fix site: the `d43_emit.py`/`pcmodels` sidecar's model texture-blob offsets (N64 8B-Gfx vs PC 16B-Gfx GDL extent). -### C3/C6 update (M-14, D125) — NOT a converter type-gap -Root cause narrowed: the converted `propDefs` blob in RAM does **not** match -`tools_pc/d88_propdefs.py`'s offline output (after record 0 it is -zeros/garbage), so the runtime `sizepropdef()` walk drifts +101 records by -the first door → `setupDoor`/`modelLoad` get the wrong `pdefIndex`/modelid. -Histogram diff is clean (no type absent from passing levels). Suspect a -destination-offset / `pd_end` / header-reloc bug in `tools_pc/d88_emit.py`'s -propdefs emit path (retrofitted post-D122; docstring still says "opaque -passthrough"). Not fixed — see §F/§H D125. **d88_propdefs.py `convert_stream` -itself and `sizepropdef()` PORT strides are verified correct.** +### C3/C6 update (M-14/M-15, D125) — offline pipeline RULED OUT, bug is runtime +Symptom: converted `propDefs` blob in RAM does not match the offline output +(after record 0 it is zeros/garbage), so the runtime `sizepropdef()` walk +drifts and `setupDoor`/`modelLoad` get the wrong `pdefIndex`/modelid. -**Overseer note (M-13, unverified — first task next session):** likely a -pass-1 delta / region-`end` mismatch in `d88_emit.py` ~L308–340. The -tiled `propdefs` region's `end` (pd_end, from the region tiling) may not -equal `pd_start + convert_propdefs()._n64len`. Pass 1 does -`cum += len(propdefs_pc) - (end - start)` with that `end`; if it is -**shorter** than the real record-stream length, the cumulative delta is -under-counted, every subsequent region (aistream / opaque) is placed too -low, and the emit loop's `out[do:...] = src[...]` for the next region -**overwrites the tail of `propdefs_pc`** — leaving record 0 intact and the -rest clobbered, exactly the observed symptom. Cheap confirm: print -`pd_start`, `pd_end`, `_n64len`, `len(propdefs_pc)`, and the next region's -`(start, do)` for `UsetupsevbZ`; check `pd_end == pd_start + _n64len`. -Fix = set the propdefs region `end` to `pd_start + _n64len` (or make -`convert_propdefs` and the tiler agree on the boundary). +**M-15: the offline side is proven correct.** `tools_pc/d125_check.py` +byte-compares the emitted `pccg.bin` propDefs slice (post-RZ-roundtrip, at +the relocated header offset) to `convert_stream()` → **MATCH for all 21 +levels**. The M-13 overseer hypothesis (pass-1 delta / `pd_end` mismatch in +`d88_emit.py` ~L308–340) is **disproven**: instrumented run shows +`tiled_pd_end == H[intro]` and `_n64len == pd_end - pd_start` exactly for +all 21, next region always `intro`. `sizepropdef()` PORT strides re-checked +vs `PROPDEF_PC_BYTES/4` — all 28 match. -### C3 — door model / ModelNode walk (2 levels: Aztec, Bunker2) -`propobj.c:13601` derefs `door->model->obj->RootNode->Child->Child`; -`:13523` walks the `linkedDoor` list. Fault addrs 0x10 / 0x8 → a null-ish -base + field offset. Door prop model not loaded, or `door->model` / -`->linkedDoor` pointer-width mis-set. Same family as D122 (propDef obj -half-swap) — Aztec/Bunker2 emit a door subtype BUNKER1/Silo don't, or the -door propDef `linkedDoor`/`model` field is pointer-width-sized wrong. -Files: `src/game/propobj.c`, `tools_pc/d88_propdefs.py` (door handler), -`src/game/prop.c`. +**→ The corruption is at/after runtime load.** Prime suspect: +`decompressdata()` truncation in `port/src/rzdecomp.c` for the larger +converted setup files (the `while (ret == Z_OK)` inflate loop), or the +STAGE bank buffer / `langLoadToAddr` (`prop.c:1274`) overwriting the tail. +Next step: probe `decompressdata()` `ret`/`produced` vs expected size for +`UsetupsevbZ`. See §F/§H D125. + +### C3 — Aztec FIXED (D125); Bunker2 residual (C3r) +**Aztec** was D125 (boundpad plink-name blob drift from the +`d88_emit.py:374` slice-width bug corrupting `stanPackId` → NULL stan → +door `model=NULL`). Fixed + regen'd, Aztec PASSES. + +**Bunker2 (C3r)** — `door7F054FB4` `propobj.c:13536`, `var_s1` = `0xffff..` +walked from `linkedDoor`. `door->linkedDoor` is resolved at setup from +`door->linkedDoorOffset` (`prop.c:1204-1206`, N64 struct word 32 / 0x80) — +a read-before-write int id, D123's `ailist` pattern. `d88_propdefs.py`'s +DOOR handler (`DOOR_TAIL_PTR_WORDS`, tail loop from word 32) vs the +compiled PC `DoorRecord` layout is the suspect: either `linkedDoorOffset` +lands at the wrong PC byte, or a tail pointer word's 8-align cursor +diverges from the compiler's struct layout. Bunker2 has linked +double-doors; Aztec's doors are singletons (`linkedDoorOffset==0`) so it +never exercised this path. Needs `offsetof`/`sizeof` cross-check of PC +`DoorRecord` vs the converter cursor. Files: `tools_pc/d88_propdefs.py`, +`src/bondtypes.h` (DoorRecord), `src/game/prop.c`, `src/game/propobj.c`. ### C4 — prop tile-walk room deref (1: Depot) `prop.c:902` — `*arg1 = sp4C->room` where `sp4C` came back bad from @@ -156,13 +162,13 @@ id from a propDef record). Fault addr 0x0. Files: `tools_pc/d88_propdefs.py`, crash on load, not silence. May be dodge-able with a narrow guard until audio lands. Files: `src/snd.c`, `port/src/` audio stubs. -## Next (9 crashes, 6 classes — priority order) +## Next (8 crashes, 6 classes — priority order) 1. **C2** Runway + Facility — model-GDL relocation align bug in `objecthandler_2.c`/`texLoadFromGdl` (D80/D82/D83 area). Shared infra, 2 levels + likely helps Jungle/Streets. Biggest win. -2. **C3** Aztec + Bunker2 — door model / `linkedDoor`; brief - `docs/BRIEF-C3-C6-prop-model.md` (dispatch after any `d88_propdefs.py` - work settles). +2. **C3r** Bunker2 — DOOR-tail `linkedDoorOffset`/`linkedDoor` layout in + `d88_propdefs.py` vs compiled PC `DoorRecord` (Aztec cleared by D125); + fold into the converter write-audit. Brief `docs/BRIEF-C3-C6-prop-model.md`. 3. **C2m** Jungle (+ Streets intermittent) — explosion-DL `G_MTX` (`gfx_sp_matrix`), D75/matrix family. 4. **C6** Surface2 — `PitemZ_entries` modelid (D122/D123 continuation). diff --git a/docs/PCPortResearch.md b/docs/PCPortResearch.md index abe5b050..8eed3c6f 100644 --- a/docs/PCPortResearch.md +++ b/docs/PCPortResearch.md @@ -818,7 +818,7 @@ covers D24–D69; the log continues in §H (D32 procedure, D70–D121). | D121 | WS1 frictionless per-level boot: bare `-level_XX` injects its `memallocstringtable[]` `-m*` row (`#ifdef PORT` in `boss.c`) | resolved | | D122 | per-level prop/item model-load crash (Dam/Facility/Runway `modelLoad`/`modelInitRwData`): `d88_propdefs.py` had no handler for 6 ObjectRecord-derived propDef types (47/39/40/45/13/20) → generic arm half-swapped the `[s16 obj][s16 pad]` word → OOB `PitemZ_entries[]` | converter fixed; residual chr/fast3d crashes on those levels are separate | | D123 | crash class C1: `chrIsNotDeadOrShot` NULL deref on 6 levels (Dam/Runway/Frigate/Statue/Streets/Cradle). D122's `OBJ_TAIL_DESC` zeroed the widened `VehichleRecord/AircraftRecord.ailist` slot (w32), but `prop.c:1764/1786` reads a pre-populated int AI-list id there → `ailistFindById(0)` → `GAILIST_AIM_AT_BOND` → `ai()` runs a CHR aim list with `ChrEntityp==NULL` | converter fixed (`OBJ_ID_WORDS`); C1 cleared on all 6, residual crashes are fast3d (C2) | -| D125 | crash classes C3+C6 (Aztec/Bunker2/Surface2): the converted `propDefs` blob in RAM ≠ `d88_propdefs.py` offline output — after record 0 it is zeros/garbage, so the `sizepropdef()` walk drifts (+101 records by the first door) and every `pdefIndex` lookup (`setupDoor`→`linkedDoor`, `modelLoad` modelid) hits the wrong record. NOT a forgotten-type gap (histogram diff clean). Suspect a destination-offset / `pd_end` / header-reloc bug in `tools_pc/d88_emit.py`'s propdefs emit, not in the converter | **not fixed** — investigation cut for usage reset; next: diff pccg.bin bytes vs `convert_stream`, trace `pd_start/pd_end/do` in d88_emit.py | +| D125 | crash classes C3+C6 (Aztec/Bunker2/Surface2): **root cause found (M-16)** — `tools_pc/d88_emit.py:374` assigns a 4-byte literal to an 8-byte slice (`out[dst_o+0x30:dst_o+0x38] = b"\x00…"`, the stan zero-fill in `emit_pad`); every pad/boundpad record silently shrinks the output bytearray by 4 B, and once the tail falls below the boundpad plink string blob, later verbatim leaf writes hit Python out-of-range slice semantics (insert at current end, not relocated offset) → boundpad names drift/truncate (Aztec pad33 `p138d2`→`8d2`) → `stanPackId()` reject → `getposstan()` NULL stan → `setupDoor` leaves door `model=NULL` → crash `propobj.c:13601`. M-14's "propDefs zeros in RAM" was a misread; sidecar propdefs were always correct (M-15) and post-load RAM matches the sidecar byte-for-byte (M-16) | **resolved (M-16b)** — line 374 → 8 NULs; all 21 sidecars regen'd; **Aztec `-level_28` now PASSES** (was C3 CRASH). Bunker2 falls through to a separate DOOR-tail `linkedDoor` layout bug (C3 residual) | | D124 | crash class C2: fast3d bad texture pointer. **Jungle** (`0xabcd0824`): `gimgSyncCompiledGlobalDLs()` slot-detect keyed on the post-fixup marker, which `texLoad()` had already overwritten → compiled `globalDL_0xNNN` explosion DLs kept link-time `IMAGESEG` words → latent on every level, tripped by the first explosion-DL draw. **Facility** (`0x72181ee8`): separate — model/prop GDL from the `texLoadFromGdl()`/`sub_GAME_7F0762E0` relocation path writes non-16-aligned `dst` (N64 8B vs PC 16B `Gfx` stride mix in `objecthandler_2.c`), open D80/D82/D83 area | Jungle fixed (`port/src/gimgfixup.c`); Facility diagnosed, not fixed | Phase 2 replaced the Phase-1 demo loop with the real `mainproc()` on real OS @@ -4221,29 +4221,52 @@ levels emit no propDef type absent from the 12 passing levels — TINTED_GLASS and AUTOGUN appear in passing Caverns/Control too. **This is not a forgotten-type converter gap like D122/D123.** -*Best hypothesis (confidence: mechanism high, exact fault medium).* The -converted `propdefs_pc` bytes produced in `tools_pc/d88_emit.py` -(`convert_propdefs`, line 320) are **not landing in the sidecar at the -offset `g_CurrentSetup.propDefs` resolves to**, OR the `propDefs` header -field is not being relocated to the converted blob's new offset, OR -`pd_end` (the `'propdefs'` region end from the region-tiling at -`d88_emit.py:315-318`) is computed wrong so only the first record is -converted and the rest of the region is left as `bytearray(total_new)` -zero-fill. The "idx0 correct, then zeros" signature points hard at a -destination-offset / length bug in `d88_emit.py`'s propdefs emit -(lines 266-267, 308-338, 471-476), not in `d88_propdefs.py`. Note -`d88_emit.py`'s module docstring still describes propDefs as an "opaque -passthrough blob" (stale post-D122) — the emit path was retrofitted and -may not have been fully reconciled with the region-tiling / delta-reloc -logic. +*Best hypothesis (M-14) — DISPROVEN (session M-15).* M-14 guessed the +converted `propdefs_pc` bytes were not landing in the sidecar at the +relocated `propDefs` offset (a `d88_emit.py` region-tiling / delta-reloc +bug, lines ~308-338/471-476). The M-13 overseer note refined this to a +pass-1 delta / region-`end` mismatch (`pd_end != pd_start + _n64len`). +**Both are wrong.** New diagnostic `tools_pc/d125_check.py` decompresses +the *emitted* `data/pccg-ntsc-final/pccg.bin` entry for every Usetup*Z, +reads the relocated `propDefs` header field, and byte-compares that slice +to a fresh `convert_stream()`: -*Next step.* Dump the actual bytes at `data/pccg-ntsc-final/pccg.bin` for -`UsetupsevbZ`'s propDefs region and compare to `convert_stream` output; -add a `[D88emit]` print of `pd_start / pd_end / len(propdefs_pc) / do` -(destination offset) in `d88_emit.py`; check whether the `propDefs` -header word in the emitted file points at `do`. Likely a 1-region -off-by-one in the sorted region list (intro vs propDefs ordering) or the -converted blob written at the pre-growth offset. + - **All 21 levels: `MATCH`** (Aztec/Bunker2/Surface2 included). The + sidecar's propDefs blob, post-RZ-roundtrip, is byte-identical to the + converter output, placed exactly where the relocated header field + points. + - Instrumented `d88_emit.py` (D125_DEBUG, since reverted): for all 21, + `tiled_pd_end == H[intro]` exactly, `_n64len == pd_end - pd_start` + exactly, the region after `propdefs` in the sorted list is always + `intro`, and the growth `pclen - n64len` fed to `cum` is correct. + There is **no** region-end/delta bug at `d88_emit.py:308-343`. + - Full re-cross-check of `sizepropdef()` PORT strides vs + `PROPDEF_PC_BYTES/4`: all 28 types match (as M-14 already found). + +**Therefore the offline pipeline (converter → emit → RZ compress) is +correct end-to-end. The "idx0 correct, then zeros" RAM signature M-14 +observed must originate at or after runtime load** — candidates, in +rough order: (a) `decompressdata()` truncation — `port/src/rzdecomp.c` +inflates with `Z_FINISH` and a 4 MiB `avail_out`, loops while `Z_OK`; +if the deflate stream for these larger converted files hits `Z_BUF_ERROR` +or `Z_STREAM_END` mid-output the `while (ret == Z_OK)` exits early and +the tail of `dst` stays zero (needs a probe: log `ret` + `produced` vs +expected decompressed size for `UsetupsevbZ`); (b) the STAGE bank alloc / +`mempAllocBytesInBank` giving a buffer that is fine for the N64-size file +but the converted file is ~15-25% larger — check `mempGetBankSizeLeft` +at the `_fileNameLoadToBank(strResource, …, 256, MEMPOOL_STAGE)` call +(`prop.c:1271`) vs the decompressed size; (c) the `prop.c:1280-1308` +rebase loop or `langLoadToAddr` (`prop.c:1274`, runs right after the +load, same bank) overwriting the tail of the setup file. M-14's own +runtime dump had `g_CurrentSetup.propDefs = 0x701ce814` (`&15 == 4`). + +*Next step.* Probe `decompressdata()` for `UsetupsevbZ`: print `ret`, +`produced`, and the first 64 bytes at `dst + 16` (offline record 1 +start). If `produced` < the offline decompressed file size → truncation +(fix the inflate loop / `avail_in`). If `produced` is correct but the +bytes are still zero → something overwrites it post-load (bisect +`langLoadToAddr` / the rebase loop). `tools_pc/d125_check.py` stays in +tree as the offline-side regression guard. *Probes:* **none left in tree** — the two `#if defined(PORT)`/`GE_C3` scratch prints in `src/game/prop.c` (lines ~1206 and ~1865) and the @@ -4253,6 +4276,85 @@ entry) + `docs/LEVEL-STATUS.md` (one line). No code, no converter, no sidecar change — **sidecars do NOT need regen**, tree builds clean at `f2beae4b` + M-13 uncommitted set. +**D125 addendum (session M-16) — ROOT CAUSE FOUND: `d88_emit.py:374` +4-byte literal into an 8-byte slice; per-record buffer shrink; out-of-range +leaf writes insert at the current end → boundpad name blob drift.** + +*Crash mechanism (Aztec, `-level_28`), fully traced.* Door idx 226 +(`objid=307`, `pad=33`) is left with `model=NULL`/`prop=NULL` by +`setupDoor()`: its boundpad's plink name in the sidecar is `"8d2"` +(original ROM: `"p138d2"`) → `stanPackId()` rejects it (valid = `p`/`q` ++ decimal ≤32767 + letter + optional digit 0-7; probe line `D88 +stanMatchTileName id=8d2 hi=ffff lo=ff` is the failure sentinel) → +`init_pathtable_something()` returns 0 with `*tile_stack=NULL` → +`getposstan()` nonzero → `setupDoor` sets `door->prop=NULL`, skips +`doorInit()` → later tick `door7F054FB4()` (`propobj.c:13601`) derefs +`door->model->obj`; `Model.obj` sits at PC offset 0x10 (widened `chr` +ptr) → fault addr 0x10, Rcx=0. Linked-door pair: idx225 (`pad=32`, valid) +has `linkedDoorOffset=1` → idx226. All 178 regular pads resolve fine; only +boundpad names are corrupt. + +*The bug.* `tools_pc/d88_emit.py` `emit_pad()` line 374: +`out[dst_o + 0x30:dst_o + 0x38] = b"\x00\x00\x00\x00"` — the stan zero-fill +assigns a **4-byte** literal to an **8-byte** slice. Python silently allows +in-range short-RHS slice assignment, so each of Aztec's 179 pads + 148 +boundpads shrinks the output `bytearray` by exactly 4 B (measured: after +header 0x14d1c → after pads 0x14a50 → after boundpads 0x14800). Later +verbatim leaf writes (the plink string blob, relocated to ~0x147EC+) then +target offsets at/past the shrunken tail; for those, Python clamps the +slice to an empty range **at the current end** and inserts the data there — +each string lands at the buffer tail instead of its relocated offset, +drifting earlier with every write. A simulation of exactly this semantics +reproduces the observed sidecar names byte-for-byte: pad30 `p139d2` +(intact), pad31 `139d2`, pad32 `37d1`, **pad33 `8d2`**, pad34 `2`, +pad35 `1`, pad36 `` (empty). + +*Why it was missed.* No error is raised; the final file is only +`(n_pads+n_boundpads)*4 − re-extension` bytes short of `total_new` +(Aztec: 85276 vs 85145) because later leaf writes re-extend the buffer; +`d125_check.py` verified only the propDefs slice. M-14's "propDefs zeros in +RAM" signature was a misread — M-15 already proved sidecar propdefs +byte-correct, and M-16's post-load RAM dump matched the sidecar byte-for- +byte (85,145 B), so runtime load is clean too. + +*Fix (pending).* Line 374 → `b"\x00\x00\x00\x00\x00\x00\x00\x00"` (8 NULs). +Audit of every other `out[` write in the file: all remaining slice/RHS +pairs are width-matched — this is the only one. **All 21 Usetup*Z sidecars +are affected to varying degrees** (shrink = 4×(pads+boundpads) per level; +which names drift depends on where the blob lands vs the shrunken tail) — +regen all: `python tools_pc/d88_emit.py ntsc-final --regen`, then re-run +`tools_pc/level_sweep.sh` (expect C3 Aztec+Bunker2 to clear; currently- +passing levels can only improve, but the sweep is the proof). + +*Probes in tree (temporary — remove after fix verified):* `GE_D125` +env-gated prints in `port/src/rzdecomp.c` (decompress ret/produced), +`src/game/prop.c` (setup header + post-load 0x15000 dump to +`d125_setupdump.bin` + post-`setupDoor()` door state), `src/game/propobj.c` +(`door7F054FB4` NULL-model detect+skip, `objInit` failure print, plus temp +`#ifdef PORT` stdio/stdlib includes). Scratch: `tools_pc/d125_convtest.py` +(instrumented converter copy — source of the shrink evidence), +`tools_pc/d125_check.py` (**keep** — offline regression guard), +`tools_pc/d125_inflate_test.c`, `d125_offs_tmp.c`, `d125_t.c`, +`d125_run.log`, `d125_setupdump.bin`, `build-pc/d125conv/`. + +**D125 addendum (session M-16b) — FIX LANDED.** `d88_emit.py:374` changed +to 8 NUL bytes (+ a warning comment). Offline proof (Aztec, throwaway +`d125_proof.py`): the fixed converter's boundpad plink names 30–36 read +`p139d2 p139d2 p837d1 p138d2 p85a2 p1377b1 p504d2` (= original ROM), 147 +boundpad records recovered (M-14's "46" was the corrupt read — no second +bug). `d43/d69/d88 --regen` → 21/21; `d125_check.py` → 21/21 propDefs +still MATCH. Direct re-test: **Aztec `-level_28` PASSES** (91.0%, 900+ +frames, was C3 CRASH at `propobj.c:13601`); Bunker1/Silo unregressed. +**Bunker2 `-level_27` still crashes** — moved to `door7F054FB4` +`propobj.c:13536` (`var_s1->openPosition`, `var_s1` = `0xffff…` from +`linkedDoor`). Separate bug: the DOOR-tail converter in `d88_propdefs.py` +(`DOOR_TAIL_PTR_WORDS` / `linkedDoorOffset` word 32) vs the compiled PC +`DoorRecord` layout — `linkedDoor` is resolved from `linkedDoorOffset` +(`prop.c:1204`), a read-before-write int id like D123's `ailist`; Bunker2 +has linked double-doors, Aztec's are singletons so it dodged it. Folded +into the converter write-audit (C3 residual). Sweep: 12 → 13/21. +All M-16 probes reverted, scratch files deleted, `d125_check.py` kept. + **D124-Facility addendum (session M-14 — partial, NOT fixed, out of time).** Re-instrumented with a `GE_C2GDL` probe in `sub_GAME_7F0762E0` (`objecthandler_2.c`) + a bad-`G_SETTIMG` catcher in fast3d diff --git a/docs/PORT-LEARNINGS.md b/docs/PORT-LEARNINGS.md index 75360e62..a804696c 100644 --- a/docs/PORT-LEARNINGS.md +++ b/docs/PORT-LEARNINGS.md @@ -53,6 +53,17 @@ through a converter or a runtime bswap fixup reads scrambled. (StandTile id/room, header mid/tail). - **Rule:** prefer an offline sidecar converter (D43, D69, D88 pattern, `tools_pc/d*_emit.py`) over a runtime fixup for a whole format. +- **Python slice-assignment width is a silent corruptor** (D125): + `out[a:a+8] = b"\x00\x00\x00\x00"` on a `bytearray` does NOT raise — it + deletes `(8 - len(rhs))` bytes, shrinking `out` and shifting everything + after `a` down. A widened pointer/`stan` slot filled with a 4-byte + literal shrank `d88_emit.py`'s output 4 B per pad; once `len(out)` fell + below a later region's write offset, `out[off:off+n] = data` clamped to + an empty range at the current end and *inserted* there — string blobs + drifted, boundpad names truncated (`p138d2`→`8d2`), `stanPackId` rejected + → door `model=NULL` → crash. When emitting into a pre-sized bytearray, + every `out[a:b] = rhs` MUST have `len(rhs) == b - a`; grep the emitter + for width-mismatched slice writes when a converted region lands wrong. - A one-shot "sync patched values from the ROM copy into a compiled shadow array" pass must key its slot-detection on a field that is still intact *at sync time*. D124: `gimgSyncCompiledGlobalDLs` looked for the diff --git a/tools_pc/d125_check.py b/tools_pc/d125_check.py new file mode 100644 index 00000000..acf6fbfe --- /dev/null +++ b/tools_pc/d125_check.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 +"""D125 diagnostic: does the propDefs blob in the *emitted* pccg sidecar +(after RZ round-trip) byte-match tools_pc/d88_propdefs.convert_stream()? + +Tests the M-14 claim that the RAM propDefs is "zeros/garbage after record 0". +If this script says MATCH, the corruption is downstream of the sidecar +(runtime rebase / bank alloc / sizepropdef walk); if MISMATCH, it is in +d88_emit.py's emit/placement or the RZ recompress. +""" +import csv, struct, zlib, sys +from d88_propdefs import convert_stream, PROPDEF_PC_BYTES + +REGION = "ntsc-final" +rom = open(f"data/ge007.{REGION}.z64", "rb").read() +fl = {} +for r in csv.reader(open("scripts/filelist.u.csv")): + if len(r) >= 3 and r[2]: + b = r[2].rsplit("/", 1)[-1] + b = b[:-4] if b.endswith(".bin") else b + fl[b] = (int(r[0]), int(r[1])) + +man = {} +for r in csv.reader(open(f"data/pccg-{REGION}/manifest.csv")): + if r and r[0].startswith("Usetup"): + man[r[0]] = (int(r[1]), int(r[2])) +sidebin = open(f"data/pccg-{REGION}/pccg.bin", "rb").read() + +be = lambda b, o: struct.unpack_from(">I", b, o)[0] +targets = sys.argv[1:] or sorted(man) +for nm in targets: + a, s = fl[nm] + dec_n64 = zlib.decompress(rom[a:a + s][2:], -15) + pd_n64, intro_n64 = be(dec_n64, 12), be(dec_n64, 8) + want, n64len, pclen = convert_stream(dec_n64, pd_n64, intro_n64) + + off, size = man[nm] + dec_pc = zlib.decompress(sidebin[off:off + size][2:], -15) + # PC header: 8-byte slots, low 4 = LE offset + pd_pc = struct.unpack_from("