D125: fix d88_emit.py:374 8-byte slice / 4-byte literal (Aztec C3 crash)

emit_pad()'s widened-stan zero-fill assigned a 4-byte literal to an
8-byte bytearray slice. Python slice-assign with a short RHS silently
deletes the surplus bytes, so `out` shrank 4 B per pad+boundpad; once
len(out) dropped below a later leaf write's target offset, the write
clamped to empty-at-end and inserted there, drifting/truncating the
boundpad plink-name blob (Aztec pad33 "p138d2" -> "8d2") -> stanPackId
reject -> NULL stan -> door model=NULL -> crash propobj.c:13601.

Fix: 8 NUL bytes (only width-mismatched out[ write in the file).
All 21 Usetup*Z sidecars regenerated. Aztec (-level_28) now PASSES
(was C3 CRASH); Bunker1/Silo unregressed; d125_check.py 21/21 MATCH.
Sweep 12 -> 13/21.

Bunker2 (-level_27) still crashes: separate DOOR-tail linkedDoor
layout bug in d88_propdefs.py (C3r) -- linkedDoor is resolved from
linkedDoorOffset (read-before-write int id, D123 pattern). Folded
into the converter write-audit.

M-16 probes reverted; d125_check.py kept as the offline regression guard.
Generalisable quirk added to PORT-LEARNINGS.md §C.
This commit is contained in:
James Dansereau committed 2026-08-29 19:12:52 -04:00
1 parent 710d0b12b5
commit 060a7252e5
6 files changed
+279 -71

No files matched your search

+37 -6
View File
@@ -78,15 +78,46 @@ Full workstream breakdown (WS1–WS6) in `docs/PLAN-linear-level-sweep.md`.
`HANDOFF.md`, `AGENTS.md`, `CLAUDE.md`): the D121 + D122 entries in
`docs/PCPortResearch.md` §F/§H and this file's edits. Commit or fold in.
## Next task — WS4/WS5: drive the 21-level sweep
## Next task — converter write-audit (C3r / C4 / C5 / C6)
**WS4 matrix DONE** — `docs/LEVEL-STATUS.md`. **7 / 21 PASS** (Bunker1 09,
Silo 20, Archives 24, Train 25, Caverns 39, Egypt 32, Cuba 54). 14 crashes
in 7 classes C1–C7. Reusable sweep runner: `tools_pc/level_sweep.sh`
(bare `-level_XX`, `GE_PCDUMP="80-260:40"`, 24 s watchdog,
`taskkill //F //IM ge007.x86_64.exe`; **`export PATH=".../mingw64/bin:$PATH"`
**D125 is LANDED (M-16b).** `tools_pc/d88_emit.py:374` now writes 8 NUL
bytes; all 21 `Usetup*Z` sidecars regen'd; **Aztec `-level_28` PASSES**
(was C3 crash), Bunker1/Silo unregressed, `d125_check.py` 21/21 MATCH.
M-16 probes reverted, scratch deleted, `d125_check.py` kept. Sweep 12 →
13/21. Full write-up: §H D125 M-16b.
The last 3 sessions' converter bugs are one family — width/endianness/
stride errors in `tools_pc/d*_emit.py` (D112 `put_f32`, D120 opcode-0x18,
D125 slice width). Pi audited every `out[` write in `d88_emit.py`;
`d43_emit.py` + the BG sidecar emitters never got that pass, and the open
crashes line up with exactly those files:
- **C3r** Bunker2 — DOOR-tail `linkedDoorOffset`/`linkedDoor` layout in
`d88_propdefs.py` vs compiled PC `DoorRecord` (`offsetof`/`sizeof`
cross-check; D123 read-before-write-id pattern).
- **C6** Surface2 — `PitemZ_entries[modelid]` (D122/D123 cont.,
`d43_emit.py` / `d88_propdefs.py`).
- **C4** Depot — BG tile/room table (`prop.c:902`).
- **C5** Control — BG portal `offset_portal` unresolved/BE (`bg.c:5723`).
Audit `d43_emit.py` + BG sidecar emit paths for slice-width mismatches,
un-byteswapped fields, N64-vs-PC stride math. Then the structural piece:
**C2-GDL** (Runway + Facility) — `docs/BRIEF-C2gdl-model-reloc.md`, its
own focused session. Defer C2m Jungle (matrix), C7 Surface1 (add a narrow
`#ifdef PORT` `sndSetupSound` load guard only — audio is Phase 3).
Sweep note: `level_sweep.sh` was flaky this session (spurious NO-FRAMES on
known-good levels under machine load / the 24 s watchdog + D117
nondeterminism). Verify individual levels with a 35–45 s window and
`GE_PCDUMP="60-500:40"` when a sweep row looks wrong.
Sweep runner: `tools_pc/level_sweep.sh` (bare `-level_XX`, `GE_PCDUMP="80-260:40"`,
24 s watchdog, `taskkill //F //IM ge007.x86_64.exe`; **`export PATH=".../mingw64/bin:$PATH"`
before `addr2line`** — the script's own symbolication no-op'd without it).
The M-13 matrix below is partially superseded by M-14 (see §H D125 for the current
12/21 status) — treat §H as authoritative.
| Class | Levels | Site |
|---|---|---|
| **C1** | Dam 33, Runway 35, Frigate 26, Statue 22, Streets 29, Cradle 41 | `chrIsNotDeadOrShot` chraction.c:4483 — `self` = AI-list rodata ptr |
+47 -41
View File
@@ -27,16 +27,19 @@ Build = `f2beae4b` + `tools_pc/d88_propdefs.py` (D123) + `port/src/gimgfixup.c`
| Runway | 35 | CRASH | `import_texture_i8` gfx_pc.cpp:821 (fault 0x721b8ee8) — same as Facility | **C2** |
| Facility | 34 | CRASH | `import_texture_i8` gfx_pc.cpp:821 (fault 0x72181ee8) — model-GDL relocation align bug, D124, NOT fixed | **C2** |
| Jungle | 37 | CRASH | `gfx_sp_matrix` gfx_pc.cpp:1046 (fault 0x401c68e0) — C2 texture crash fixed (D124), now explosion-DL `G_MTX` (D75/matrix family) | **C2m** |
| Aztec | 28 | CRASH | `door7F054FB4` propobj.c:13601 (`door->model->obj->RootNode->Child->Child`) | **C3** |
| Bunker2 | 27 | CRASH | `door7F054FB4` propobj.c:13523 (door displacement list walk) | **C3** |
| Aztec | 28 | **PASS** 91.0% | — (C3 fixed, D125) | — |
| Bunker2 | 27 | CRASH | `door7F054FB4` propobj.c:13536 (`var_s1->openPosition`, `var_s1`=0xffff.. from `linkedDoor`) | **C3r** |
| Depot | 30 | CRASH | `sub_GAME_7F00324C` prop.c:902 (`sp4C->room` after `walkTilesBetweenPoints`) | **C4** |
| Control | 23 | CRASH | `sub_GAME_7F0BA2D4` bg.c:5723 (`portal_pts->numPoints`, via chrprop.c:62) | **C5** |
| Surface2 | 43 | CRASH | `modelLoad` loadobjectmodel.c:393 (`PitemZ_entries[modelid].header->RootNode`) | **C6** |
| Surface1 | 36 | CRASH | `sndSetupSound` snd.c:653 (fault 0x56220001…, ascii-ish) | **C7** |
**12 / 21 PASS.** D123 cleared C1 on all 6 (Dam/Frigate/Statue/Cradle/Streets
**13 / 21 PASS.** D123 cleared C1 on all 6 (Dam/Frigate/Statue/Cradle/Streets
PASS; Runway falls through to C2). D124 cleared Jungle's texture crash but
it now hits an explosion-DL matrix crash. 9 crashes remain in 6 classes.
it now hits an explosion-DL matrix crash. D125 (`d88_emit.py:374` 8-byte
slice / 4-byte literal → boundpad-name blob drift) cleared C3-Aztec;
Bunker2 falls through to a separate DOOR-tail `linkedDoor` layout bug
(C3r). 8 crashes remain in 6 classes.
Notes: Streets passed this run but the C1 agent saw a `gfx_sp_matrix`
crash — likely the same explosion-DL `G_MTX` (C2m) as Jungle, timing-
@@ -98,41 +101,44 @@ Full write-up + next-step probe plan: PCPortResearch §F D124-Facility
addendum. Likely fix site: the `d43_emit.py`/`pcmodels` sidecar's model
texture-blob offsets (N64 8B-Gfx vs PC 16B-Gfx GDL extent).
### C3/C6 update (M-14, D125) — NOT a converter type-gap
Root cause narrowed: the converted `propDefs` blob in RAM does **not** match
`tools_pc/d88_propdefs.py`'s offline output (after record 0 it is
zeros/garbage), so the runtime `sizepropdef()` walk drifts +101 records by
the first door → `setupDoor`/`modelLoad` get the wrong `pdefIndex`/modelid.
Histogram diff is clean (no type absent from passing levels). Suspect a
destination-offset / `pd_end` / header-reloc bug in `tools_pc/d88_emit.py`'s
propdefs emit path (retrofitted post-D122; docstring still says "opaque
passthrough"). Not fixed — see §F/§H D125. **d88_propdefs.py `convert_stream`
itself and `sizepropdef()` PORT strides are verified correct.**
### C3/C6 update (M-14/M-15, D125) — offline pipeline RULED OUT, bug is runtime
Symptom: converted `propDefs` blob in RAM does not match the offline output
(after record 0 it is zeros/garbage), so the runtime `sizepropdef()` walk
drifts and `setupDoor`/`modelLoad` get the wrong `pdefIndex`/modelid.
**Overseer note (M-13, unverified — first task next session):** likely a
pass-1 delta / region-`end` mismatch in `d88_emit.py` ~L308–340. The
tiled `propdefs` region's `end` (pd_end, from the region tiling) may not
equal `pd_start + convert_propdefs()._n64len`. Pass 1 does
`cum += len(propdefs_pc) - (end - start)` with that `end`; if it is
**shorter** than the real record-stream length, the cumulative delta is
under-counted, every subsequent region (aistream / opaque) is placed too
low, and the emit loop's `out[do:...] = src[...]` for the next region
**overwrites the tail of `propdefs_pc`** — leaving record 0 intact and the
rest clobbered, exactly the observed symptom. Cheap confirm: print
`pd_start`, `pd_end`, `_n64len`, `len(propdefs_pc)`, and the next region's
`(start, do)` for `UsetupsevbZ`; check `pd_end == pd_start + _n64len`.
Fix = set the propdefs region `end` to `pd_start + _n64len` (or make
`convert_propdefs` and the tiler agree on the boundary).
**M-15: the offline side is proven correct.** `tools_pc/d125_check.py`
byte-compares the emitted `pccg.bin` propDefs slice (post-RZ-roundtrip, at
the relocated header offset) to `convert_stream()` → **MATCH for all 21
levels**. The M-13 overseer hypothesis (pass-1 delta / `pd_end` mismatch in
`d88_emit.py` ~L308–340) is **disproven**: instrumented run shows
`tiled_pd_end == H[intro]` and `_n64len == pd_end - pd_start` exactly for
all 21, next region always `intro`. `sizepropdef()` PORT strides re-checked
vs `PROPDEF_PC_BYTES/4` — all 28 match.
### C3 — door model / ModelNode walk (2 levels: Aztec, Bunker2)
`propobj.c:13601` derefs `door->model->obj->RootNode->Child->Child`;
`:13523` walks the `linkedDoor` list. Fault addrs 0x10 / 0x8 → a null-ish
base + field offset. Door prop model not loaded, or `door->model` /
`->linkedDoor` pointer-width mis-set. Same family as D122 (propDef obj
half-swap) — Aztec/Bunker2 emit a door subtype BUNKER1/Silo don't, or the
door propDef `linkedDoor`/`model` field is pointer-width-sized wrong.
Files: `src/game/propobj.c`, `tools_pc/d88_propdefs.py` (door handler),
`src/game/prop.c`.
**→ The corruption is at/after runtime load.** Prime suspect:
`decompressdata()` truncation in `port/src/rzdecomp.c` for the larger
converted setup files (the `while (ret == Z_OK)` inflate loop), or the
STAGE bank buffer / `langLoadToAddr` (`prop.c:1274`) overwriting the tail.
Next step: probe `decompressdata()` `ret`/`produced` vs expected size for
`UsetupsevbZ`. See §F/§H D125.
### C3 — Aztec FIXED (D125); Bunker2 residual (C3r)
**Aztec** was D125 (boundpad plink-name blob drift from the
`d88_emit.py:374` slice-width bug corrupting `stanPackId` → NULL stan →
door `model=NULL`). Fixed + regen'd, Aztec PASSES.
**Bunker2 (C3r)** — `door7F054FB4` `propobj.c:13536`, `var_s1` = `0xffff..`
walked from `linkedDoor`. `door->linkedDoor` is resolved at setup from
`door->linkedDoorOffset` (`prop.c:1204-1206`, N64 struct word 32 / 0x80) —
a read-before-write int id, D123's `ailist` pattern. `d88_propdefs.py`'s
DOOR handler (`DOOR_TAIL_PTR_WORDS`, tail loop from word 32) vs the
compiled PC `DoorRecord` layout is the suspect: either `linkedDoorOffset`
lands at the wrong PC byte, or a tail pointer word's 8-align cursor
diverges from the compiler's struct layout. Bunker2 has linked
double-doors; Aztec's doors are singletons (`linkedDoorOffset==0`) so it
never exercised this path. Needs `offsetof`/`sizeof` cross-check of PC
`DoorRecord` vs the converter cursor. Files: `tools_pc/d88_propdefs.py`,
`src/bondtypes.h` (DoorRecord), `src/game/prop.c`, `src/game/propobj.c`.
### C4 — prop tile-walk room deref (1: Depot)
`prop.c:902` — `*arg1 = sp4C->room` where `sp4C` came back bad from
@@ -156,13 +162,13 @@ id from a propDef record). Fault addr 0x0. Files: `tools_pc/d88_propdefs.py`,
crash on load, not silence. May be dodge-able with a narrow guard until
audio lands. Files: `src/snd.c`, `port/src/` audio stubs.
## Next (9 crashes, 6 classes — priority order)
## Next (8 crashes, 6 classes — priority order)
1. **C2** Runway + Facility — model-GDL relocation align bug in
`objecthandler_2.c`/`texLoadFromGdl` (D80/D82/D83 area). Shared infra,
2 levels + likely helps Jungle/Streets. Biggest win.
2. **C3** Aztec + Bunker2 — door model / `linkedDoor`; brief
`docs/BRIEF-C3-C6-prop-model.md` (dispatch after any `d88_propdefs.py`
work settles).
2. **C3r** Bunker2 — DOOR-tail `linkedDoorOffset`/`linkedDoor` layout in
`d88_propdefs.py` vs compiled PC `DoorRecord` (Aztec cleared by D125);
fold into the converter write-audit. Brief `docs/BRIEF-C3-C6-prop-model.md`.
3. **C2m** Jungle (+ Streets intermittent) — explosion-DL `G_MTX`
(`gfx_sp_matrix`), D75/matrix family.
4. **C6** Surface2 — `PitemZ_entries` modelid (D122/D123 continuation).
+125 -23
View File
@@ -818,7 +818,7 @@ covers D24–D69; the log continues in §H (D32 procedure, D70–D121).
| D121 | WS1 frictionless per-level boot: bare `-level_XX` injects its `memallocstringtable[]` `-m*` row (`#ifdef PORT` in `boss.c`) | resolved |
| D122 | per-level prop/item model-load crash (Dam/Facility/Runway `modelLoad`/`modelInitRwData`): `d88_propdefs.py` had no handler for 6 ObjectRecord-derived propDef types (47/39/40/45/13/20) → generic arm half-swapped the `[s16 obj][s16 pad]` word → OOB `PitemZ_entries[]` | converter fixed; residual chr/fast3d crashes on those levels are separate |
| D123 | crash class C1: `chrIsNotDeadOrShot` NULL deref on 6 levels (Dam/Runway/Frigate/Statue/Streets/Cradle). D122's `OBJ_TAIL_DESC` zeroed the widened `VehichleRecord/AircraftRecord.ailist` slot (w32), but `prop.c:1764/1786` reads a pre-populated int AI-list id there → `ailistFindById(0)` → `GAILIST_AIM_AT_BOND` → `ai()` runs a CHR aim list with `ChrEntityp==NULL` | converter fixed (`OBJ_ID_WORDS`); C1 cleared on all 6, residual crashes are fast3d (C2) |
| D125 | crash classes C3+C6 (Aztec/Bunker2/Surface2): the converted `propDefs` blob in RAM ≠ `d88_propdefs.py` offline output — after record 0 it is zeros/garbage, so the `sizepropdef()` walk drifts (+101 records by the first door) and every `pdefIndex` lookup (`setupDoor`→`linkedDoor`, `modelLoad` modelid) hits the wrong record. NOT a forgotten-type gap (histogram diff clean). Suspect a destination-offset / `pd_end` / header-reloc bug in `tools_pc/d88_emit.py`'s propdefs emit, not in the converter | **not fixed** — investigation cut for usage reset; next: diff pccg.bin bytes vs `convert_stream`, trace `pd_start/pd_end/do` in d88_emit.py |
| D125 | crash classes C3+C6 (Aztec/Bunker2/Surface2): **root cause found (M-16)** — `tools_pc/d88_emit.py:374` assigns a 4-byte literal to an 8-byte slice (`out[dst_o+0x30:dst_o+0x38] = b"\x00…"`, the stan zero-fill in `emit_pad`); every pad/boundpad record silently shrinks the output bytearray by 4 B, and once the tail falls below the boundpad plink string blob, later verbatim leaf writes hit Python out-of-range slice semantics (insert at current end, not relocated offset) → boundpad names drift/truncate (Aztec pad33 `p138d2`→`8d2`) → `stanPackId()` reject → `getposstan()` NULL stan → `setupDoor` leaves door `model=NULL` → crash `propobj.c:13601`. M-14's "propDefs zeros in RAM" was a misread; sidecar propdefs were always correct (M-15) and post-load RAM matches the sidecar byte-for-byte (M-16) | **resolved (M-16b)** — line 374 → 8 NULs; all 21 sidecars regen'd; **Aztec `-level_28` now PASSES** (was C3 CRASH). Bunker2 falls through to a separate DOOR-tail `linkedDoor` layout bug (C3 residual) |
| D124 | crash class C2: fast3d bad texture pointer. **Jungle** (`0xabcd0824`): `gimgSyncCompiledGlobalDLs()` slot-detect keyed on the post-fixup marker, which `texLoad()` had already overwritten → compiled `globalDL_0xNNN` explosion DLs kept link-time `IMAGESEG` words → latent on every level, tripped by the first explosion-DL draw. **Facility** (`0x72181ee8`): separate — model/prop GDL from the `texLoadFromGdl()`/`sub_GAME_7F0762E0` relocation path writes non-16-aligned `dst` (N64 8B vs PC 16B `Gfx` stride mix in `objecthandler_2.c`), open D80/D82/D83 area | Jungle fixed (`port/src/gimgfixup.c`); Facility diagnosed, not fixed |
Phase 2 replaced the Phase-1 demo loop with the real `mainproc()` on real OS
@@ -4221,29 +4221,52 @@ levels emit no propDef type absent from the 12 passing levels — TINTED_GLASS
and AUTOGUN appear in passing Caverns/Control too. **This is not a
forgotten-type converter gap like D122/D123.**
*Best hypothesis (confidence: mechanism high, exact fault medium).* The
converted `propdefs_pc` bytes produced in `tools_pc/d88_emit.py`
(`convert_propdefs`, line 320) are **not landing in the sidecar at the
offset `g_CurrentSetup.propDefs` resolves to**, OR the `propDefs` header
field is not being relocated to the converted blob's new offset, OR
`pd_end` (the `'propdefs'` region end from the region-tiling at
`d88_emit.py:315-318`) is computed wrong so only the first record is
converted and the rest of the region is left as `bytearray(total_new)`
zero-fill. The "idx0 correct, then zeros" signature points hard at a
destination-offset / length bug in `d88_emit.py`'s propdefs emit
(lines 266-267, 308-338, 471-476), not in `d88_propdefs.py`. Note
`d88_emit.py`'s module docstring still describes propDefs as an "opaque
passthrough blob" (stale post-D122) — the emit path was retrofitted and
may not have been fully reconciled with the region-tiling / delta-reloc
logic.
*Best hypothesis (M-14) — DISPROVEN (session M-15).* M-14 guessed the
converted `propdefs_pc` bytes were not landing in the sidecar at the
relocated `propDefs` offset (a `d88_emit.py` region-tiling / delta-reloc
bug, lines ~308-338/471-476). The M-13 overseer note refined this to a
pass-1 delta / region-`end` mismatch (`pd_end != pd_start + _n64len`).
**Both are wrong.** New diagnostic `tools_pc/d125_check.py` decompresses
the *emitted* `data/pccg-ntsc-final/pccg.bin` entry for every Usetup*Z,
reads the relocated `propDefs` header field, and byte-compares that slice
to a fresh `convert_stream()`:
*Next step.* Dump the actual bytes at `data/pccg-ntsc-final/pccg.bin` for
`UsetupsevbZ`'s propDefs region and compare to `convert_stream` output;
add a `[D88emit]` print of `pd_start / pd_end / len(propdefs_pc) / do`
(destination offset) in `d88_emit.py`; check whether the `propDefs`
header word in the emitted file points at `do`. Likely a 1-region
off-by-one in the sorted region list (intro vs propDefs ordering) or the
converted blob written at the pre-growth offset.
- **All 21 levels: `MATCH`** (Aztec/Bunker2/Surface2 included). The
sidecar's propDefs blob, post-RZ-roundtrip, is byte-identical to the
converter output, placed exactly where the relocated header field
points.
- Instrumented `d88_emit.py` (D125_DEBUG, since reverted): for all 21,
`tiled_pd_end == H[intro]` exactly, `_n64len == pd_end - pd_start`
exactly, the region after `propdefs` in the sorted list is always
`intro`, and the growth `pclen - n64len` fed to `cum` is correct.
There is **no** region-end/delta bug at `d88_emit.py:308-343`.
- Full re-cross-check of `sizepropdef()` PORT strides vs
`PROPDEF_PC_BYTES/4`: all 28 types match (as M-14 already found).
**Therefore the offline pipeline (converter → emit → RZ compress) is
correct end-to-end. The "idx0 correct, then zeros" RAM signature M-14
observed must originate at or after runtime load** — candidates, in
rough order: (a) `decompressdata()` truncation — `port/src/rzdecomp.c`
inflates with `Z_FINISH` and a 4 MiB `avail_out`, loops while `Z_OK`;
if the deflate stream for these larger converted files hits `Z_BUF_ERROR`
or `Z_STREAM_END` mid-output the `while (ret == Z_OK)` exits early and
the tail of `dst` stays zero (needs a probe: log `ret` + `produced` vs
expected decompressed size for `UsetupsevbZ`); (b) the STAGE bank alloc /
`mempAllocBytesInBank` giving a buffer that is fine for the N64-size file
but the converted file is ~15-25% larger — check `mempGetBankSizeLeft`
at the `_fileNameLoadToBank(strResource, …, 256, MEMPOOL_STAGE)` call
(`prop.c:1271`) vs the decompressed size; (c) the `prop.c:1280-1308`
rebase loop or `langLoadToAddr` (`prop.c:1274`, runs right after the
load, same bank) overwriting the tail of the setup file. M-14's own
runtime dump had `g_CurrentSetup.propDefs = 0x701ce814` (`&15 == 4`).
*Next step.* Probe `decompressdata()` for `UsetupsevbZ`: print `ret`,
`produced`, and the first 64 bytes at `dst + 16` (offline record 1
start). If `produced` < the offline decompressed file size → truncation
(fix the inflate loop / `avail_in`). If `produced` is correct but the
bytes are still zero → something overwrites it post-load (bisect
`langLoadToAddr` / the rebase loop). `tools_pc/d125_check.py` stays in
tree as the offline-side regression guard.
*Probes:* **none left in tree** — the two `#if defined(PORT)`/`GE_C3`
scratch prints in `src/game/prop.c` (lines ~1206 and ~1865) and the
@@ -4253,6 +4276,85 @@ entry) + `docs/LEVEL-STATUS.md` (one line). No code, no converter, no
sidecar change — **sidecars do NOT need regen**, tree builds clean at
`f2beae4b` + M-13 uncommitted set.
**D125 addendum (session M-16) — ROOT CAUSE FOUND: `d88_emit.py:374`
4-byte literal into an 8-byte slice; per-record buffer shrink; out-of-range
leaf writes insert at the current end → boundpad name blob drift.**
*Crash mechanism (Aztec, `-level_28`), fully traced.* Door idx 226
(`objid=307`, `pad=33`) is left with `model=NULL`/`prop=NULL` by
`setupDoor()`: its boundpad's plink name in the sidecar is `"8d2"`
(original ROM: `"p138d2"`) → `stanPackId()` rejects it (valid = `p`/`q`
+ decimal ≤32767 + letter + optional digit 0-7; probe line `D88
stanMatchTileName id=8d2 hi=ffff lo=ff` is the failure sentinel) →
`init_pathtable_something()` returns 0 with `*tile_stack=NULL` →
`getposstan()` nonzero → `setupDoor` sets `door->prop=NULL`, skips
`doorInit()` → later tick `door7F054FB4()` (`propobj.c:13601`) derefs
`door->model->obj`; `Model.obj` sits at PC offset 0x10 (widened `chr`
ptr) → fault addr 0x10, Rcx=0. Linked-door pair: idx225 (`pad=32`, valid)
has `linkedDoorOffset=1` → idx226. All 178 regular pads resolve fine; only
boundpad names are corrupt.
*The bug.* `tools_pc/d88_emit.py` `emit_pad()` line 374:
`out[dst_o + 0x30:dst_o + 0x38] = b"\x00\x00\x00\x00"` — the stan zero-fill
assigns a **4-byte** literal to an **8-byte** slice. Python silently allows
in-range short-RHS slice assignment, so each of Aztec's 179 pads + 148
boundpads shrinks the output `bytearray` by exactly 4 B (measured: after
header 0x14d1c → after pads 0x14a50 → after boundpads 0x14800). Later
verbatim leaf writes (the plink string blob, relocated to ~0x147EC+) then
target offsets at/past the shrunken tail; for those, Python clamps the
slice to an empty range **at the current end** and inserts the data there —
each string lands at the buffer tail instead of its relocated offset,
drifting earlier with every write. A simulation of exactly this semantics
reproduces the observed sidecar names byte-for-byte: pad30 `p139d2`
(intact), pad31 `139d2`, pad32 `37d1`, **pad33 `8d2`**, pad34 `2`,
pad35 `1`, pad36 `` (empty).
*Why it was missed.* No error is raised; the final file is only
`(n_pads+n_boundpads)*4 − re-extension` bytes short of `total_new`
(Aztec: 85276 vs 85145) because later leaf writes re-extend the buffer;
`d125_check.py` verified only the propDefs slice. M-14's "propDefs zeros in
RAM" signature was a misread — M-15 already proved sidecar propdefs
byte-correct, and M-16's post-load RAM dump matched the sidecar byte-for-
byte (85,145 B), so runtime load is clean too.
*Fix (pending).* Line 374 → `b"\x00\x00\x00\x00\x00\x00\x00\x00"` (8 NULs).
Audit of every other `out[` write in the file: all remaining slice/RHS
pairs are width-matched — this is the only one. **All 21 Usetup*Z sidecars
are affected to varying degrees** (shrink = 4×(pads+boundpads) per level;
which names drift depends on where the blob lands vs the shrunken tail) —
regen all: `python tools_pc/d88_emit.py ntsc-final --regen`, then re-run
`tools_pc/level_sweep.sh` (expect C3 Aztec+Bunker2 to clear; currently-
passing levels can only improve, but the sweep is the proof).
*Probes in tree (temporary — remove after fix verified):* `GE_D125`
env-gated prints in `port/src/rzdecomp.c` (decompress ret/produced),
`src/game/prop.c` (setup header + post-load 0x15000 dump to
`d125_setupdump.bin` + post-`setupDoor()` door state), `src/game/propobj.c`
(`door7F054FB4` NULL-model detect+skip, `objInit` failure print, plus temp
`#ifdef PORT` stdio/stdlib includes). Scratch: `tools_pc/d125_convtest.py`
(instrumented converter copy — source of the shrink evidence),
`tools_pc/d125_check.py` (**keep** — offline regression guard),
`tools_pc/d125_inflate_test.c`, `d125_offs_tmp.c`, `d125_t.c`,
`d125_run.log`, `d125_setupdump.bin`, `build-pc/d125conv/`.
**D125 addendum (session M-16b) — FIX LANDED.** `d88_emit.py:374` changed
to 8 NUL bytes (+ a warning comment). Offline proof (Aztec, throwaway
`d125_proof.py`): the fixed converter's boundpad plink names 30–36 read
`p139d2 p139d2 p837d1 p138d2 p85a2 p1377b1 p504d2` (= original ROM), 147
boundpad records recovered (M-14's "46" was the corrupt read — no second
bug). `d43/d69/d88 --regen` → 21/21; `d125_check.py` → 21/21 propDefs
still MATCH. Direct re-test: **Aztec `-level_28` PASSES** (91.0%, 900+
frames, was C3 CRASH at `propobj.c:13601`); Bunker1/Silo unregressed.
**Bunker2 `-level_27` still crashes** — moved to `door7F054FB4`
`propobj.c:13536` (`var_s1->openPosition`, `var_s1` = `0xffff…` from
`linkedDoor`). Separate bug: the DOOR-tail converter in `d88_propdefs.py`
(`DOOR_TAIL_PTR_WORDS` / `linkedDoorOffset` word 32) vs the compiled PC
`DoorRecord` layout — `linkedDoor` is resolved from `linkedDoorOffset`
(`prop.c:1204`), a read-before-write int id like D123's `ailist`; Bunker2
has linked double-doors, Aztec's are singletons so it dodged it. Folded
into the converter write-audit (C3 residual). Sweep: 12 → 13/21.
All M-16 probes reverted, scratch files deleted, `d125_check.py` kept.
**D124-Facility addendum (session M-14 — partial, NOT fixed, out of time).**
Re-instrumented with a `GE_C2GDL` probe in `sub_GAME_7F0762E0`
(`objecthandler_2.c`) + a bad-`G_SETTIMG` catcher in fast3d
+11
View File
@@ -53,6 +53,17 @@ through a converter or a runtime bswap fixup reads scrambled.
(StandTile id/room, header mid/tail).
- **Rule:** prefer an offline sidecar converter (D43, D69, D88 pattern,
`tools_pc/d*_emit.py`) over a runtime fixup for a whole format.
- **Python slice-assignment width is a silent corruptor** (D125):
`out[a:a+8] = b"\x00\x00\x00\x00"` on a `bytearray` does NOT raise — it
deletes `(8 - len(rhs))` bytes, shrinking `out` and shifting everything
after `a` down. A widened pointer/`stan` slot filled with a 4-byte
literal shrank `d88_emit.py`'s output 4 B per pad; once `len(out)` fell
below a later region's write offset, `out[off:off+n] = data` clamped to
an empty range at the current end and *inserted* there — string blobs
drifted, boundpad names truncated (`p138d2`→`8d2`), `stanPackId` rejected
→ door `model=NULL` → crash. When emitting into a pre-sized bytearray,
every `out[a:b] = rhs` MUST have `len(rhs) == b - a`; grep the emitter
for width-mismatched slice writes when a converted region lands wrong.
- A one-shot "sync patched values from the ROM copy into a compiled
shadow array" pass must key its slot-detection on a field that is still
intact *at sync time*. D124: `gimgSyncCompiledGlobalDLs` looked for the
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env python3
"""D125 diagnostic: does the propDefs blob in the *emitted* pccg sidecar
(after RZ round-trip) byte-match tools_pc/d88_propdefs.convert_stream()?
Tests the M-14 claim that the RAM propDefs is "zeros/garbage after record 0".
If this script says MATCH, the corruption is downstream of the sidecar
(runtime rebase / bank alloc / sizepropdef walk); if MISMATCH, it is in
d88_emit.py's emit/placement or the RZ recompress.
"""
import csv, struct, zlib, sys
from d88_propdefs import convert_stream, PROPDEF_PC_BYTES
REGION = "ntsc-final"
rom = open(f"data/ge007.{REGION}.z64", "rb").read()
fl = {}
for r in csv.reader(open("scripts/filelist.u.csv")):
if len(r) >= 3 and r[2]:
b = r[2].rsplit("/", 1)[-1]
b = b[:-4] if b.endswith(".bin") else b
fl[b] = (int(r[0]), int(r[1]))
man = {}
for r in csv.reader(open(f"data/pccg-{REGION}/manifest.csv")):
if r and r[0].startswith("Usetup"):
man[r[0]] = (int(r[1]), int(r[2]))
sidebin = open(f"data/pccg-{REGION}/pccg.bin", "rb").read()
be = lambda b, o: struct.unpack_from(">I", b, o)[0]
targets = sys.argv[1:] or sorted(man)
for nm in targets:
a, s = fl[nm]
dec_n64 = zlib.decompress(rom[a:a + s][2:], -15)
pd_n64, intro_n64 = be(dec_n64, 12), be(dec_n64, 8)
want, n64len, pclen = convert_stream(dec_n64, pd_n64, intro_n64)
off, size = man[nm]
dec_pc = zlib.decompress(sidebin[off:off + size][2:], -15)
# PC header: 8-byte slots, low 4 = LE offset
pd_pc = struct.unpack_from("<I", dec_pc, 8 * 3)[0] # propDefs = field 3
intro_pc = struct.unpack_from("<I", dec_pc, 8 * 2)[0] # intro = field 2
got = dec_pc[pd_pc:pd_pc + len(want)]
if got == want:
print(f"MATCH {nm:18} pd@{pd_pc:#x} len={len(want)}")
else:
# first differing byte + which record it lands in
d = next((i for i in range(min(len(got), len(want))) if got[i] != want[i]), None)
rec, o = 0, 0
while o < (d or 0):
t = want[o + 3]
o += PROPDEF_PC_BYTES.get(t, 4)
rec += 1
print(f"MISMATCH {nm:18} pd@{pd_pc:#x} want={len(want)} got={len(got)} "
f"first diff @+{d} (~record {rec}, off {o:#x}); "
f"gotlen_vs_intro: {intro_pc - pd_pc} vs pclen {pclen}")
+4 -1
View File
@@ -371,7 +371,10 @@ def convert_usetup(name, src):
out[dst_o + 0x2c:dst_o + 0x30] = b"\x00\x00\x00\x00"
# stan: dead-on-load (init_pathtable_something always sets it before
# use, D88 same class as D79) -- zero it, never read from file.
out[dst_o + 0x30:dst_o + 0x38] = b"\x00\x00\x00\x00"
# NOTE: this slice is 8 bytes wide (widened stan ptr); the RHS MUST be
# 8 NUL bytes -- a 4-byte literal here makes `out` SHRINK by 4 per pad
# via Python slice-assign semantics, drifting every later region (D125).
out[dst_o + 0x30:dst_o + 0x38] = b"\x00\x00\x00\x00\x00\x00\x00\x00"
if has_bbox:
for k in range(6):
v = be32(src, src_o + 0x2c + 4 * k)