mirror of
https://github.com/jkdansereau/goldeneye-pc-port.git
synced 2026-10-23 15:52:42 +02:00
D125: fix d88_emit.py:374 8-byte slice / 4-byte literal (Aztec C3 crash)
emit_pad()'s widened-stan zero-fill assigned a 4-byte literal to an 8-byte bytearray slice. Python slice-assign with a short RHS silently deletes the surplus bytes, so `out` shrank 4 B per pad+boundpad; once len(out) dropped below a later leaf write's target offset, the write clamped to empty-at-end and inserted there, drifting/truncating the boundpad plink-name blob (Aztec pad33 "p138d2" -> "8d2") -> stanPackId reject -> NULL stan -> door model=NULL -> crash propobj.c:13601. Fix: 8 NUL bytes (only width-mismatched out[ write in the file). All 21 Usetup*Z sidecars regenerated. Aztec (-level_28) now PASSES (was C3 CRASH); Bunker1/Silo unregressed; d125_check.py 21/21 MATCH. Sweep 12 -> 13/21. Bunker2 (-level_27) still crashes: separate DOOR-tail linkedDoor layout bug in d88_propdefs.py (C3r) -- linkedDoor is resolved from linkedDoorOffset (read-before-write int id, D123 pattern). Folded into the converter write-audit. M-16 probes reverted; d125_check.py kept as the offline regression guard. Generalisable quirk added to PORT-LEARNINGS.md §C.
This commit is contained in:
1 parent
710d0b12b5
commit
060a7252e5
6 files changed
+279
-71
No files matched your search
+37
-6
@@ -78,15 +78,46 @@ Full workstream breakdown (WS1–WS6) in `docs/PLAN-linear-level-sweep.md`.
|
||||
`HANDOFF.md`, `AGENTS.md`, `CLAUDE.md`): the D121 + D122 entries in
|
||||
`docs/PCPortResearch.md` §F/§H and this file's edits. Commit or fold in.
|
||||
|
||||
## Next task — WS4/WS5: drive the 21-level sweep
|
||||
## Next task — converter write-audit (C3r / C4 / C5 / C6)
|
||||
|
||||
**WS4 matrix DONE** — `docs/LEVEL-STATUS.md`. **7 / 21 PASS** (Bunker1 09,
|
||||
Silo 20, Archives 24, Train 25, Caverns 39, Egypt 32, Cuba 54). 14 crashes
|
||||
in 7 classes C1–C7. Reusable sweep runner: `tools_pc/level_sweep.sh`
|
||||
(bare `-level_XX`, `GE_PCDUMP="80-260:40"`, 24 s watchdog,
|
||||
`taskkill //F //IM ge007.x86_64.exe`; **`export PATH=".../mingw64/bin:$PATH"`
|
||||
**D125 is LANDED (M-16b).** `tools_pc/d88_emit.py:374` now writes 8 NUL
|
||||
bytes; all 21 `Usetup*Z` sidecars regen'd; **Aztec `-level_28` PASSES**
|
||||
(was C3 crash), Bunker1/Silo unregressed, `d125_check.py` 21/21 MATCH.
|
||||
M-16 probes reverted, scratch deleted, `d125_check.py` kept. Sweep 12 →
|
||||
13/21. Full write-up: §H D125 M-16b.
|
||||
|
||||
The last 3 sessions' converter bugs are one family — width/endianness/
|
||||
stride errors in `tools_pc/d*_emit.py` (D112 `put_f32`, D120 opcode-0x18,
|
||||
D125 slice width). Pi audited every `out[` write in `d88_emit.py`;
|
||||
`d43_emit.py` + the BG sidecar emitters never got that pass, and the open
|
||||
crashes line up with exactly those files:
|
||||
|
||||
- **C3r** Bunker2 — DOOR-tail `linkedDoorOffset`/`linkedDoor` layout in
|
||||
`d88_propdefs.py` vs compiled PC `DoorRecord` (`offsetof`/`sizeof`
|
||||
cross-check; D123 read-before-write-id pattern).
|
||||
- **C6** Surface2 — `PitemZ_entries[modelid]` (D122/D123 cont.,
|
||||
`d43_emit.py` / `d88_propdefs.py`).
|
||||
- **C4** Depot — BG tile/room table (`prop.c:902`).
|
||||
- **C5** Control — BG portal `offset_portal` unresolved/BE (`bg.c:5723`).
|
||||
|
||||
Audit `d43_emit.py` + BG sidecar emit paths for slice-width mismatches,
|
||||
un-byteswapped fields, N64-vs-PC stride math. Then the structural piece:
|
||||
**C2-GDL** (Runway + Facility) — `docs/BRIEF-C2gdl-model-reloc.md`, its
|
||||
own focused session. Defer C2m Jungle (matrix), C7 Surface1 (add a narrow
|
||||
`#ifdef PORT` `sndSetupSound` load guard only — audio is Phase 3).
|
||||
|
||||
Sweep note: `level_sweep.sh` was flaky this session (spurious NO-FRAMES on
|
||||
known-good levels under machine load / the 24 s watchdog + D117
|
||||
nondeterminism). Verify individual levels with a 35–45 s window and
|
||||
`GE_PCDUMP="60-500:40"` when a sweep row looks wrong.
|
||||
|
||||
Sweep runner: `tools_pc/level_sweep.sh` (bare `-level_XX`, `GE_PCDUMP="80-260:40"`,
|
||||
24 s watchdog, `taskkill //F //IM ge007.x86_64.exe`; **`export PATH=".../mingw64/bin:$PATH"`
|
||||
before `addr2line`** — the script's own symbolication no-op'd without it).
|
||||
|
||||
The M-13 matrix below is partially superseded by M-14 (see §H D125 for the current
|
||||
12/21 status) — treat §H as authoritative.
|
||||
|
||||
| Class | Levels | Site |
|
||||
|---|---|---|
|
||||
| **C1** | Dam 33, Runway 35, Frigate 26, Statue 22, Streets 29, Cradle 41 | `chrIsNotDeadOrShot` chraction.c:4483 — `self` = AI-list rodata ptr |
|
||||
|
||||
+47
-41
@@ -27,16 +27,19 @@ Build = `f2beae4b` + `tools_pc/d88_propdefs.py` (D123) + `port/src/gimgfixup.c`
|
||||
| Runway | 35 | CRASH | `import_texture_i8` gfx_pc.cpp:821 (fault 0x721b8ee8) — same as Facility | **C2** |
|
||||
| Facility | 34 | CRASH | `import_texture_i8` gfx_pc.cpp:821 (fault 0x72181ee8) — model-GDL relocation align bug, D124, NOT fixed | **C2** |
|
||||
| Jungle | 37 | CRASH | `gfx_sp_matrix` gfx_pc.cpp:1046 (fault 0x401c68e0) — C2 texture crash fixed (D124), now explosion-DL `G_MTX` (D75/matrix family) | **C2m** |
|
||||
| Aztec | 28 | CRASH | `door7F054FB4` propobj.c:13601 (`door->model->obj->RootNode->Child->Child`) | **C3** |
|
||||
| Bunker2 | 27 | CRASH | `door7F054FB4` propobj.c:13523 (door displacement list walk) | **C3** |
|
||||
| Aztec | 28 | **PASS** 91.0% | — (C3 fixed, D125) | — |
|
||||
| Bunker2 | 27 | CRASH | `door7F054FB4` propobj.c:13536 (`var_s1->openPosition`, `var_s1`=0xffff.. from `linkedDoor`) | **C3r** |
|
||||
| Depot | 30 | CRASH | `sub_GAME_7F00324C` prop.c:902 (`sp4C->room` after `walkTilesBetweenPoints`) | **C4** |
|
||||
| Control | 23 | CRASH | `sub_GAME_7F0BA2D4` bg.c:5723 (`portal_pts->numPoints`, via chrprop.c:62) | **C5** |
|
||||
| Surface2 | 43 | CRASH | `modelLoad` loadobjectmodel.c:393 (`PitemZ_entries[modelid].header->RootNode`) | **C6** |
|
||||
| Surface1 | 36 | CRASH | `sndSetupSound` snd.c:653 (fault 0x56220001…, ascii-ish) | **C7** |
|
||||
|
||||
**12 / 21 PASS.** D123 cleared C1 on all 6 (Dam/Frigate/Statue/Cradle/Streets
|
||||
**13 / 21 PASS.** D123 cleared C1 on all 6 (Dam/Frigate/Statue/Cradle/Streets
|
||||
PASS; Runway falls through to C2). D124 cleared Jungle's texture crash but
|
||||
it now hits an explosion-DL matrix crash. 9 crashes remain in 6 classes.
|
||||
it now hits an explosion-DL matrix crash. D125 (`d88_emit.py:374` 8-byte
|
||||
slice / 4-byte literal → boundpad-name blob drift) cleared C3-Aztec;
|
||||
Bunker2 falls through to a separate DOOR-tail `linkedDoor` layout bug
|
||||
(C3r). 8 crashes remain in 6 classes.
|
||||
|
||||
Notes: Streets passed this run but the C1 agent saw a `gfx_sp_matrix`
|
||||
crash — likely the same explosion-DL `G_MTX` (C2m) as Jungle, timing-
|
||||
@@ -98,41 +101,44 @@ Full write-up + next-step probe plan: PCPortResearch §F D124-Facility
|
||||
addendum. Likely fix site: the `d43_emit.py`/`pcmodels` sidecar's model
|
||||
texture-blob offsets (N64 8B-Gfx vs PC 16B-Gfx GDL extent).
|
||||
|
||||
### C3/C6 update (M-14, D125) — NOT a converter type-gap
|
||||
Root cause narrowed: the converted `propDefs` blob in RAM does **not** match
|
||||
`tools_pc/d88_propdefs.py`'s offline output (after record 0 it is
|
||||
zeros/garbage), so the runtime `sizepropdef()` walk drifts +101 records by
|
||||
the first door → `setupDoor`/`modelLoad` get the wrong `pdefIndex`/modelid.
|
||||
Histogram diff is clean (no type absent from passing levels). Suspect a
|
||||
destination-offset / `pd_end` / header-reloc bug in `tools_pc/d88_emit.py`'s
|
||||
propdefs emit path (retrofitted post-D122; docstring still says "opaque
|
||||
passthrough"). Not fixed — see §F/§H D125. **d88_propdefs.py `convert_stream`
|
||||
itself and `sizepropdef()` PORT strides are verified correct.**
|
||||
### C3/C6 update (M-14/M-15, D125) — offline pipeline RULED OUT, bug is runtime
|
||||
Symptom: converted `propDefs` blob in RAM does not match the offline output
|
||||
(after record 0 it is zeros/garbage), so the runtime `sizepropdef()` walk
|
||||
drifts and `setupDoor`/`modelLoad` get the wrong `pdefIndex`/modelid.
|
||||
|
||||
**Overseer note (M-13, unverified — first task next session):** likely a
|
||||
pass-1 delta / region-`end` mismatch in `d88_emit.py` ~L308–340. The
|
||||
tiled `propdefs` region's `end` (pd_end, from the region tiling) may not
|
||||
equal `pd_start + convert_propdefs()._n64len`. Pass 1 does
|
||||
`cum += len(propdefs_pc) - (end - start)` with that `end`; if it is
|
||||
**shorter** than the real record-stream length, the cumulative delta is
|
||||
under-counted, every subsequent region (aistream / opaque) is placed too
|
||||
low, and the emit loop's `out[do:...] = src[...]` for the next region
|
||||
**overwrites the tail of `propdefs_pc`** — leaving record 0 intact and the
|
||||
rest clobbered, exactly the observed symptom. Cheap confirm: print
|
||||
`pd_start`, `pd_end`, `_n64len`, `len(propdefs_pc)`, and the next region's
|
||||
`(start, do)` for `UsetupsevbZ`; check `pd_end == pd_start + _n64len`.
|
||||
Fix = set the propdefs region `end` to `pd_start + _n64len` (or make
|
||||
`convert_propdefs` and the tiler agree on the boundary).
|
||||
**M-15: the offline side is proven correct.** `tools_pc/d125_check.py`
|
||||
byte-compares the emitted `pccg.bin` propDefs slice (post-RZ-roundtrip, at
|
||||
the relocated header offset) to `convert_stream()` → **MATCH for all 21
|
||||
levels**. The M-13 overseer hypothesis (pass-1 delta / `pd_end` mismatch in
|
||||
`d88_emit.py` ~L308–340) is **disproven**: instrumented run shows
|
||||
`tiled_pd_end == H[intro]` and `_n64len == pd_end - pd_start` exactly for
|
||||
all 21, next region always `intro`. `sizepropdef()` PORT strides re-checked
|
||||
vs `PROPDEF_PC_BYTES/4` — all 28 match.
|
||||
|
||||
### C3 — door model / ModelNode walk (2 levels: Aztec, Bunker2)
|
||||
`propobj.c:13601` derefs `door->model->obj->RootNode->Child->Child`;
|
||||
`:13523` walks the `linkedDoor` list. Fault addrs 0x10 / 0x8 → a null-ish
|
||||
base + field offset. Door prop model not loaded, or `door->model` /
|
||||
`->linkedDoor` pointer-width mis-set. Same family as D122 (propDef obj
|
||||
half-swap) — Aztec/Bunker2 emit a door subtype BUNKER1/Silo don't, or the
|
||||
door propDef `linkedDoor`/`model` field is pointer-width-sized wrong.
|
||||
Files: `src/game/propobj.c`, `tools_pc/d88_propdefs.py` (door handler),
|
||||
`src/game/prop.c`.
|
||||
**→ The corruption is at/after runtime load.** Prime suspect:
|
||||
`decompressdata()` truncation in `port/src/rzdecomp.c` for the larger
|
||||
converted setup files (the `while (ret == Z_OK)` inflate loop), or the
|
||||
STAGE bank buffer / `langLoadToAddr` (`prop.c:1274`) overwriting the tail.
|
||||
Next step: probe `decompressdata()` `ret`/`produced` vs expected size for
|
||||
`UsetupsevbZ`. See §F/§H D125.
|
||||
|
||||
### C3 — Aztec FIXED (D125); Bunker2 residual (C3r)
|
||||
**Aztec** was D125 (boundpad plink-name blob drift from the
|
||||
`d88_emit.py:374` slice-width bug corrupting `stanPackId` → NULL stan →
|
||||
door `model=NULL`). Fixed + regen'd, Aztec PASSES.
|
||||
|
||||
**Bunker2 (C3r)** — `door7F054FB4` `propobj.c:13536`, `var_s1` = `0xffff..`
|
||||
walked from `linkedDoor`. `door->linkedDoor` is resolved at setup from
|
||||
`door->linkedDoorOffset` (`prop.c:1204-1206`, N64 struct word 32 / 0x80) —
|
||||
a read-before-write int id, D123's `ailist` pattern. `d88_propdefs.py`'s
|
||||
DOOR handler (`DOOR_TAIL_PTR_WORDS`, tail loop from word 32) vs the
|
||||
compiled PC `DoorRecord` layout is the suspect: either `linkedDoorOffset`
|
||||
lands at the wrong PC byte, or a tail pointer word's 8-align cursor
|
||||
diverges from the compiler's struct layout. Bunker2 has linked
|
||||
double-doors; Aztec's doors are singletons (`linkedDoorOffset==0`) so it
|
||||
never exercised this path. Needs `offsetof`/`sizeof` cross-check of PC
|
||||
`DoorRecord` vs the converter cursor. Files: `tools_pc/d88_propdefs.py`,
|
||||
`src/bondtypes.h` (DoorRecord), `src/game/prop.c`, `src/game/propobj.c`.
|
||||
|
||||
### C4 — prop tile-walk room deref (1: Depot)
|
||||
`prop.c:902` — `*arg1 = sp4C->room` where `sp4C` came back bad from
|
||||
@@ -156,13 +162,13 @@ id from a propDef record). Fault addr 0x0. Files: `tools_pc/d88_propdefs.py`,
|
||||
crash on load, not silence. May be dodge-able with a narrow guard until
|
||||
audio lands. Files: `src/snd.c`, `port/src/` audio stubs.
|
||||
|
||||
## Next (9 crashes, 6 classes — priority order)
|
||||
## Next (8 crashes, 6 classes — priority order)
|
||||
1. **C2** Runway + Facility — model-GDL relocation align bug in
|
||||
`objecthandler_2.c`/`texLoadFromGdl` (D80/D82/D83 area). Shared infra,
|
||||
2 levels + likely helps Jungle/Streets. Biggest win.
|
||||
2. **C3** Aztec + Bunker2 — door model / `linkedDoor`; brief
|
||||
`docs/BRIEF-C3-C6-prop-model.md` (dispatch after any `d88_propdefs.py`
|
||||
work settles).
|
||||
2. **C3r** Bunker2 — DOOR-tail `linkedDoorOffset`/`linkedDoor` layout in
|
||||
`d88_propdefs.py` vs compiled PC `DoorRecord` (Aztec cleared by D125);
|
||||
fold into the converter write-audit. Brief `docs/BRIEF-C3-C6-prop-model.md`.
|
||||
3. **C2m** Jungle (+ Streets intermittent) — explosion-DL `G_MTX`
|
||||
(`gfx_sp_matrix`), D75/matrix family.
|
||||
4. **C6** Surface2 — `PitemZ_entries` modelid (D122/D123 continuation).
|
||||
|
||||
+125
-23
@@ -818,7 +818,7 @@ covers D24–D69; the log continues in §H (D32 procedure, D70–D121).
|
||||
| D121 | WS1 frictionless per-level boot: bare `-level_XX` injects its `memallocstringtable[]` `-m*` row (`#ifdef PORT` in `boss.c`) | resolved |
|
||||
| D122 | per-level prop/item model-load crash (Dam/Facility/Runway `modelLoad`/`modelInitRwData`): `d88_propdefs.py` had no handler for 6 ObjectRecord-derived propDef types (47/39/40/45/13/20) → generic arm half-swapped the `[s16 obj][s16 pad]` word → OOB `PitemZ_entries[]` | converter fixed; residual chr/fast3d crashes on those levels are separate |
|
||||
| D123 | crash class C1: `chrIsNotDeadOrShot` NULL deref on 6 levels (Dam/Runway/Frigate/Statue/Streets/Cradle). D122's `OBJ_TAIL_DESC` zeroed the widened `VehichleRecord/AircraftRecord.ailist` slot (w32), but `prop.c:1764/1786` reads a pre-populated int AI-list id there → `ailistFindById(0)` → `GAILIST_AIM_AT_BOND` → `ai()` runs a CHR aim list with `ChrEntityp==NULL` | converter fixed (`OBJ_ID_WORDS`); C1 cleared on all 6, residual crashes are fast3d (C2) |
|
||||
| D125 | crash classes C3+C6 (Aztec/Bunker2/Surface2): the converted `propDefs` blob in RAM ≠ `d88_propdefs.py` offline output — after record 0 it is zeros/garbage, so the `sizepropdef()` walk drifts (+101 records by the first door) and every `pdefIndex` lookup (`setupDoor`→`linkedDoor`, `modelLoad` modelid) hits the wrong record. NOT a forgotten-type gap (histogram diff clean). Suspect a destination-offset / `pd_end` / header-reloc bug in `tools_pc/d88_emit.py`'s propdefs emit, not in the converter | **not fixed** — investigation cut for usage reset; next: diff pccg.bin bytes vs `convert_stream`, trace `pd_start/pd_end/do` in d88_emit.py |
|
||||
| D125 | crash classes C3+C6 (Aztec/Bunker2/Surface2): **root cause found (M-16)** — `tools_pc/d88_emit.py:374` assigns a 4-byte literal to an 8-byte slice (`out[dst_o+0x30:dst_o+0x38] = b"\x00…"`, the stan zero-fill in `emit_pad`); every pad/boundpad record silently shrinks the output bytearray by 4 B, and once the tail falls below the boundpad plink string blob, later verbatim leaf writes hit Python out-of-range slice semantics (insert at current end, not relocated offset) → boundpad names drift/truncate (Aztec pad33 `p138d2`→`8d2`) → `stanPackId()` reject → `getposstan()` NULL stan → `setupDoor` leaves door `model=NULL` → crash `propobj.c:13601`. M-14's "propDefs zeros in RAM" was a misread; sidecar propdefs were always correct (M-15) and post-load RAM matches the sidecar byte-for-byte (M-16) | **resolved (M-16b)** — line 374 → 8 NULs; all 21 sidecars regen'd; **Aztec `-level_28` now PASSES** (was C3 CRASH). Bunker2 falls through to a separate DOOR-tail `linkedDoor` layout bug (C3 residual) |
|
||||
| D124 | crash class C2: fast3d bad texture pointer. **Jungle** (`0xabcd0824`): `gimgSyncCompiledGlobalDLs()` slot-detect keyed on the post-fixup marker, which `texLoad()` had already overwritten → compiled `globalDL_0xNNN` explosion DLs kept link-time `IMAGESEG` words → latent on every level, tripped by the first explosion-DL draw. **Facility** (`0x72181ee8`): separate — model/prop GDL from the `texLoadFromGdl()`/`sub_GAME_7F0762E0` relocation path writes non-16-aligned `dst` (N64 8B vs PC 16B `Gfx` stride mix in `objecthandler_2.c`), open D80/D82/D83 area | Jungle fixed (`port/src/gimgfixup.c`); Facility diagnosed, not fixed |
|
||||
|
||||
Phase 2 replaced the Phase-1 demo loop with the real `mainproc()` on real OS
|
||||
@@ -4221,29 +4221,52 @@ levels emit no propDef type absent from the 12 passing levels — TINTED_GLASS
|
||||
and AUTOGUN appear in passing Caverns/Control too. **This is not a
|
||||
forgotten-type converter gap like D122/D123.**
|
||||
|
||||
*Best hypothesis (confidence: mechanism high, exact fault medium).* The
|
||||
converted `propdefs_pc` bytes produced in `tools_pc/d88_emit.py`
|
||||
(`convert_propdefs`, line 320) are **not landing in the sidecar at the
|
||||
offset `g_CurrentSetup.propDefs` resolves to**, OR the `propDefs` header
|
||||
field is not being relocated to the converted blob's new offset, OR
|
||||
`pd_end` (the `'propdefs'` region end from the region-tiling at
|
||||
`d88_emit.py:315-318`) is computed wrong so only the first record is
|
||||
converted and the rest of the region is left as `bytearray(total_new)`
|
||||
zero-fill. The "idx0 correct, then zeros" signature points hard at a
|
||||
destination-offset / length bug in `d88_emit.py`'s propdefs emit
|
||||
(lines 266-267, 308-338, 471-476), not in `d88_propdefs.py`. Note
|
||||
`d88_emit.py`'s module docstring still describes propDefs as an "opaque
|
||||
passthrough blob" (stale post-D122) — the emit path was retrofitted and
|
||||
may not have been fully reconciled with the region-tiling / delta-reloc
|
||||
logic.
|
||||
*Best hypothesis (M-14) — DISPROVEN (session M-15).* M-14 guessed the
|
||||
converted `propdefs_pc` bytes were not landing in the sidecar at the
|
||||
relocated `propDefs` offset (a `d88_emit.py` region-tiling / delta-reloc
|
||||
bug, lines ~308-338/471-476). The M-13 overseer note refined this to a
|
||||
pass-1 delta / region-`end` mismatch (`pd_end != pd_start + _n64len`).
|
||||
**Both are wrong.** New diagnostic `tools_pc/d125_check.py` decompresses
|
||||
the *emitted* `data/pccg-ntsc-final/pccg.bin` entry for every Usetup*Z,
|
||||
reads the relocated `propDefs` header field, and byte-compares that slice
|
||||
to a fresh `convert_stream()`:
|
||||
|
||||
*Next step.* Dump the actual bytes at `data/pccg-ntsc-final/pccg.bin` for
|
||||
`UsetupsevbZ`'s propDefs region and compare to `convert_stream` output;
|
||||
add a `[D88emit]` print of `pd_start / pd_end / len(propdefs_pc) / do`
|
||||
(destination offset) in `d88_emit.py`; check whether the `propDefs`
|
||||
header word in the emitted file points at `do`. Likely a 1-region
|
||||
off-by-one in the sorted region list (intro vs propDefs ordering) or the
|
||||
converted blob written at the pre-growth offset.
|
||||
- **All 21 levels: `MATCH`** (Aztec/Bunker2/Surface2 included). The
|
||||
sidecar's propDefs blob, post-RZ-roundtrip, is byte-identical to the
|
||||
converter output, placed exactly where the relocated header field
|
||||
points.
|
||||
- Instrumented `d88_emit.py` (D125_DEBUG, since reverted): for all 21,
|
||||
`tiled_pd_end == H[intro]` exactly, `_n64len == pd_end - pd_start`
|
||||
exactly, the region after `propdefs` in the sorted list is always
|
||||
`intro`, and the growth `pclen - n64len` fed to `cum` is correct.
|
||||
There is **no** region-end/delta bug at `d88_emit.py:308-343`.
|
||||
- Full re-cross-check of `sizepropdef()` PORT strides vs
|
||||
`PROPDEF_PC_BYTES/4`: all 28 types match (as M-14 already found).
|
||||
|
||||
**Therefore the offline pipeline (converter → emit → RZ compress) is
|
||||
correct end-to-end. The "idx0 correct, then zeros" RAM signature M-14
|
||||
observed must originate at or after runtime load** — candidates, in
|
||||
rough order: (a) `decompressdata()` truncation — `port/src/rzdecomp.c`
|
||||
inflates with `Z_FINISH` and a 4 MiB `avail_out`, loops while `Z_OK`;
|
||||
if the deflate stream for these larger converted files hits `Z_BUF_ERROR`
|
||||
or `Z_STREAM_END` mid-output the `while (ret == Z_OK)` exits early and
|
||||
the tail of `dst` stays zero (needs a probe: log `ret` + `produced` vs
|
||||
expected decompressed size for `UsetupsevbZ`); (b) the STAGE bank alloc /
|
||||
`mempAllocBytesInBank` giving a buffer that is fine for the N64-size file
|
||||
but the converted file is ~15-25% larger — check `mempGetBankSizeLeft`
|
||||
at the `_fileNameLoadToBank(strResource, …, 256, MEMPOOL_STAGE)` call
|
||||
(`prop.c:1271`) vs the decompressed size; (c) the `prop.c:1280-1308`
|
||||
rebase loop or `langLoadToAddr` (`prop.c:1274`, runs right after the
|
||||
load, same bank) overwriting the tail of the setup file. M-14's own
|
||||
runtime dump had `g_CurrentSetup.propDefs = 0x701ce814` (`&15 == 4`).
|
||||
|
||||
*Next step.* Probe `decompressdata()` for `UsetupsevbZ`: print `ret`,
|
||||
`produced`, and the first 64 bytes at `dst + 16` (offline record 1
|
||||
start). If `produced` < the offline decompressed file size → truncation
|
||||
(fix the inflate loop / `avail_in`). If `produced` is correct but the
|
||||
bytes are still zero → something overwrites it post-load (bisect
|
||||
`langLoadToAddr` / the rebase loop). `tools_pc/d125_check.py` stays in
|
||||
tree as the offline-side regression guard.
|
||||
|
||||
*Probes:* **none left in tree** — the two `#if defined(PORT)`/`GE_C3`
|
||||
scratch prints in `src/game/prop.c` (lines ~1206 and ~1865) and the
|
||||
@@ -4253,6 +4276,85 @@ entry) + `docs/LEVEL-STATUS.md` (one line). No code, no converter, no
|
||||
sidecar change — **sidecars do NOT need regen**, tree builds clean at
|
||||
`f2beae4b` + M-13 uncommitted set.
|
||||
|
||||
**D125 addendum (session M-16) — ROOT CAUSE FOUND: `d88_emit.py:374`
|
||||
4-byte literal into an 8-byte slice; per-record buffer shrink; out-of-range
|
||||
leaf writes insert at the current end → boundpad name blob drift.**
|
||||
|
||||
*Crash mechanism (Aztec, `-level_28`), fully traced.* Door idx 226
|
||||
(`objid=307`, `pad=33`) is left with `model=NULL`/`prop=NULL` by
|
||||
`setupDoor()`: its boundpad's plink name in the sidecar is `"8d2"`
|
||||
(original ROM: `"p138d2"`) → `stanPackId()` rejects it (valid = `p`/`q`
|
||||
+ decimal ≤32767 + letter + optional digit 0-7; probe line `D88
|
||||
stanMatchTileName id=8d2 hi=ffff lo=ff` is the failure sentinel) →
|
||||
`init_pathtable_something()` returns 0 with `*tile_stack=NULL` →
|
||||
`getposstan()` nonzero → `setupDoor` sets `door->prop=NULL`, skips
|
||||
`doorInit()` → later tick `door7F054FB4()` (`propobj.c:13601`) derefs
|
||||
`door->model->obj`; `Model.obj` sits at PC offset 0x10 (widened `chr`
|
||||
ptr) → fault addr 0x10, Rcx=0. Linked-door pair: idx225 (`pad=32`, valid)
|
||||
has `linkedDoorOffset=1` → idx226. All 178 regular pads resolve fine; only
|
||||
boundpad names are corrupt.
|
||||
|
||||
*The bug.* `tools_pc/d88_emit.py` `emit_pad()` line 374:
|
||||
`out[dst_o + 0x30:dst_o + 0x38] = b"\x00\x00\x00\x00"` — the stan zero-fill
|
||||
assigns a **4-byte** literal to an **8-byte** slice. Python silently allows
|
||||
in-range short-RHS slice assignment, so each of Aztec's 179 pads + 148
|
||||
boundpads shrinks the output `bytearray` by exactly 4 B (measured: after
|
||||
header 0x14d1c → after pads 0x14a50 → after boundpads 0x14800). Later
|
||||
verbatim leaf writes (the plink string blob, relocated to ~0x147EC+) then
|
||||
target offsets at/past the shrunken tail; for those, Python clamps the
|
||||
slice to an empty range **at the current end** and inserts the data there —
|
||||
each string lands at the buffer tail instead of its relocated offset,
|
||||
drifting earlier with every write. A simulation of exactly this semantics
|
||||
reproduces the observed sidecar names byte-for-byte: pad30 `p139d2`
|
||||
(intact), pad31 `139d2`, pad32 `37d1`, **pad33 `8d2`**, pad34 `2`,
|
||||
pad35 `1`, pad36 `` (empty).
|
||||
|
||||
*Why it was missed.* No error is raised; the final file is only
|
||||
`(n_pads+n_boundpads)*4 − re-extension` bytes short of `total_new`
|
||||
(Aztec: 85276 vs 85145) because later leaf writes re-extend the buffer;
|
||||
`d125_check.py` verified only the propDefs slice. M-14's "propDefs zeros in
|
||||
RAM" signature was a misread — M-15 already proved sidecar propdefs
|
||||
byte-correct, and M-16's post-load RAM dump matched the sidecar byte-for-
|
||||
byte (85,145 B), so runtime load is clean too.
|
||||
|
||||
*Fix (pending).* Line 374 → `b"\x00\x00\x00\x00\x00\x00\x00\x00"` (8 NULs).
|
||||
Audit of every other `out[` write in the file: all remaining slice/RHS
|
||||
pairs are width-matched — this is the only one. **All 21 Usetup*Z sidecars
|
||||
are affected to varying degrees** (shrink = 4×(pads+boundpads) per level;
|
||||
which names drift depends on where the blob lands vs the shrunken tail) —
|
||||
regen all: `python tools_pc/d88_emit.py ntsc-final --regen`, then re-run
|
||||
`tools_pc/level_sweep.sh` (expect C3 Aztec+Bunker2 to clear; currently-
|
||||
passing levels can only improve, but the sweep is the proof).
|
||||
|
||||
*Probes in tree (temporary — remove after fix verified):* `GE_D125`
|
||||
env-gated prints in `port/src/rzdecomp.c` (decompress ret/produced),
|
||||
`src/game/prop.c` (setup header + post-load 0x15000 dump to
|
||||
`d125_setupdump.bin` + post-`setupDoor()` door state), `src/game/propobj.c`
|
||||
(`door7F054FB4` NULL-model detect+skip, `objInit` failure print, plus temp
|
||||
`#ifdef PORT` stdio/stdlib includes). Scratch: `tools_pc/d125_convtest.py`
|
||||
(instrumented converter copy — source of the shrink evidence),
|
||||
`tools_pc/d125_check.py` (**keep** — offline regression guard),
|
||||
`tools_pc/d125_inflate_test.c`, `d125_offs_tmp.c`, `d125_t.c`,
|
||||
`d125_run.log`, `d125_setupdump.bin`, `build-pc/d125conv/`.
|
||||
|
||||
**D125 addendum (session M-16b) — FIX LANDED.** `d88_emit.py:374` changed
|
||||
to 8 NUL bytes (+ a warning comment). Offline proof (Aztec, throwaway
|
||||
`d125_proof.py`): the fixed converter's boundpad plink names 30–36 read
|
||||
`p139d2 p139d2 p837d1 p138d2 p85a2 p1377b1 p504d2` (= original ROM), 147
|
||||
boundpad records recovered (M-14's "46" was the corrupt read — no second
|
||||
bug). `d43/d69/d88 --regen` → 21/21; `d125_check.py` → 21/21 propDefs
|
||||
still MATCH. Direct re-test: **Aztec `-level_28` PASSES** (91.0%, 900+
|
||||
frames, was C3 CRASH at `propobj.c:13601`); Bunker1/Silo unregressed.
|
||||
**Bunker2 `-level_27` still crashes** — moved to `door7F054FB4`
|
||||
`propobj.c:13536` (`var_s1->openPosition`, `var_s1` = `0xffff…` from
|
||||
`linkedDoor`). Separate bug: the DOOR-tail converter in `d88_propdefs.py`
|
||||
(`DOOR_TAIL_PTR_WORDS` / `linkedDoorOffset` word 32) vs the compiled PC
|
||||
`DoorRecord` layout — `linkedDoor` is resolved from `linkedDoorOffset`
|
||||
(`prop.c:1204`), a read-before-write int id like D123's `ailist`; Bunker2
|
||||
has linked double-doors, Aztec's are singletons so it dodged it. Folded
|
||||
into the converter write-audit (C3 residual). Sweep: 12 → 13/21.
|
||||
All M-16 probes reverted, scratch files deleted, `d125_check.py` kept.
|
||||
|
||||
**D124-Facility addendum (session M-14 — partial, NOT fixed, out of time).**
|
||||
Re-instrumented with a `GE_C2GDL` probe in `sub_GAME_7F0762E0`
|
||||
(`objecthandler_2.c`) + a bad-`G_SETTIMG` catcher in fast3d
|
||||
|
||||
@@ -53,6 +53,17 @@ through a converter or a runtime bswap fixup reads scrambled.
|
||||
(StandTile id/room, header mid/tail).
|
||||
- **Rule:** prefer an offline sidecar converter (D43, D69, D88 pattern,
|
||||
`tools_pc/d*_emit.py`) over a runtime fixup for a whole format.
|
||||
- **Python slice-assignment width is a silent corruptor** (D125):
|
||||
`out[a:a+8] = b"\x00\x00\x00\x00"` on a `bytearray` does NOT raise — it
|
||||
deletes `(8 - len(rhs))` bytes, shrinking `out` and shifting everything
|
||||
after `a` down. A widened pointer/`stan` slot filled with a 4-byte
|
||||
literal shrank `d88_emit.py`'s output 4 B per pad; once `len(out)` fell
|
||||
below a later region's write offset, `out[off:off+n] = data` clamped to
|
||||
an empty range at the current end and *inserted* there — string blobs
|
||||
drifted, boundpad names truncated (`p138d2`→`8d2`), `stanPackId` rejected
|
||||
→ door `model=NULL` → crash. When emitting into a pre-sized bytearray,
|
||||
every `out[a:b] = rhs` MUST have `len(rhs) == b - a`; grep the emitter
|
||||
for width-mismatched slice writes when a converted region lands wrong.
|
||||
- A one-shot "sync patched values from the ROM copy into a compiled
|
||||
shadow array" pass must key its slot-detection on a field that is still
|
||||
intact *at sync time*. D124: `gimgSyncCompiledGlobalDLs` looked for the
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env python3
|
||||
"""D125 diagnostic: does the propDefs blob in the *emitted* pccg sidecar
|
||||
(after RZ round-trip) byte-match tools_pc/d88_propdefs.convert_stream()?
|
||||
|
||||
Tests the M-14 claim that the RAM propDefs is "zeros/garbage after record 0".
|
||||
If this script says MATCH, the corruption is downstream of the sidecar
|
||||
(runtime rebase / bank alloc / sizepropdef walk); if MISMATCH, it is in
|
||||
d88_emit.py's emit/placement or the RZ recompress.
|
||||
"""
|
||||
import csv, struct, zlib, sys
|
||||
from d88_propdefs import convert_stream, PROPDEF_PC_BYTES
|
||||
|
||||
REGION = "ntsc-final"
|
||||
rom = open(f"data/ge007.{REGION}.z64", "rb").read()
|
||||
fl = {}
|
||||
for r in csv.reader(open("scripts/filelist.u.csv")):
|
||||
if len(r) >= 3 and r[2]:
|
||||
b = r[2].rsplit("/", 1)[-1]
|
||||
b = b[:-4] if b.endswith(".bin") else b
|
||||
fl[b] = (int(r[0]), int(r[1]))
|
||||
|
||||
man = {}
|
||||
for r in csv.reader(open(f"data/pccg-{REGION}/manifest.csv")):
|
||||
if r and r[0].startswith("Usetup"):
|
||||
man[r[0]] = (int(r[1]), int(r[2]))
|
||||
sidebin = open(f"data/pccg-{REGION}/pccg.bin", "rb").read()
|
||||
|
||||
be = lambda b, o: struct.unpack_from(">I", b, o)[0]
|
||||
targets = sys.argv[1:] or sorted(man)
|
||||
for nm in targets:
|
||||
a, s = fl[nm]
|
||||
dec_n64 = zlib.decompress(rom[a:a + s][2:], -15)
|
||||
pd_n64, intro_n64 = be(dec_n64, 12), be(dec_n64, 8)
|
||||
want, n64len, pclen = convert_stream(dec_n64, pd_n64, intro_n64)
|
||||
|
||||
off, size = man[nm]
|
||||
dec_pc = zlib.decompress(sidebin[off:off + size][2:], -15)
|
||||
# PC header: 8-byte slots, low 4 = LE offset
|
||||
pd_pc = struct.unpack_from("<I", dec_pc, 8 * 3)[0] # propDefs = field 3
|
||||
intro_pc = struct.unpack_from("<I", dec_pc, 8 * 2)[0] # intro = field 2
|
||||
got = dec_pc[pd_pc:pd_pc + len(want)]
|
||||
|
||||
if got == want:
|
||||
print(f"MATCH {nm:18} pd@{pd_pc:#x} len={len(want)}")
|
||||
else:
|
||||
# first differing byte + which record it lands in
|
||||
d = next((i for i in range(min(len(got), len(want))) if got[i] != want[i]), None)
|
||||
rec, o = 0, 0
|
||||
while o < (d or 0):
|
||||
t = want[o + 3]
|
||||
o += PROPDEF_PC_BYTES.get(t, 4)
|
||||
rec += 1
|
||||
print(f"MISMATCH {nm:18} pd@{pd_pc:#x} want={len(want)} got={len(got)} "
|
||||
f"first diff @+{d} (~record {rec}, off {o:#x}); "
|
||||
f"gotlen_vs_intro: {intro_pc - pd_pc} vs pclen {pclen}")
|
||||
@@ -371,7 +371,10 @@ def convert_usetup(name, src):
|
||||
out[dst_o + 0x2c:dst_o + 0x30] = b"\x00\x00\x00\x00"
|
||||
# stan: dead-on-load (init_pathtable_something always sets it before
|
||||
# use, D88 same class as D79) -- zero it, never read from file.
|
||||
out[dst_o + 0x30:dst_o + 0x38] = b"\x00\x00\x00\x00"
|
||||
# NOTE: this slice is 8 bytes wide (widened stan ptr); the RHS MUST be
|
||||
# 8 NUL bytes -- a 4-byte literal here makes `out` SHRINK by 4 per pad
|
||||
# via Python slice-assign semantics, drifting every later region (D125).
|
||||
out[dst_o + 0x30:dst_o + 0x38] = b"\x00\x00\x00\x00\x00\x00\x00\x00"
|
||||
if has_bbox:
|
||||
for k in range(6):
|
||||
v = be32(src, src_o + 0x2c + 4 * k)
|
||||
|
||||
Reference in new issue
Block a user