diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index d6fc151a..aa11eb08 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -1,23 +1,28 @@ # Handoff brief — GoldenEye 007 PC port (Phase 1.5 done, Phase 2: first frames render) _Paste-ready briefing for the next agent session. The authoritative detail lives in -`AGENTS.md` and `docs/PCPortResearch.md` (§H handoff, §F/D31–D43 findings); this is -the summary + the immediate task._ +`AGENTS.md` and `docs/PCPortResearch.md` (§H handoff, §F/D31–D44 findings); this is +the summary + the immediate tasks._ ## Your job -Boot is complete AND **real frames render** (`[NOTE] frame N rendered in … us`). The -current blocker is **D43**: model-file loading ABI mismatch — stage object load faults -in `modelPromoteNodeOffsetsToPointers()` (src/game/model.c:5688) because ROM model files -are serialized with N64 layout (4-byte pointer fields) but read on PC as 8-byte-pointer -structs. Fix it with a D37-style re-layout in `port/src/romdata.c`, then keep pushing -rendering forward (more asset types will fault the same way). Work agentically — fix → -build → verify under gdb → commit at each working milestone. +Two tasks, in order: +1. **D44 (quick win, ~15 min):** one-line fix to the crash handler so every crash + auto-logs a backtrace — see Immediate task 1. This shrinks the D43 debug loop + from "gdb launch session per fault" to "read ge007.crash.log". +2. **D43 (main task):** model-file loading ABI mismatch — stage object load faults + in `modelPromoteNodeOffsetsToPointers()` (src/game/model.c:5688) because ROM model + files are serialized with N64 layout (4-byte pointer fields) but read on PC as + 8-byte-pointer structs. Fix it with a D37-style re-layout in `port/src/romdata.c`, + then keep pushing rendering forward (more asset types will fault the same way). + +Work agentically — fix → build (~5 s) → verify → commit at each working milestone. +Commit message style: `PC port: — `. ## Read first (authoritative, in order) 1. `AGENTS.md` — non-negotiables, critical files, build, verification ritual, phase status. -2. `docs/PCPortResearch.md` §H (handoff + plan), then **§F/D43** (the open model-file - task) and **D37** (the libaudio bank-tree re-layout — the pattern to copy), D39–D42 - for the first-render fixes, D31–D38 for background. +2. `docs/PCPortResearch.md` §H (handoff + plan), then **§F/D44** (the quick win), + **D43** (the open model-file task), **D37** (the libaudio bank-tree re-layout — + the pattern to copy), D39–D42 for the first-render fixes, D31–D38 for background. ## Non-negotiables (the ones that bite) - N64 build files (`Makefile`, `tools/`, `rsp/`, `ld/`) — never touch. @@ -27,35 +32,47 @@ build → verify under gdb → commit at each working milestone. as a D3x finding. No behavior changes. - Region macros in `CMakeLists.txt` must mirror the N64 Makefile exactly. -## Current state (verified this session) +## Current state (verified live this session, 2026-08-22) +- **Re-ran the committed build:** frames 1–4 render (`[NOTE] frame N rendered in … us`, + frame 1 ~21 ms), then FATAL `0xc0000005` at PC `0x14007a31a` = + `modelPromoteNodeOffsetsToPointers()` model.c:5688 — exactly D43. The black screen you + see is the clear color only: display lists execute but no stage geometry has loaded yet. +- **Dev loop measured (16-core box):** full clean rebuild ≈ **5 s** wall; launch → crash + ≈ **3.7 s**. Neither is a bottleneck — *diagnosis* is (gdb launch-only mode + manual + addr2line arithmetic per fault). ccache and boot fast-forward are NOT worth it. +- **D44 identified + root-caused (fix pending):** `crashStackTraceSym()` in + `port/src/crash.c` calls `GetCurrentThreadStackLimits(low, high)` — passing NULL pointer + *values* instead of `&low, &high`. The API writes to address 0x0 *inside the SEH filter* + → the process dies before printing the backtrace. That's why `ge007.crash.log` contains + only Phase 1 (EXCEPTION/PC/MODULE lines) and no `BACKTRACE:` section, and why every D3x + fault so far cost a gdb session. The build already uses `-fno-omit-frame-pointer` + (CMakeLists.txt:193), so the EBP-chain walk will work once the call is fixed. - **ROM dump verified:** byte-identical to the No-Intro good dump, header CRCs re-compute clean (`tools_pc/romverify.c`). Fingerprint + provenance in Environment below. -- **Frames render.** The chain of fixes that got us here (all documented in §F): - - **D31–D37:** boot chain — real-zlib file decompression, animation-table ABI+endianness, - ANIM_DATA lvalues, music seq-table fixup, music-heap sizing, libaudio bank-tree - re-layout (`romdataFixupAudioBank`). - - **D38:** implicit function declarations truncated pointer returns on x86-64 → - `port/include/pc_protos.h` prototype shim (398 decls; generator `scripts/gen_pcprotos.py`; - anchored in `port/shim/PR/ucode.h`; C-only). Build is now warning-clean. - - **D39:** Globalimagetable rebasing — `texReset()`'s `globalbank_rdram_offset + (u32)&sym` - idiom needs the N64 `(u32)&sym == 0x02000000 + off` value; PC branch uses a per-symbol - offset enum + `GIMG_OFF()` in `src/game/image_bank.c`. Segment end markers extended to - the true 0x13F8 size (the CSV's `Globalimagetable.bin` entry is truncated by 0x930). - - **D40:** `g_ModelHitEntries` BSS pool was N64-sized (12 KB for 600×20 B entries); the - 40-byte PC stride overflowed .bss and clobbered `is_ramrom_flag`. PC branch declares a - `600 * sizeof(ModelHitEntry)` pool. - - **D41:** WGL contexts are current on one thread only; the host main thread created + - held the context while the game thread renders. `videoInit()` now calls - `gfx_sdl_release_context()` after init; the game thread re-binds per frame. NOTE: this - SDL2 build's `SDL_GL_MakeCurrent` return value is an inverted ABI artifact (0 = success, - -1/255 = failure) — do NOT branch on it. - - **D42:** rsp.c's XOR toggle of `g_gfxTaskSettingsList` truncated exe pointers to u32 → - explicit toggle under `#if defined(__x86_64__)`. +- Frames render because of D31–D42 (all documented in §F): boot chain (real-zlib + decompression, animation-table ABI+endianness, ANIM_DATA lvalues, music seq-table fixup, + music-heap sizing, libaudio bank-tree re-layout), D38 prototype shim (warning-clean + build), D39 Globalimagetable rebasing (`GIMG_OFF()` idiom), D40 ModelHitEntry pool + sizing, D41 WGL single-thread context currency (`gfx_sdl_release_context()`), D42 + rsp.c pointer-toggle truncation. -## Immediate task — D43: model-file loading ABI mismatch +## Immediate task 1 — D44: crash-handler one-liner (do this FIRST) +1. In `port/src/crash.c`, function `crashStackTraceSym()`: change + `GetCurrentThreadStackLimits(low, high)` → `GetCurrentThreadStackLimits(&low, &high)`. + (Phase 1's `crashStackTraceRaw` already passes `&low, &high` — mirror it.) +2. Rebuild (`./build-pc.sh ntsc-final`, ~5 s), run the exe unmodified. Expect a + `BACKTRACE:` section on stdout AND in `ge007.crash.log` after the Phase 1 lines. +3. Confirm the logged backtrace shows the D43 chain: #00 at model.c:5688, then + `load_object_fill_header` / objecthandler frames above it. That proves the whole + diagnosis loop works without gdb. (If a frame's symbol is `?`, that's fine — + addr2line offline still resolves it; see `sym` tip below.) +4. Commit: `PC port: fix crash-handler Phase 2 self-fault, auto-backtrace on crash (D44)`. + +## Immediate task 2 — D43: model-file loading ABI mismatch 1. Reproduce: run; after a few rendered frames it SIGSEGVs in `modelPromoteNodeOffsetsToPointers()` (model.c:5688) during stage object load via - `load_object_fill_header()` (src/game/objecthandler_2.c:89). + `load_object_fill_header()` (src/game/objecthandler_2.c:89). With D44 fixed, the + crash log gives you the call chain for free. 2. Understand the file format from `bondtypes.h` (`ModelFileHeader`, `ModelNode`, `ModelRoData` union + per-opcode records) and `bondconstants.h` (`MODELNODE_OPCODE_*`). N64 layout: header 24 B, node 20 B, all pointers 4 bytes. @@ -65,19 +82,32 @@ build → verify under gdb → commit at each working milestone. existing `PROMOTE` rebase (`diff = fileramaddr - vma`, vma = 0x5000000) works unmodified. Hook it between the romCopy in `load_resource()` and first use (or at the `load_object_fill_header` call sites — choose the narrowest correct hook). -4. Validate one small model against ROM ground truth under gdb before generalizing - (opcode sequence, node links, a few Data records). +4. Validate one small model against ROM ground truth before generalizing (opcode + sequence, node links, a few Data records). gdb works; or write the optional + `tools_pc/romdump.py` inspector (below) and validate statically. 5. Expect follow-on faults in other asset types (BG/stan/propobj tables) — same procedure. -6. D40-class audit (done this session, negative): the dangerous pattern is a raw - `char[N]`/`u8[N]` BSS pool sized for N64 that gets cast to a pointer-containing - struct and written with PC stride. Only 3 raw byte arrays ≥1 KiB exist in game code - (crash/debug buffers — not pools); the 362 `dword_CODE_bss_*` placeholders are - mostly scalars or self-sizing typed arrays (e.g. bg.c `s_bound_info[204]`). Re-audit - per asset type as loading progresses. + +## Dev-loop speedups (build these when they pay off, not before) +Priority order from the last session's measurements: +1. **D44** (above) — automatic backtraces; biggest win per diagnosis. +2. **`tools_pc/sym.sh`** — tiny wrapper: takes a PC or rel offset, adds image base + `0x140000000`, runs `addr2line -e build-pc/ge007.x86_64.exe -f -C`. Kills the manual + arithmetic; pairs with the crash log (one command per crash). Re-verify the image base + with `info address` after any link-layout change. +3. **`tools_pc/romdump.py`** — Python ROM/asset inspector: load `data/ge007.ntsc-final.z64`, + parse ModelFileHeader/ModelNode trees using the bondtypes.h layout, print opcode + sequences / node links / Data records for a given cart address. Accelerates D43 and + every follow-on asset type; validates re-layout output against ROM ground truth without + running the game. (Python here: no f-strings with nested quotes — use `%` formatting.) +4. **Frame capture + pixel assert** — env-gated framebuffer PPM dump every N frames in + `port/`, plus a script that counts non-clear-color pixels / diffs two captures. Turns + "the scene contains actual model geometry" into an assert, makes the 30 s soak + scriptable, and enables screenshot diffs against an emulator (Mupen64Plus with the + verified No-Intro dump as ground truth). ## Standing procedure (you will hit more of these) Every ROM-serialized struct with a pointer field faults the same way once you reach more -asset loading. The D32 fix procedure (doc §H): at the fault run `ptype /o `; if a +asset loading. The D32 fix procedure (doc §H): at the fault, `ptype /o `; if a pointer field's offset/size diverges from its N64 offset comment, change it to `u32` + cast at the use sites; verify the load-time rebase yields valid V1 addresses (< 0x80000000); **also check the ROM bytes' endianness per-field (D33 rule: u32 bswap32 / u16 bswap16 / u8 @@ -90,7 +120,8 @@ Log each as D3x in §F and note it in AGENTS.md phase status. - MSYS2/MinGW tools are in `/c/msys64/mingw64/bin/` — **not** on PATH. Prefix every shell: `export PATH="/c/msys64/mingw64/bin:$PATH"`. - Build: `./build-pc.sh ntsc-final` (incremental; full rebuild only when `port/shim/` - changes). ROM is at `data/ge007.ntsc-final.z64` (= `baserom.u.z64`, byte-identical). + changes; a *clean* full build is ~5 s on the 16-core dev box). ROM is at + `data/ge007.ntsc-final.z64` (= `baserom.u.z64`, byte-identical). - The ROM is **mapped at cart base 0x10000000** on PC (`romdataInit`), so `&Symbol` values (e.g. `_sfxctlSegmentRomStart = 0x102EBDE0`) are directly dereferenceable host pointers into the ROM image; `romCopy()` copies from those cart addresses. All game RAM @@ -105,16 +136,18 @@ Log each as D3x in §F and note it in AGENTS.md phase status. offline with `addr2line -e build-pc/ge007.x86_64.exe -f -C <0x140000000+rel>`. Image base is `0x140000000` (re-verify with `info address` after a rebuild). Debug scripts live in `build-pc/` (`d37_tree.py`, `d39_*.gdb`, `d41_probe.gdb`, …). +- Crash log: `ge007.crash.log` (repo root) currently holds Phase 1 only — the backtrace + phase self-faults (**D44**, one-line fix, Immediate task 1). Until then attribute crashes + via gdb; after the fix the crash log is the first stop for any fault. - `objdump -d --disassemble=` / `objdump -s -j .data` on build-pc/ge007.x86_64.exe is a fast way to check what the compiler actually emitted. - Many init functions use a fake RBP — compute stack offsets from the **entry RSP**, not RBP. -- The D30 crash handler did **not** write `ge007.crash.log` for game-thread faults; - attribute crashes via gdb (the in-process dump does print PC/registers to stdout). - Python on this box: no f-strings with nested quotes (< 3.12) — use `%` formatting. ## Definition of done (this milestone) -Stage objects load without faulting and the rendered scene contains actual model geometry -(not just the clear color / background), stable under a 30 s+ soak. After that, Phase 2 -makes it look right (fast3d CC/RM correctness vs `gmain.s`) and Phase 3 adds sound -(libaudio → SDL). Commit + push to `origin/master` at each working checkpoint using the -message style `PC port: — `. +Checkpoint A: D44 committed — crash log shows a real backtrace for the D43 fault. +Then the milestone proper: stage objects load without faulting and the rendered scene +contains actual model geometry (not just the clear color / background), stable under a +30 s+ soak. After that, Phase 2 makes it look right (fast3d CC/RM correctness vs +`gmain.s`) and Phase 3 adds sound (libaudio → SDL). Commit + push to `origin/master` at +each working checkpoint using the message style `PC port: — `. diff --git a/docs/PCPortResearch.md b/docs/PCPortResearch.md index bb215ba4..d8dd1706 100644 --- a/docs/PCPortResearch.md +++ b/docs/PCPortResearch.md @@ -1238,6 +1238,22 @@ operates on misaligned fields. Needs a D37-style re-layout of the model file image (header + switches array + texture table + node tree + ModelRoData records). Phase-2 project; see §H. +**D44 (OPEN)** Crash-handler Phase 2 backtrace self-faults before printing. +`crashStackTraceSym()` in `port/src/crash.c` calls +`GetCurrentThreadStackLimits(low, high)` passing the NULL pointer *values* +rather than `&low, &high` (Phase 1's `crashStackTraceRaw` passes them +correctly). The API writes the stack limits to address 0x0 → access violation +*inside the SEH filter* → Windows terminates the process without a second +dispatch. Symptom: `ge007.crash.log` contains only Phase 1 (EXCEPTION/PC/ +MxCsr/thread-stack/MODULE lines) and no `BACKTRACE:` section — verified twice +live; this is why every D3x fault so far cost a full gdb launch-mode session. +Fix: one line, `GetCurrentThreadStackLimits(&low, &high)`. The build already +compiles with `-fno-omit-frame-pointer` (CMakeLists.txt:193), so after the fix +every crash auto-logs an EBP-chain backtrace to console + ge007.crash.log, +symbolicable offline with addr2line. Verify: run, confirm `BACKTRACE:` with +#00..#NN frames appears in the log and the D43 chain (modelPromoteNodeOffsets- +ToPointers ← load_object_fill_header ← …) is visible without gdb. + ### G. Phase 2 status (current) Done: PD fast3d integrated (`port/fast3d/`, replaces the deleted gfxstub.c); @@ -1324,6 +1340,24 @@ validate one small model against ROM ground truth under gdb before generalizing. Expect more D32-class ABI fixes as other per-level asset types are first touched — standing sub-task, not a one-off. +**Dev-loop measurements + speedups (session of 2026-08-22).** Measured +iteration costs on the dev box (16 cores): full clean rebuild ≈ 5 s wall +(239 ninja steps) — the build is NOT a bottleneck, ccache would be wasted +effort; launch → D43 crash ≈ 3.7 s — runtime wait is negligible. The cost is +in *diagnosis*: gdb launch-only mode + manual `addr2line` image-base arithmetic +per fault. Speedups, in priority order: (1) the D44 one-line fix → automatic +backtrace on every crash; (2) a `tools_pc/sym.sh` wrapper (PC or rel offset → +add image base 0x140000000 → `addr2line -f -C`); (3) a Python ROM/asset +inspector (`tools_pc/romdump.py`) that parses ModelFileHeader/ModelNode trees +straight from the .z64 using the bondtypes.h layout (opcode sequences, node +links, Data records) — accelerates D43 and every follow-on asset type, and can +validate re-layout output against ROM ground truth without running the game; +(4) env-gated framebuffer PPM dumps every N frames + a non-clear-color pixel +count/diff script — turns "the scene contains actual model geometry" into an +assert, makes the 30 s soak scriptable, and enables screenshot diffs against +an emulator (Mupen64Plus with the verified No-Intro dump). NOT worth it +(measured): ccache, boot fast-forward/skip-to-mainloop hooks. + **Audio (Phase 3 prep).** Bank trees are correctly laid out at boot and the libaudio manager + audio thread run without crashing; actual sound output still needs an SDL backend for `alDriver`, and the ASP strategy question (C2) remains @@ -1364,5 +1398,7 @@ prefix `export PATH=…`). Build: `./build-pc.sh ntsc-final`. gdb **launch** mod only (attach fails, error 87); symbolicate offline with `addr2line -e build-pc/ge007.x86_64.exe -f -C <0x140000000+rel>`. Image base 0x140000000. `load_resource`/many init fns use a fake RBP — compute stack offsets from entry -RSP. The D30 crash handler did **not** write `ge007.crash.log` for the game-thread -SIGSEGV this session (attribute via gdb instead; worth a separate look). +RSP. The D30 crash handler writes Phase 1 of `ge007.crash.log` (raw PC/ +registers) but its Phase 2 backtrace self-faults before printing (**D44**, +one-line fix pending) — until fixed, attribute crashes via gdb; after the fix +the crash log is the first stop.