diff --git a/AGENTS.md b/AGENTS.md index ef2e2744..367c5936 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,6 +48,6 @@ Run `/linkcheck` for this sweep. Record new findings in `docs/PCPortResearch.md` - **Compile+link milestone:** all ~235 TUs compile and `ge007.x86_64.exe` links clean (`ninja ge007 -k 0`, 236/236). Findings D7–D17 in `docs/PCPortResearch.md` §11; asset stubs in `port/src/assetstubs.c`, fast3d no-ops in `port/src/gfxstub.c` (delete when real fast3d lands). - **Phase 1 (boot to window):** done — ROM loads/validates/maps at cart base 0x10000000 (`romdata.c`), SDL2 window + GL clear loop (`video.c`), absolute asset symbols from `scripts/gen_romassets.py` → `port/src/romassets_.s` (findings D18–D23). - **Phase 1.5 (boot to first frame):** DONE — boot chain complete AND real frames render (`[NOTE] frame N rendered in … us`). D34–D42 closed the blockers on top of D31–D33: ANIM_DATA lvalues, music seq-table ABI+endianness, music-heap sizing, libaudio bank-tree re-layout, implicit-declaration prototype shim (D38), Globalimagetable rebasing + true segment size (D39), ModelHitEntry pool sizing (D40), cross-thread GL context release (D41), rsp task-settings toggle (D42). ROM dump verified against the No-Intro good dump (byte-identical, header CRCs re-checked — `tools_pc/romverify.c`). -- **Phase 2 (rendering):** IN PROGRESS — fast3d integrated + first frames render; current blocker is **D43** (model-file loading ABI mismatch: N64-layout ROM model files read as 8-byte-pointer PC structs → D37-style re-layout in `port/src/romdata.c`; a Rare-validated near-analogue exists in the PD port's `preprocess/filemodel.c` — §2.4). **D43 status:** all 512 model files validated; buffer sizing finalized by the D46 overlap-safety analysis (strict bound from N64 ground truth); step-2 mechanical game-code edits APPLIED (uncommitted); **D47 closed every open converter question** — compaction state values, G_VTX encoding, opcode-aware remap rule, vertex formats, staging contract — and the full conversion spec is implemented + validated 512/512 by `build-pc/d43_convert.py` (max D_PC = 0xB7E0, ratio ≤ 1.31). Done this session: D46 buffer corrections (front.c 0x25000 / initmenus.c 0x85000) + fast3d seg_addr seg-5 raw-VMA case. Remaining: C implementation of `romdataFixupModelFile` in `port/src/romdata.c` (port d43_convert.py 1:1) + hook in load_object_fill_header (+ poolRemaining=0 reset), build + soak — full checklist in `docs/HANDOFF.md`; findings D47 in §F. Recurring sub-task: ABI/layout fixes per the D32 procedure (§H) as more asset types are first touched (PD's `preprocess/` module already solves most of them). +- **Phase 2 (rendering):** IN PROGRESS — fast3d integrated + first frames render; current blocker is **D43** (model-file loading ABI mismatch: N64-layout ROM model files read as 8-byte-pointer PC structs → D37-style re-layout). **D43 status:** all 512 model files validated; buffer sizing finalized by the D46 overlap-safety analysis; step-2 mechanical game-code edits APPLIED + committed (ecefffdb); **D47 closed every open converter question** — compaction state values, G_VTX encoding, opcode-aware remap rule, vertex formats — spec implemented + validated 512/512 by `tools_pc/d43_convert.py` (max D_PC = 0xB7E0, ratio ≤ 1.31). **D48 review re-plan:** the old "port d43_convert.py 1:1 to C" task is discarded — that script does not emit bytes, and **Plan B (offline pre-conversion)** is now the default: Python emit pass → `data/pcmodels-/` RZ sidecars served via a port-layer `hw_address`/`rom_size` table patch (no C converter; full mechanics in §F/D48). Next session independently reviews Plan B (HANDOFF Task 1 → finding D49), then executes (Task 2); fallback Plan A = D47.13 C converter with the corrected staging guard (D48.4: `D_N64 + D_PC ≤ avail`). All 23 d43_*.py scripts moved from gitignored build-pc/ to tracked `tools_pc/`. Recurring sub-task: ABI/layout fixes per the D32 procedure (§H) as more asset types are first touched (PD's `preprocess/` module already solves most of them; prefer the offline pattern for format work — D48). - **Phase 3 (audio + input):** libaudio → SDL — largely copy-and-adapt from PD (`mixer.c` 722 lines vs our 31-line stub; `input.c` SDL_GameController backend; §2.4); decide the ASP strategy (C2: `aspMain*` dummies in `ucode.c`). - **Phase 4 (saves + polish):** file-backed EEPROM — build on PD's `fs.c` (file-backed save dir); PFS/motor already no-op stubs. diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index d60b3439..83f364a6 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -1,110 +1,254 @@ -# Handoff brief — GoldenEye 007 PC port (Phase 1.5 done, Phase 2: first frames render) +# Handoff brief — GoldenEye 007 PC port (Phase 2: first frames render; D43 re-planned as offline pre-conversion) -_Paste-ready briefing for the next agent session. The authoritative detail lives in -`AGENTS.md` and `docs/PCPortResearch.md` (§H handoff, §F/D31–D46 findings); this is -the summary + the immediate tasks._ +_Paste-ready briefing. Authoritative detail lives in `AGENTS.md` and +`docs/PCPortResearch.md` (§F/D31–D48 findings, §H handoff); this is the summary ++ the immediate tasks._ -## Your job -One task left on D43 (everything else is done and validated): -1. ~~**D44:** one-line fix to the crash handler~~ — **DONE, committed.** Every - crash now auto-logs a `BACKTRACE:` section to stdout and `ge007.crash.log`. -2. **D43 (main task):** model-file loading ABI mismatch — stage object load faults - in `modelPromoteNodeOffsetsToPointers()` (src/game/model.c:5688) because ROM model - files are serialized with N64 layout (4-byte pointer fields) but read on PC as - 8-byte-pointer structs. **Investigation COMPLETE; the converter spec is fully - pinned down and validated 512/512 by `build-pc/d43_convert.py`** (see §F/D47). - What remains: **the C implementation of `romdataFixupModelFile()` in - `port/src/romdata.c`, the hook in `load_object_fill_header`, build + soak.** - After that keep pushing rendering forward (more asset types will fault the - same way — Tbg_*_stanZ backgrounds next). +## Your job — TWO sessions, in order -Work agentically — fix → build (~5 s) → verify → commit at each working milestone. -Commit message style: `PC port: — `. +- **Session A (next): INDEPENDENT REVIEW of Plan B (finding D48).** Do NOT + implement anything. Verify the 8 claims in §Task 1 against code + ROM, each + with its falsification criterion. Record results as finding **D49** in + `docs/PCPortResearch.md` §F (per-item CONFIRMED/REFUTED + evidence), update + this file's status, commit. If every item is confirmed, Session B is green-lit. + If any is refuted: record why, then either adjust Plan B per the finding or + fall back to Plan A (§Task 3). The goal does not change: get stage geometry + on screen. +- **Session B (after D49 confirms): EXECUTE Plan B** per §Task 2 — Python emit + pass + sidecar generation, ~60 lines of port-layer plumbing, one-line hook, + build + pixel-assert soak. Commit at each working checkpoint + (`PC port: phase 2 — `). + +Work agentically — fix → build (~5 s) → verify → commit at each milestone. ## Read first (authoritative, in order) -1. `AGENTS.md` — non-negotiables, critical files, build, verification ritual, phase status. -2. `docs/PCPortResearch.md` §H (handoff + plan), then **§F/D47** (converter - contract finalized — read before writing the C converter), **D46** (overlap - safety + final buffer sizing), **D45** (buffer constants), **D43** (the open - model-file task), **§2.4** (PD-port copy-candidate audit — - read before starting D43 or Phase 3/4), **D37** (the libaudio bank-tree - re-layout pattern), D39–D42 for the first-render fixes, D31–D38 for background. +1. `AGENTS.md` — non-negotiables, critical files, build, verification ritual. +2. `docs/PCPortResearch.md` **§F/D48** (the plan under review — read fully), + then **D47** (converter contract — still ground truth for the emit pass), + **D46** (overlap safety + buffer sizing), **D45** (buffer constants), + **D43** (the model-file task), §2.4 (PD-port copy-candidate audit). +3. The code paths D48 cites: `port/src/rzdecomp.c`, `port/src/libultra.c` + (`piServiceDma`), `port/src/romdata.c` (`romdataInit`, + `romdataCartAddrValid`), `src/ramrom.c`, `src/game/ob.c` (`load_resource`, + `fileIndexLoadToBank`, `fileSetSize`, `obInit`), `src/game/objecthandler_2.c` + (`load_object_fill_header`). ## Non-negotiables (the ones that bite) - N64 build files (`Makefile`, `tools/`, `rsp/`, `ld/`) — never touch. -- **Game logic is unmodified.** Hardware deps go in `port/`. Narrow exception: mechanical, - semantics-preserving ABI/layout fixes forced by the 32→64-bit transition (embedded - pointers → u32 + cast at use; PC-guarded pool sizing / idiom branches), each documented - as a D3x finding. No behavior changes. +- **Game logic is unmodified.** Hardware deps go in `port/`. Narrow exception: + mechanical, semantics-preserving ABI/layout fixes forced by the 32→64-bit + transition (embedded pointers → u32 + cast at use; PC-guarded pool sizing / + idiom branches), each documented as a D3x finding. No behavior changes. + (Plan B's only game-code change is the PORT-guarded hook block in + `load_object_fill_header` — bookkeeping reset + one-shot patch call, no logic.) - Region macros in `CMakeLists.txt` must mirror the N64 Makefile exactly. ## Current state -- **D43 status (latest session, §F/D47):** - - DONE + APPLIED (uncommitted): all step-2 mechanical game-code edits - (bondtypes.h Vertex u32 union; model.c PROMOTE32; chr.c casts; - objecthandler_2.c Textures offset; tex.c Gfx-size/dispatch/byte-check/ - copy-count fixes; gun.c/front.c/initmenus.c D45 buffer constants). - - DONE (this session): **D46 buffer corrections applied** (front.c - bufferRemaining 0x1C000→0x25000, initmenus.c logo 0x7C000→0x85000) and the - **fast3d seg_addr seg-5 raw-VMA case** (port/fast3d/gfx_pc.cpp). - - DONE (this session): every open converter question resolved — compaction - state values (delta = R−P; accessors at 0x80090000), G_VTX w0 encoding - (GBI1 style; fast3d already correct), G_TRI4 false-positive eliminated - (opcode-aware remap rule), vertex formats verified, no embedded blobs, - zero-vtx-array rule, reload hazard + poolRemaining-reset decision. - - **VALIDATED:** `build-pc/d43_convert.py` implements the full conversion spec - and runs it on all 512 ROM files: ALL CLEAN (every pointer remap resolves; - max D_PC = 0xB7E0; max ratio D_PC/D_N64 = 1.31). - - **REMAINING:** port that validated logic to C as - `romdataFixupModelFile()` in port/src/romdata.c (two-pass: walk+layout, - then emit PC image staged at the tail of the reserved block and memmove to - F), hook it into load_object_fill_header (+ poolRemaining=0 reset before - _fileNameLoadToBank, per D47.4), set poolRemaining = D_PC after conversion, - build + run + 30 s soak. -- (Earlier session, verified live 2026-08-22) -- **Re-ran the committed build:** frames 1–4 render (`[NOTE] frame N rendered in … us`, - frame 1 ~21 ms), then FATAL `0xc0000005` at PC `0x14007a31a` = - `modelPromoteNodeOffsetsToPointers()` model.c:5688 — exactly D43. The black screen you - see is the clear color only: display lists execute but no stage geometry has loaded yet. -- **Dev loop measured (16-core box):** full clean rebuild ≈ **5 s** wall; launch → crash - ≈ **3.7 s**. Neither is a bottleneck — *diagnosis* is (gdb launch-only mode + manual - addr2line arithmetic per fault). ccache and boot fast-forward are NOT worth it. -- **D44 fixed + committed:** `crashStackTraceSym()` in - `port/src/crash.c` calls `GetCurrentThreadStackLimits(low, high)` — passing NULL pointer - *values* instead of `&low, &high`. The API writes to address 0x0 *inside the SEH filter* - → the process dies before printing the backtrace. That's why `ge007.crash.log` contains - only Phase 1 (EXCEPTION/PC/MODULE lines) and no `BACKTRACE:` section, and why every D3x - fault so far cost a gdb session. The build already uses `-fno-omit-frame-pointer` - (CMakeLists.txt:193), so the EBP-chain walk will work once the call is fixed. -- **ROM dump verified:** byte-identical to the No-Intro good dump, header CRCs re-compute - clean (`tools_pc/romverify.c`). Fingerprint + provenance in Environment below. -- Frames render because of D31–D42 (all documented in §F): boot chain (real-zlib - decompression, animation-table ABI+endianness, ANIM_DATA lvalues, music seq-table fixup, - music-heap sizing, libaudio bank-tree re-layout), D38 prototype shim (warning-clean - build), D39 Globalimagetable rebasing (`GIMG_OFF()` idiom), D40 ModelHitEntry pool - sizing, D41 WGL single-thread context currency (`gfx_sdl_release_context()`), D42 - rsp.c pointer-toggle truncation. +- **D43 status (latest review session, §F/D48):** investigation complete; + converter spec finalized (D47) and validated 512/512 (`tools_pc/d43_convert.py`, + max D_PC = 0xB7E0, ratio ≤ 1.31). **The remaining task was re-planned:** the + old plan ("port d43_convert.py 1:1 to C as `romdataFixupModelFile`") is + discarded because (a) d43_convert.py does not emit bytes — the emission code + is unimplemented in any language, and (b) **Plan B (offline pre-conversion)** + reuses the existing ROM load chain and eliminates the C converter entirely: + a Python emit pass writes 512 PC-layout RZ sidecars to + `data/pcmodels-/`; the port layer serves them by patching + `file_resource_table[i].hw_address` + `resource_lookup_data_array[i].rom_size` + (full mechanics in D48.2). Plan A (D47.13 C converter) remains the fallback, + with a corrected staging guard (D48.4: `D_N64 + D_PC ≤ avail`, not + `D_PC ≤ avail`). +- DONE + committed (ecefffdb, pushed): all step-2 mechanical game-code edits + (bondtypes.h Vertex u32 union; model.c PROMOTE32; chr.c casts; + objecthandler_2.c Textures offset; tex.c Gfx-size/dispatch/byte-check/ + copy-count fixes), D45/D46 buffer constants (gun.c/front.c/initmenus.c), + fast3d seg_addr seg-5 raw-VMA case (port/fast3d/gfx_pc.cpp). +- DONE this review session: **all 23 `d43_*.py` scripts moved from gitignored + `build-pc/` to tracked `tools_pc/` and committed** — the spec is now + versioned. Converter re-run from new location: ALL CLEAN. +- Verified environment facts (do not rediscover): frames 1–4 render then FATAL + at `modelPromoteNodeOffsetsToPointers()` (model.c:5688) = D43; black screen + is clear color only (no stage geometry loaded yet). ROM dump byte-identical + to the No-Intro good dump (`tools_pc/romverify.exe`). Crash log with real + backtraces works (D44). Build clean 237/237; full rebuild ~5 s. -## ~~Immediate task 1 — D44~~ (DONE — committed) -The one-line fix (`GetCurrentThreadStackLimits(&low, &high)` in -`crashStackTraceSym()`) is in. Verified: `BACKTRACE:` with #00 at model.c:5688 -and #01 `load_object_fill_header` appears on stdout + in `ge007.crash.log`. -Caveats learned: (a) frames past the true chain can be stale stack data — -validate a frame's address is inside `.text` before trusting it; (b) addr2line -wants the full absolute hex address directly (`addr2line -e -build-pc/ge007.x86_64.exe -f -C 0x14007a31a`). +## Task 1 — Plan B review checklist (Session A) -## Immediate task 2 — D43: model-file loading ABI mismatch (INVESTIGATION DONE — implement) +Verify each claim. For each: state CONFIRMED or REFUTED in D49 with the +evidence you used (file:line, command output). If a falsification criterion +fires, stop and record it — do not paper over it. -### Verified facts (all checked against ROM ground truth, all 512 files / 1661 nodes — do NOT re-derive) +**R1. RZ format is reproducible.** Claim: every model ROM region is +`[0x11 0x72][raw deflate]`; `decompressdata()` (port/src/rzdecomp.c) skips +exactly those 2 bytes and inflates raw (`inflateInit2(-15)`) with a generous +avail_in bound, so a sidecar file in that format works through +`load_resource` unmodified. Verify: read rzdecomp.c; confirm the 2-byte prefix +is constant across all 512 (one-liner over `scripts/filelist.u.csv` + ROM — +`tools_pc/d43_convert.py` already decompresses all 512 with `[2:], -15`, re-run +it). **Falsified if:** the prefix varies per file, or decompressdata depends on +header state beyond 2 bytes. + +**R2. The DMA path is a plain host memcpy.** Claim: `romCopy` (src/ramrom.c) → +`osPiStartDma` → port shim `piServiceDma` (port/src/libultra.c) = `memcpy` from +`(void*)srcPA`, gated ONLY by `romdataCartAddrValid()` (port/src/romdata.c:212, +checks `addr ≥ CART_BASE && off+size ≤ romSize`). The ROM is VirtualAlloc'd at +CART_BASE 0x10000000 sized exactly romSize — so extending the reservation by +the sidecar total and extending the validity bound serves sidecars with zero +other changes. Verify: read both shims end-to-end; confirm no other gating +(alignment, hardcoded 0x10C00000 limits elsewhere, cache effects). Note +`romCopyAligned` exists but is NOT used by this load path. **Falsified if:** +any additional gate rejects non-ROM host pointers or sizes we can't satisfy. + +**R3. The resource table is the single choke point and is writable.** Claim: +`file_resource_table` is a plain (non-const) global defined via include in +src/game/ob.c:22; ALL 512 model loads go through `load_object_fill_header` +(dst==0 → `_fileNameLoadToBank`, custom-buffer callers in front.c/gun.c/ +bondview2.c/initmenus.c → `_fileNameLoadToAddr`), both reading `hw_address` via +`load_resource`; no game code dereferences C*/G*/P*Z asset symbols directly +outside the table. Verify: grep callers of `fileLoad` / +`load_object_fill_header`; grep every model symbol (e.g. `CcamguardZ`, +`PflagZ`) outside `assets/obseg/file_resource_table.inc.c`, +`port/src/romassets_*.s`, and N64-only files (`obseg.h` decls, `ob_seg.s`). +**Falsified if:** any compiled game code reads model bytes straight from the +cart (it would bypass the sidecar and crash on N64 layout). + +**R4. Patch timing: after obInit, before first model load.** Claim: `obInit()` +(src/boss.c:179) runs before the first model load and computes `rom_size` from +adjacent-entry hw_address DELTAS (ob.c:122) — so the patch must run after it +and set `rom_size` explicitly; nothing between obInit and the first model load +consumes model entries' rom_size/hw_address. Verify: trace the init chain +(bossInitMainthreadData → … → first stage object load; the D43 crash stack +shows the order); grep `rom_size` uses (ob.c:122, 253, 291; +`obLoadBGFileBytesAtOffset` is BG-only). **Falsified if:** anything in that +window reads model entries' rom_size/hw_address, or obInit can run after a +model load. + +**R5. Load-flow fit for pre-converted data.** Claim: with sidecars, a fresh +dst==0 load allocates S_bank = whole remaining stage bank; the compressed +source placement `round8(compressed)+8 ≤ S_bank` holds (max single-file +compressed ≪ ~2.3 MB bank); `load_resource` sets poolRemaining := D_PC +automatically; compaction headroom R = S_bank satisfies D46's strict bound with +P_conv = D_PC (D46 already proved the buffers against these sizes). ADDITIONALLY: +for every dst!=0 caller, the patched compressed size + 8 ≤ the caller's buffer +size (gun.c 0x18000/0xBD70/0x17000, front.c wallet 0x17000 / cast screen +0x25000, initmenus.c logo 0x85000, bondview2.c weaponbuf `size0`). Verify: have +the emit pass (or a quick Python pass) print per-file compressed sizes and +cross-check against the D45/D46 constants; re-run `tools_pc/d43_chainbound.py`. +**Falsified if:** any buffer/chain is undersized for its file. + +**R6. Reload hazard + the one-line reset.** Claim: `fileSetSize` (ob.c:346-347) +leaves poolRemaining = rom_remaining = post-compaction size; without a reset, a +same-stage reload allocates S_bank' = P_old which can be < round8(compressed)+8 +→ `load_resource` hits the `source − ptrdata < 8` branch → poolRemaining = 0 → +silent load failure. A PORT-guarded `poolRemaining = 0` reset immediately before +`_fileNameLoadToBank` restores fresh-bank semantics every load with identical +steady-state usage. Verify: read `fileSetSize`, `fileIndexLoadToBank`, +`load_resource`; grep consumers of `get_pc_remaining_buffer_for_index` (should +be compaction only) to confirm the reset clobbers nothing. **Falsified if:** +the reset breaks a consumer, or the hazard cannot actually occur (then drop the +reset and say so in D49). + +**R7. Region handling.** Claim: sidecars are derived from a specific region ROM; +the generator takes the region, reads `data/ge007..z64`, writes +`data/pcmodels-/`; romdataInit picks the directory matching the ROM it +loaded (same token logic as ROM discovery in `romdataInit`). Verify: read +`build-pc.sh` + `romdataInit`'s ROM discovery; confirm name sets are identical +across regions is NOT required (the generator maps whatever files exist). +**Falsified if:** the build flow can't regenerate sidecars per region, or +romdataInit can't tell which directory matches. + +**R8. Emission spec completeness.** Claim: the "Verified facts" section below + +D47.13 cover every field of every opcode present in ROM (1,2,4,8,9,10,12,13, +15,18,21,22,23,24), and the round-trip validator (§Task 2 step 1) catches any +miss. Verify: walk each `ModelRoData_*Record` in src/bondtypes.h and confirm an +emission rule exists for every pointer/float/int field; spot-check 2–3 PC +record sizes against a fresh C probe (the Python `PC_REC` table was +probe-verified once — re-confirm it hasn't drifted). **Falsified if:** any +field has no rule, or a PC record size is wrong. + +**Review output:** D49 in §F (8 items adjudicated + evidence), HANDOFF status +updated, committed + pushed. If all CONFIRMED → Session B executes §Task 2. + +## Task 2 — Execution checklist (Session B; only after D49 confirms) + +1. **Emit pass in Python** (`tools_pc/d43_emit.py`, importing/reusing + `d43_convert.py`'s walk + region map). For each of the 512 files, emit the + PC image per D47.13 + Verified facts below: switches (NS×8B LE) → + texconfigs (NT×12B; bswap TextureID u32) → nodes+records in DFS preorder + (PC layout mirroring bondtypes.h; every promoted pointer field = + zero-extended u64 of `0x05|new_off`; Primary/Secondary GDL ptrs remapped but + NOT promoted; BaseAddr emitted 0) → vertex arrays immediately after their + record (normal: bswap s16 x,y,z,index,s,t, bytes @C–F raw; collision: bswap + s16 x,y,z,index + remap LinkedTo u32@8 + bswap CollisionRelatedIndex s16@C / + reserved s16@E; nv==0 with non-null ptr → size up to next object offset, + D47.10) → PointUsage (2×numVertices s16, after op24 collision verts) → + **GDLs LAST, contiguous, visit order, 16B LE slots**: w0' = bswap32(w0); + w1' = bswap32(raw) EXCEPT seg-5 of {G_VTX=0x04, G_SETTIMG=0xFD, + G_LOADBLOCK=0xF3} → remap low24 via the region map; **no LSB set**; emit up + to and including ENDDL. Write each file as `[0x11 0x72] + raw deflate(PC + image)` (Python: `zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, + zlib.DEFLATED, -15)`) into `data/pcmodels-/` plus a + `manifest.csv` (`name,offset,size`) with offsets assigned contiguously in + `file_resource_table` order (parse the quoted names from + `assets/obseg/file_resource_table.inc.c`). + **Round-trip validator (mandatory before trusting output):** re-parse each + emitted image as PC layout — 48B node stride, `PC_REC` record sizes, LE Gfx + slots; assert every promoted pointer equals the expected new offset, every + seg-5 GDL w1 target resolves inside the image, D_PC matches the pass-1 size. + Report per-file compressed sizes (feed R5's cross-check). +2. **Port-layer plumbing in `port/src/romdata.c`** (~60 lines): + - `romdataInit`: if `data/pcmodels-/manifest.csv` exists for the + loaded ROM, reserve the extended region — VirtualAlloc at CART_BASE sized + `romSize + sidecarTotal` (aligned), memcpy the ROM as today, then load the + sidecar blob(s) at `[CART_BASE + romSize, …)` per manifest offsets. Log + count + total bytes. Missing directory → warn and continue (ROM-only + mode = current behavior). + - Extend `romdataCartAddrValid` to accept up to + `CART_BASE + romSize + sidecarTotal`. + - New `void romdataPatchModelSidecars(void)` — one-shot (static flag): for + each `file_resource_table[i]` whose name is in the manifest, set + `hw_address = (u8*)(CART_BASE + romSize + off)` and + `resource_lookup_data_array[i].rom_size = size`. `extern` both arrays from + ob.c (same pattern as the existing `resource_lookup_data_array` extern). +3. **Hook in `load_object_fill_header`** (src/game/objecthandler_2.c, ONE + `#if defined(PORT)` block, D48-class bookkeeping — document as D50): + (a) call `romdataPatchModelSidecars()` (lazy one-shot; obInit has run by + first model load — R4); (b) reset + `resource_lookup_data_array[fileGetIndex(name)].poolRemaining = 0` + immediately before the `_fileNameLoadToBank`/`_fileNameLoadToAddr` call + (R6). No other game-code changes. +4. **Build, run, verify + pixel assert (build it NOW — dev-loop item #1):** + env-gated framebuffer PPM dump every N frames in `port/video.c` + (e.g. `GE_PCDUMP=1` → `build-pc/frame_%05d.ppm`) + `tools_pc/pixcount.py` + (counts non-clear-color pixels; diffs two captures). Milestone: no SIGSEGV + in `modelPromoteNodeOffsetsToPointers`; pixel count rises above a small + threshold and stays up through a 30 s+ soak. If geometry is wrong/missing, + the converted sidecars are inspectable OFFLINE (Python) — that's the point + of Plan B; don't gdb the game first. +5. **Commit checkpoints** (`PC port: phase 2 — …`); record findings D49/D50+ + in §F; update AGENTS.md phase status + this file. Then move on to the next + asset type (Tbg_*_stanZ backgrounds) with the SAME offline pattern — PD's + `port/src/preprocess/filebg.c` is the per-field reference (§2.4). + +## Task 3 — Plan A fallback (only if D49 refutes Plan B) + +D47.13 C converter in `port/src/romdata.c`: two-pass walk+layout then emit, +staged at `F + R − D_PC`, memmove to F; hook after the filedata assignment and +before Switches/Textures/RootNode setup; set `poolRemaining = D_PC` exactly +(never rom_remaining); poolRemaining=0 reset before `_fileNameLoadToBank`. +**Use the corrected guard (D48.4): fail if `D_N64 + D_PC > avail`** — the +staging region overlaps the live N64 image otherwise. The Python emit pass from +Task 2 step 1 is still worth writing first: it becomes the byte-level oracle to +diff the C output against (convert one file in C, compare with Python output). + +## Verified facts (converter spec — ground truth for the emit pass; do NOT re-derive) **Format / layout:** - All 512 C*/G*/P*Z model files (Tbg_*_stanZ backgrounds are a different format/path — later) decompress fine; each file starts directly with the switches array (no in-file header — NS/NT come from the exe `ModelFileHeader` record via `objheader->numSwitches/numtextures`). - File layout: `[switches NS×4B][textures NT×12B][root node @ 4NS+12NT]…`; all embedded pointers are vma `0x05xxxxxx` (mask 0xFFFFFF). -- **Tiling invariant verified, 0 failures in all 512 files** (`build-pc/d43_invariants.py`): `[0,D)` is exactly tiled by {switches, textures, nodes(24B), records(RSZ per opcode), vertex arrays(nv×16B), collision arrays(nc×16B), PointUsage (exactly numVertices s16 entries — +2 cases are pure alignment padding), GDLs (each extends to the next GDL start or D; 8-byte aligned, inter-GDL gaps copied as NOOP)}. Records INTERLEAVE with vertex arrays in character bodies — do not assume "all records before bulk". +- **Tiling invariant verified, 0 failures in all 512 files** (`tools_pc/d43_invariants.py`): `[0,D)` is exactly tiled by {switches, textures, nodes(24B), records(RSZ per opcode), vertex arrays(nv×16B), collision arrays(nc×16B), PointUsage (exactly numVertices s16 entries — +2 cases are pure alignment padding), GDLs (each extends to the next GDL start or D; 8-byte aligned, inter-GDL gaps copied as NOOP)}. Records INTERLEAVE with vertex arrays in character bodies — do not assume "all records before bulk". - **ModelNode = 24B N64** (u16 op + 2 pad + 5×4B ptrs: Data@4, Parent@8, Next@0xC, Prev@0x10, Child@0x14) / **48B PC**. Older "20B/40B" notes are wrong — verified against raw hex stride 0x18. - **Opcodes present: 1,2,4,8,9,10,12,13,15,18,21,22,23,24** (HEADER, GROUP, DL, LOD, BSP, BOX, GUNFIRE, SHADOW, INTERLINK, SWITCH, GROUPSIMPLE, DLPRIMARY, HEADPH, DLCOLLISION). op22 N64: numVertices s32@0, Vertices@4, Primary@8, BaseAddr@0xC (size 0x10); only 5 files have it. -- Record sizes N64/PC: OP01 0x10/0x18, OP02 0x1C/0x28, OP04 0x14/0x28, OP08 0x10/0x18, OP09 0x24/0x30, OP10 0x1C/0x1C, OP12 0x28/0x30, OP13 0x20/0x30, OP15 0x1C/0x1C, OP17 0x20/0x28, OP18 0x08/0x10, OP21 0x14/0x14, OP22 0x10/0x20, OP23 0x02/0x02, OP24 0x20/0x40 (compiler-verified). Field layouts in `src/bondtypes.h` (ModelRoData_*Record); read N64 fields from packed offsets, emit via real PC structs — don't hand-compute PC offsets. +- Record sizes N64/PC: OP01 0x10/0x18, OP02 0x1C/0x28, OP04 0x14/0x28, OP08 0x10/0x18, OP09 0x24/0x30, OP10 0x1C/0x1C, OP12 0x28/0x30, OP13 0x20/0x30, OP15 0x1C/0x1C, OP17 0x20/0x28, OP18 0x08/0x10, OP21 0x14/0x14, OP22 0x10/0x20, OP23 0x02/0x02, OP24 0x20/0x40 (compiler-verified). Field layouts in `src/bondtypes.h` (ModelRoData_*Record); read N64 fields from packed offsets, emit via the real PC layout — don't hand-compute PC offsets. - **f32 fields are big-endian in ROM** — bswap all f32 (record origins, BoundingVolumeRadius, Scale, Min/MaxDistance) and every s16 in vertex arrays; also bswap32 the TextureID u32 in the texture table (texLoad reads `*updateword & 0xffff` as the texture number). - **Vertex arrays must stay 16B-stride** (GDL gSPVertex hardcodes 16B/vertex; propobj.c copies with sizeof(Vertex); model.c:4360 allocates numVertices×4 words). bswap every s16 field BE→LE. Main Vertices array is pure data; **CollisionVertices carry LinkedTo@8 = vma of a ModelNode** — remap to the new node offset (PROMOTE promotes them after rebase). - **GDLs:** 8B BE word pairs → 16B LE slots: `w0=bswap32(rom_w0); w1=bswap32(rom_w1)` — **leave w1 raw, NO LSB set** (ROM convention; fast3d's extended seg_addr case handles unmarked segmented addresses — D47.13). GE uses STANDARD libultra GBI numbering (include/PR/gbi.h); opcodes live in the TOP byte of w0. **Opcode-aware remap: only {G_VTX=0x04, G_SETTIMG=0xFD, G_LOADBLOCK=0xF3} carry addresses in w1** (seg-5 → remap low24 via region map). All other opcodes get bswap-only — G_TRI4 (GE extension 0xB1) w1 is 4-bit vertex-index data, NOT an address (D47.6); same for TRI1/TEXTURE/SET*MODE/CLEARGEO/SETGEO/syncs. **G_VTX w0 is GBI1-style** (`04<<24 | dst<<16 | 16·n`, dst=((n−1)<<4)|v0; all 2826 have v0=0, n≤16) — bswap only, no field remap (D47.5). ROM scan: all 2805 seg-5 VTX refs have even offsets → lossless; no nested G_DL in model files. @@ -113,131 +257,40 @@ build-pc/ge007.x86_64.exe -f -C 0x14007a31a`). - PROMOTE contract: emit every promoted pointer as zero-extended u32 (low word of the 8B slot, high word 0); `PROMOTE` (`(u32)var + diff`, vma 0x5000000) then works unmodified. Fields promoted per opcode (model.c switch): all nodes Data/Parent/Next/Prev/Child; OP01 FirstGroup; OP02 ChildGroup; OP04 Vertices; OP08 Affects; OP09 left/rightChild; OP18 Controls; OP24 Vertices/CollisionVertices/PointUsage/CollisionVertices[i].LinkedTo. **NOT promoted:** GDL Primary/Secondary (stay raw 0x05xxxxxx), BaseAddr (game overwrites with fileramaddr). **Load path (single hook point):** -- ALL model loads go through `load_object_fill_header()` (objecthandler_2.c:97): dst≠0 → `_fileNameLoadToAddr(name,0,dst,size)` (region = caller's bytes — see D45 for the grown per-region sizes); else `_fileNameLoadToBank(name,0,0x100,4)`. **Fresh-load semantics (D47.2-4):** `fileIndexLoadToBank` takes ALL remaining MEMPOOL_STAGE space (`S_bank`) only when poolRemaining==0; then F := alloc(S_bank), R := S_bank, load_resource decompresses D into F and sets P := D_N64. At compaction: **delta = R − P** (positive on fresh loads — the GDL mirror moves FORWARD). `fileSetSize(reallocate=1)` → `mempAddEntryOfSizeToBank` rewinds the bank cursor to the post-compaction size, giving the tail back. -- **Reload hazard + PC fix (D47.4):** fileSetSize leaves P=R=post-compaction size, so a same-stage reload would alloc(P_old) which can be < D_N64. PORT-guarded reset of poolRemaining=0 before _fileNameLoadToBank forces fresh-bank semantics every load (staging space = whole remaining bank; steady-state usage identical). -- **The fixup must set ONLY poolRemaining (= D_PC exactly — D46/D47), never rom_remaining.** Do NOT use fileSetSize for this — it clobbers rom_remaining too. Set `resource_lookup_data_array[fileGetIndex(name)].poolRemaining` directly (array defined in ob.c:14, not extern'd — declare `extern` in romdata.c; note the accessors actually index **0x80090000 + idx*20** — D47.2). -- **Hook location:** inside load_object_fill_header — (a) poolRemaining=0 reset immediately BEFORE `_fileNameLoadToBank`; (b) `u32 romdataFixupModelFile(u8 *blob, const char *name, s16 NS, s16 NT)` (returns D_PC, 0 on failure; reads entry.rom_remaining itself for the staging-space guard) called immediately after the filedata assignment and BEFORE Switches/Textures/RootNode setup + sub_GAME_7F075A90 — PROMOTE walks with PC ModelNode stride (48B), so the file must already be re-laid-out. One `#if defined(PORT)` call site. +- ALL model loads go through `load_object_fill_header()` (objecthandler_2.c:97): dst≠0 → `_fileNameLoadToAddr(name,0,dst,size)` (region = caller's bytes — see D45 for the grown per-region sizes); else `_fileNameLoadToBank(name,0,0x100,4)`. **Fresh-load semantics (D47.2-4):** `fileIndexLoadToBank` takes ALL remaining MEMPOOL_STAGE space (`S_bank`) only when poolRemaining==0; then F := alloc(S_bank), R := S_bank, load_resource decompresses D into F and sets P := decompressed size (D_N64 from ROM; **D_PC directly for sidecars — no fixup needed**). At compaction: **delta = R − P** (positive on fresh loads — the GDL mirror moves FORWARD). `fileSetSize(reallocate=1)` → `mempAddEntryOfSizeToBank` rewinds the bank cursor to the post-compaction size, giving the tail back. +- **Reload hazard + PC fix (D47.4, needed by BOTH plans):** fileSetSize leaves P=R=post-compaction size, so a same-stage reload would alloc(P_old) which can be too small for the compressed source placement → `load_resource` sets poolRemaining=0 → silent failure. PORT-guarded reset of poolRemaining=0 before the load call forces fresh-bank semantics every load (staging space = whole remaining bank; steady-state usage identical). +- **Plan A only (fallback):** the fixup must set ONLY poolRemaining (= D_PC exactly), never rom_remaining; stage at F+R−D_PC with guard `D_N64 + D_PC ≤ avail` (D48.4); hook after filedata assignment, before Switches/Textures/RootNode setup + sub_GAME_7F075A90 (PROMOTE walks PC ModelNode stride — the image must already be re-laid-out). -**Compaction contract (sub_GAME_7F0762E0, objecthandler_2.c:24-83) — the tightest constraint:** +**Compaction contract (sub_GAME_7F0762E0, objecthandler_2.c:24-83) — the tightest constraint (applies to both plans):** - It mirrors the GDL region [first_gdl_off, P_conv) into the caller-region's tail headroom (texCopyGdls to [A+R−P_conv+g, A+R)), then rewrites each GDL in place via texLoadFromGdl (expands texture markers into RDP commands), writing back starting at the original first-GDL offset; final fileSetSize = end of last rewritten GDL (= P_conv + 16·M_actual). **D46: fit and no-clobber are the SAME condition — R ≥ P_conv + 16×Σ(K_t−1) worst case — and all buffers satisfy it (see §F/D46 for the strict bound + the two cast-screen corrections).** - Per-GDL `count` = byte distance between CONSECUTIVELY VISITED gdls → **GDLs must be packed contiguously at the tail of the PC image, in modelIterateDisplayLists visit order.** - `modelIterateDisplayLists` (model.c:6254) **MUTATES the tree while walking**: LOD → `node->Child = LOD.Affects`; SWITCH → `node->Child = Switch.Controls`; BSP → `modelApplyReorderRelationsByArg(node,TRUE)` splices siblings (leftChild group before rightChild). Visit order per DL node: Primary, then Secondary (if non-null and ≠ previous), then DFS Child/Next/Parent. -- `build-pc/d43_gdlorder.py` implements this exact simulation to verify visit-order adjacency — **bug fixed + verified**: node collection now pushes child+next plus LOD.Affects/SWITCH.Controls targets; all 512 files pass (total GDLs 2301). -- Primary/Secondary are NOT promoted (raw 0x05xxxxxx), so modelNodeReplaceGdl's `Primary == find` comparisons work unchanged; after compaction Primary = replacementgdl (raw value, arbitrary parity — see next item). +- `tools_pc/d43_gdlorder.py` implements this exact simulation to verify visit-order adjacency — all 512 files pass (total GDLs 2301). +- Primary/Secondary are NOT promoted (raw 0x05xxxxxx), so modelNodeReplaceGdl's `Primary == find` comparisons work unchanged; after compaction Primary = replacementgdl (raw value, arbitrary parity). **Render-time GDL resolution:** -- The game emits `gSPDisplayList(gdl, rwdata->gdl)` with w1 = raw 0x05xxxxxx; segment 5 (SPSEGMENT_MODEL_COL1) was just set to BaseAddr (= file base) by the preceding gSPSegment. fast3d's `seg_addr` (port/fast3d/gfx_pc.cpp:2282) currently sends 0x05xxxxxx down the "direct pointer" path (invalid). **Port-layer fix:** add a case — top byte 0x05 → `segmentPointers[5] + (w1 & 0xFFFFFF)`. +- The game emits `gSPDisplayList(gdl, rwdata->gdl)` with w1 = raw 0x05xxxxxx; segment 5 (SPSEGMENT_MODEL_COL1) is set to BaseAddr (= file base) by the preceding gSPSegment. fast3d's `seg_addr` (port/fast3d/gfx_pc.cpp:2282) resolves raw 0x05xxxxxx via `segmentPointers[5] + (w1 & 0xFFFFFF)` — **DONE, committed.** - Segments 3/4 (MTX table / vertex buffer) are set by the game per-render via gSPSegment ✓. BG segments (13/14/15) belong to the background path — later task. -### Implementation checklist (in order) -0. ~~**D46 buffer corrections (2 lines)**~~ — **DONE this session:** front.c - `bufferRemaining = 0x25000`, initmenus.c logo buffer `0x85000` (comments cite D46). -1. ~~Fix + run `build-pc/d43_gdlorder.py`~~ — DONE: all 512 files pass, op22 added. -2. ~~**Mechanical game-code edits (D32 class)**~~ — **APPLIED (uncommitted):** - bondtypes.h Vertex LinkedTo/CollisionRelatedNode → u32 under `#ifdef PORT`; - model.c PROMOTE32 macro + use at line ~5735; chr.c:3257-3259 cast + !=0 check; - objecthandler_2.c Textures offset via `sizeof(struct ModelNode*) * numSwitches`; - tex.c (4 fixes: texCopyGdls count, dispatch `(u8)(in->words.w0>>24)`, - 0xba byte check `(s8)((w0>>8)&0xff)`, texLoadFromGdl count `srcsize/sizeof(Gfx)`); - gun.c size_item_buffer {0x23000,0x23000}, D_80032464 {0xF000,0xF000}, SUIT R - 0x18000 (pool = size−0x18000), TRIGGER/WATCHLASER R 0x17000; front.c wallet R - 0x17000 + cast bufferRemaining (now corrected to 0x25000 per item 0); - initmenus.c logo buffer (now 0x85000 per item 0). All PORT-guarded with D45/D46 - comments. - - `src/bondtypes.h` Vertex: union members `struct Vertex *LinkedTo; void *CollisionRelatedNode;` → `u32 LinkedTo; u32 CollisionRelatedNode;` (sizeof(Vertex) stays 16 both platforms; game heap <4GiB so lossless on PC). - - `src/game/model.c`: add `#define PROMOTE32(var) if (var) var = (u32)((u32)var + diff)` beside PROMOTE (~line 5677); line 5735 `PROMOTE(...LinkedTo)` → `PROMOTE32(...)`. - - `src/game/chr.c:3257`: cast the now-u32 field `(ModelNode *)(uintptr_t)…`; line 3259 `!= NULL` → `!= 0`. (These are the only 3 use sites of the union.) - - `src/game/objecthandler_2.c:103`: `&((s32*)filedata)[objheader->numSwitches]` → `((u8*)filedata + sizeof(ModelNode*) * objheader->numSwitches)` (N64: 4B stride, identical). - - `src/game/tex.c`: texCopyGdls `arg2 = (arg2 >> 3)` → `arg2 / sizeof(Gfx)`; texLoadFromGdl `count = srcsize >> 3` → `srcsize / sizeof(Gfx)` and dispatch `switch (*(u8*)in)` → `switch ((u8)(in->words.w0 >> 24))` (N64-identical; those are the ONLY byte-level accesses in that function — everything else already uses words.w0/w1, and the default case copies whole slots). -3. ~~**fast3d seg_addr**~~ — **DONE this session:** port/fast3d/gfx_pc.cpp now - resolves raw 0x05xxxxxx (no LSB) via segmentPointers[5] + low24 BEFORE the LSB - path; also handles unmarked segmented addresses generally (nibble 24-27 = seg, - w1 < 0x10000000). Segment 5 is set per-render by the game's gSPSegment - (BaseAddr = live host file base). -4. **`romdataFixupModelFile` in `port/src/romdata.c`** (PC-only; bswap helpers + - extern `resource_lookup_data_array` from ob.c are already there). **Port the - validated logic of `build-pc/d43_convert.py` 1:1 — it IS the spec** (512/512 - clean; D47.12). Two-pass: (a) DFS walk collecting nodes/records/GDLs + build - the region map (old_off, old_size → new_off); (b) emit the PC image into a - STAGING area at `blob + avail − D_PC` (avail = entry.rom_remaining; guard: - fail with an error if D_PC > avail), then memmove to blob. Emission order: - switches(NS×8B LE) → texconfigs(NT×12B; bswap TextureID only — no seg-5 TIDs - exist in any model file, D47.9) → nodes+records in DFS preorder (PC layout via - struct assignment; every promoted pointer field = zero-extended u64 of - 0x05|new_off — full per-opcode field map in the facts above + D47.13; - Primary/Secondary GDL ptrs remapped but NOT promoted; BaseAddr emitted 0) → - vertex arrays immediately after their record (normal: bswap s16 x,y,z,index,s,t, - bytes @C-F raw; collision: bswap s16 x,y,z,index + remap LinkedTo u32@8 + - bswap CollisionRelatedIndex s16@C/reserved@E; **nv==0 with non-null ptr → size - up to next object offset**, D47.10) → PointUsage (2×numVertices s16, after - op24 CollisionVerts) → **GDLs LAST, contiguous, visit order, 16B LE slots**: - w0'=bswap32(w0); w1' = bswap32(raw) EXCEPT for seg-5 of {G_VTX=0x04, - G_SETTIMG=0xFD, G_LOADBLOCK=0xF3} → remap low24 via the region map (sorted - regions + binary search); **no LSB set** (ROM convention; D47.6/13). Emit each - GDL up to and including its ENDDL (drop trailing junk — fast3d returns at - ENDDL, D47.11). Set `resource_lookup_data_array[fileGetIndex(name)].poolRemaining` - = D_PC EXACTLY after conversion (load_resource left it at D_N64). NEVER touch - rom_remaining. Assert `D_PC + 16×Σ_markers(K_t−1) ≤ avail` (K_BOUND - {0:38,1:46,2:36,3:18,4:15}) with a clear error. -5. **Hook** into load_object_fill_header (`#if defined(PORT)`): (a) reset - `resource_lookup_data_array[fileGetIndex(name)].poolRemaining = 0` immediately - BEFORE `_fileNameLoadToBank` (D47.4 — forces fresh-bank allocation so staging - space = whole remaining bank; avoids the reload-under-alloc hazard); - (b) call romdataFixupModelFile after the filedata assignment and BEFORE - Switches/Textures/RootNode setup + sub_GAME_7F075A90 (PROMOTE walks PC - ModelNode stride so the image must be re-laid-out first). -6. **Build, run, verify:** no SIGSEGV in modelPromoteNodeOffsetsToPointers; scene - shows actual geometry (not clear color); 30 s+ soak stable. Sizing is already - proven by D46's strict bound + D47.12's max-ratio 1.31 — if anything still - overflows, the converter has a bug, not the buffers. - -### Helper scripts (build-pc/, written across D43 sessions) -- `d43_decode.py` — decompress + decode one model's node tree + GDL command stream. -- `d43_sizes.py` / `d43_sizes2.py` — decompress all 512, report N64/PC sizes + - per-file worst-case P_final (B_pc + 2E + 16×Σ(K_t−1)). -- `d43_walk.py` — walks all trees using exe NS/NT; opcode histogram. Note: header - names lack the C/G/P prefix and trailing Z (file `CcamguardZ` → header `camguard`). +## Helper scripts (tools_pc/, tracked; run from repo root) +- `d43_convert.py` — **reference layout + validator (D47):** DFS walk with LOD/SWITCH rewiring, region map, per-opcode record/vertex/GDL analysis, remap checks; runs on all 512 ROM files (ALL CLEAN). Lays the foundation for the emit pass (Task 2 step 1) — import/reuse its walk + region map. - `d43_invariants.py` — full ownership tiling check (0 failures / 512). -- `d43_gdlorder.py` — exact visit-order simulation; **bug fixed, all 512 pass**. -- `d43_gdldump.py` / `d43_gdlhist.py` / `d43_gdlseq.py` / `d43_seg5.py` — GDL - command dumps/histograms/sequences; seg-5 VTX reference analysis (all 2805 point - into op4/op22 vertex arrays → must remap). -- `d43_cover.py` — verifies every seg-5 VTX + SETTIMG target falls inside a modeled - object (2804/2805 covered; the one exception is PexplosionbitZ's orphan array at - offset 0x98 — handled by the converter's implicit-object rule). SETTIMG targets in - logo files are vestigial (in gaps, never dereferenced). -- `d43_lutscan.py` — scans all model texture tables → image headers (via - imagelist.u.csv, order = assets/images.def): proves NO LUT textures (formats 9-12) - and maxlod distribution {0:1063, 6:7, 7:1} across all 512 files (D46). -- `d43_convert.py` — **THE reference converter + validator (D47):** implements - the full N64→PC conversion spec (DFS walk with LOD/SWITCH rewiring, region map, - per-opcode record/vertex/GDL conversion, remap checks) and runs it on all 512 - ROM files — ALL CLEAN. This IS the spec to port to C in romdata.c. -- `d43_chainbound.py` — computes the D46 strict bound B_pc + 2×(R_share_N64 − B_n64) - for every buffer/chain and compares against the PC region sizes. -- Gotcha: MODELFILEHEADER's name is field a[0] of the macro args (NOT the filename — - all header files are literally named `modelFileHeader.inc.c` in per-model dirs). +- `d43_gdlorder.py` — exact visit-order simulation; all 512 pass. +- `d43_sizes.py` / `d43_sizes2.py` — decompress all 512, report N64/PC sizes + per-file worst-case P_final. +- `d43_walk.py` / `d43_fullwalk.py` / `d43_tree_dump.py` — tree walks, opcode histogram. (Header names lack the C/G/P prefix and trailing Z: file `CcamguardZ` → header `camguard`; all header files are literally named `modelFileHeader.inc.c` in per-model dirs.) +- `d43_gdldump.py` / `d43_gdlhist.py` / `d43_gdlseq.py` — GDL command dumps/histograms/sequences. +- `d43_seg5.py` / `d43_seg5ops.py` / `d43_seg5vtx.py` / `d43_seg5vtx2.py` — seg-5 VTX reference analysis (all 2805 point into op4/op22 vertex arrays → must remap). +- `d43_cover.py` — verifies every seg-5 VTX + SETTIMG target falls inside a modeled object (2804/2805; the one exception is PexplosionbitZ's orphan array at 0x98 — handled by the zero-vtx rule). +- `d43_lutscan.py` — proves NO LUT textures (formats 9-12) and maxlod distribution across all 512 files (D46). +- `d43_chainbound.py` — computes D46's strict bound for every buffer/chain vs the PC region sizes. +- `d43_decode.py` / `d43_layout*.py` / `d43_pointusage.py` / `d43_vtxfmt.py` — earlier investigation (format pinning). +- Gotcha: MODELFILEHEADER's name is field a[0] of the macro args (NOT the filename). -### Follow-on -Expect later faults in other asset types (BG/stan/propobj tables) — same D32/D37 procedure; PD's `preprocess/` module already solves most of them (`filebg.c`, `filetiles.c`, `filepads.c`, `filesetup.c`, …) — see the PD reference section below. - -## Dev-loop speedups (build these when they pay off, not before) -Priority order from the last session's measurements: -1. **D44** (above) — automatic backtraces; biggest win per diagnosis. -2. **`tools_pc/sym.sh`** — tiny wrapper: takes a PC or rel offset, adds image base - `0x140000000`, runs `addr2line -e build-pc/ge007.x86_64.exe -f -C`. Kills the manual - arithmetic; pairs with the crash log (one command per crash). Re-verify the image base - with `info address` after any link-layout change. -3. **`tools_pc/romdump.py`** — Python ROM/asset inspector: load `data/ge007.ntsc-final.z64`, - parse ModelFileHeader/ModelNode trees using the bondtypes.h layout, print opcode - sequences / node links / Data records for a given cart address. Accelerates D43 and - every follow-on asset type; validates re-layout output against ROM ground truth without - running the game. (Python here: no f-strings with nested quotes — use `%` formatting.) -4. **Frame capture + pixel assert** — env-gated framebuffer PPM dump every N frames in - `port/`, plus a script that counts non-clear-color pixels / diffs two captures. Turns - "the scene contains actual model geometry" into an assert, makes the 30 s soak - scriptable, and enables screenshot diffs against an emulator (Mupen64Plus with the - verified No-Intro dump as ground truth). +## Dev-loop speedups (priority order) +1. **Frame capture + pixel assert** — NOW (Task 2 step 4): env-gated PPM dump every N frames in `port/video.c` + `tools_pc/pixcount.py` (non-clear-color pixel count / diff two captures). Turns "the scene contains actual model geometry" into an assert, makes the 30 s soak scriptable, and enables screenshot diffs against Mupen64Plus with the verified No-Intro dump as ground truth. +2. **`tools_pc/sym.sh`** — tiny wrapper: takes a PC or rel offset, adds image base `0x140000000`, runs `addr2line -e build-pc/ge007.x86_64.exe -f -C`. One command per crash; re-verify the image base with `info address` after any link-layout change. +3. **`tools_pc/romdump.py`** — Python ROM/asset inspector: parse ModelFileHeader/ModelNode trees using the bondtypes.h layout, print opcode sequences / node links / Data records for a given cart address; also re-parse SIDECAR files as PC layout (post-Plan B this is the first stop for "geometry looks wrong"). +4. D44 (automatic backtraces) — DONE, committed. ## PD port as standing reference (always available) `PD_PORT_CHECKOUT` is the Perfect Dark PC port — same Rare engine @@ -245,8 +298,9 @@ family, and the **standing reference/guidance source** for this project (noted i AGENTS.md). Consult it whenever a remaining work item has a PD analogue: - **Phase 2 asset conversion:** `port/src/preprocess/` (~4,100 lines) — per-ROM-segment N64→PC layout converters hooked from their `romdata.c`. `filemodel.c` = D43 near-analogue - (same vma 0x5000000); `filebg.c` (expected follow-on), `segaudio.c` (cross-check for our - D37 bank-tree fixup), `filelang.c`, `filetiles.c`, `filepads.c`, `filesetup.c`, `gbi.c`, + (same vma 0x5000000; PD converts at LOAD time — we convert OFFLINE for speed, same field + rules apply); `filebg.c` (expected follow-on), `segaudio.c` (cross-check for our D37 + bank-tree fixup), `filelang.c`, `filetiles.c`, `filepads.c`, `filesetup.c`, `gbi.c`, `segfonts.c`. - **Phase 3/4 (copy-and-adapt; our stubs were scaffolded for this):** `port/src/mixer.c` (722 lines vs our 31 — libaudio→SDL mixer), `input.c` (1,551 vs 48 — SDL_GameController @@ -265,7 +319,10 @@ at the use sites; verify the load-time rebase yields valid V1 addresses (< 0x800 identity) and any fixup loop's pointer stride on x86-64**; for packed struct trees use the D37 re-layout pattern; for address-arithmetic idioms over exe symbols (D39/D42 class) guard a PC branch that reproduces the N64 32-bit value exactly. Rebuild; confirm boot advances. -Log each as D3x in §F and note it in AGENTS.md phase status. +Log each as D3x in §F and note it in AGENTS.md phase status. **For asset FORMAT work, +prefer the offline pattern (D48): pin the format with Python over the ROM ground truth, +emit + validate offline, serve through the existing load path — runtime C conversion only +as fallback.** ## Environment (do not rediscover these) - MSYS2/MinGW tools are in `/c/msys64/mingw64/bin/` — **not** on PATH. Prefix every shell: @@ -286,7 +343,8 @@ Log each as D3x in §F and note it in AGENTS.md phase status. - gdb: **launch** mode only (`gdb -batch -ex "handle SIGSEGV stop" -ex run …`). Symbolicate offline with `addr2line -e build-pc/ge007.x86_64.exe -f -C <0x140000000+rel>`. Image base is `0x140000000` (re-verify with `info address` after a rebuild). Debug scripts live in - `build-pc/` (`d37_tree.py`, `d39_*.gdb`, `d41_probe.gdb`, …). + `build-pc/` (`d37_tree.py`, `d39_*.gdb`, `d41_probe.gdb`, …); the D43 investigation suite + is in tracked `tools_pc/`. - **Capstone gotcha (D47.1):** the ROM is BIG-ENDIAN MIPS — always disassemble with `CS_MODE_MIPS32 | CS_MODE_BIG_ENDIAN`. Default (LE) mode produces garbage that looks plausible; an earlier session's disassembly was all LE and had to be redone. @@ -299,12 +357,12 @@ Log each as D3x in §F and note it in AGENTS.md phase status. - Many init functions use a fake RBP — compute stack offsets from the **entry RSP**, not RBP. - Python on this box: no f-strings with nested quotes (< 3.12) — use `%` formatting. -## Definition of done (this milestone) -Checkpoint A: D44 committed — crash log shows a real backtrace for the D43 fault. -Then the milestone proper: stage objects load without faulting and the rendered scene -contains actual model geometry (not just the clear color / background), stable under a -30 s+ soak. After that, Phase 2 makes it look right (fast3d CC/RM correctness vs -`gmain.s` — GE-specific, no PD analogue) and Phase 3 adds sound + input, largely -copy-and-adapt from the PD port (see "PD port as standing reference" above). Commit + -push to `origin/master` at each working checkpoint using the message style -`PC port: — `. +## Definition of done +- **Session A:** D49 committed — all 8 checklist items adjudicated with evidence; + HANDOFF status updated; if any item refuted, the fallback/adjustment is spelled out. +- **Session B (the milestone proper):** stage objects load without faulting and the + rendered scene contains actual model geometry per the pixel assert (not just the clear + color / background), stable under a 30 s+ soak. Committed + pushed to `origin/master` + (`PC port: phase 2 — `). After that, Phase 2 makes it look right (fast3d CC/RM + correctness vs `gmain.s` — GE-specific, no PD analogue) and Phase 3 adds sound + input, + largely copy-and-adapt from the PD port. diff --git a/docs/PCPortResearch.md b/docs/PCPortResearch.md index cd4fcaae..fd49b08a 100644 --- a/docs/PCPortResearch.md +++ b/docs/PCPortResearch.md @@ -1494,6 +1494,81 @@ through the real structs (compiler packs); don't hand-compute. 15. **PointUsage** = 2×numVertices s16 entries, indexed by MAIN-vertex index, chain terminated by −1 (chr.c:3309-3331). +**D48 (REVIEW — D43 re-plan: offline pre-conversion "Plan B"; process fix)** +Independent review session that audited the D47 handoff against code + ROM +before implementation. Verdict: the remaining task was mis-scoped as a 1:1 C +port of a script that does not emit bytes; a cheaper, lower-risk path exists +that reuses the existing ROM load chain. Plan B is now the default; Plan A +(D47.13 C converter) is the fallback. + +1. **`d43_convert.py` does not emit bytes.** It computes the layout/region map +and validates that every pointer remap resolves (512/512 clean, re-run and +confirmed this session), but there is no emission pass in ANY language. The +byte-level contract (PC struct assignment, per-field bswaps, GDL slot writing) +is unimplemented and unvalidated. "Port d43_convert.py 1:1 to C" would have the +next session write ~250-300 lines of new emission code in C and debug it at +runtime (crash → backtrace cycles). The emission must be written once either +way — do it in Python where iteration is seconds, validate offline, then ship +the data. +2. **Plan B: offline pre-conversion through the existing ROM load path.** +Generate 512 PC-layout RZ sidecar files with a Python emit pass; serve them by +patching `file_resource_table[i].hw_address` + +`resource_lookup_data_array[i].rom_size` from the port layer. Verified +mechanical facts (all re-checkable in <1 h): + - **RZ format is trivially reproducible**: 2-byte header (`0x11 0x72`) + raw +deflate; `decompressdata()` (port/src/rzdecomp.c) skips the header and inflates +with a generous avail_in bound. A sidecar `[0x11 0x72][raw-deflate(PC image)]` +works through `load_resource` UNMODIFIED — and it sets `poolRemaining = D_PC` +automatically (the decompressed size), so no fixup call is needed at all. + - **`romCopy` on PC is a host memcpy**: src/ramrom.c → `osPiStartDma` → port +shim `piServiceDma` (port/src/libultra.c), gated only by +`romdataCartAddrValid()` (port/src/romdata.c:212). The ROM is VirtualAlloc'd at +CART_BASE 0x10000000 sized romSize — extend the reservation by the sidecar +total, place sidecars at [CART_BASE+romSize, …), extend the validity check. + - **`file_resource_table` is a plain writable global** (included in +src/game/ob.c:22). ALL 512 model loads funnel through +`load_object_fill_header` (dst==0 → `_fileNameLoadToBank`; custom-buffer +callers in front.c/gun.c/bondview2.c/initmenus.c → `_fileNameLoadToAddr`) — +both read `hw_address` via `load_resource`. No game code dereferences C*/G*/P*Z +symbols directly outside the table (verified by grep; symbols exist only in +the table + romassets_.s markers). + - **The patch must run AFTER `obInit()`** (called at src/boss.c:179): +obInit computes `rom_size` from adjacent-entry hw_address DELTAS (ob.c:122), so +a pre-obInit patch would corrupt rom_size. Lazy one-shot at the PORT hook site +in load_object_fill_header is simplest — by first model load, obInit has +definitely run. + - **Footprint**: 512 files = 1,277,088 B compressed total (1.2 MB), +3,289,344 B decompressed N64 total; PC decompressed ≤ 1.31× per file (D47.12). + - **The indy path is NOT used** (`resource_load_from_indy`, ob.c:56): it is +the N64 host-protocol loader gated by `indy_ready` (src/game/indy_comms.c, +dormant on PC), and its pPayload placement underflows on reload when +poolRemaining == pc_size exactly. Table patching reuses the proven ROM path. + - **What Plan B eliminates**: the C converter (~300 lines), two-pass staging ++ memmove, staging-space guard, and the "emission bug only visible at runtime" +risk class. What it keeps (all already committed): ABI edits, fast3d seg-5 +case, D46 buffer sizing — plus the one-line poolRemaining=0 reset (item 3). + - **Sidecars are region-specific** (derived from the region ROM); generator +must take the region and write `data/pcmodels-/`. +3. **The poolRemaining=0 reset is STILL needed under Plan B.** fileSetSize +leaves P=R=post-compaction size (ob.c:346-347); a same-stage reload would then +alloc S_bank' = P_old, which can be < round8(compressed)+8 → `load_resource` +hits the `source − ptrdata < 8` branch → poolRemaining=0 → silent load +failure. The D47.4 reset before `_fileNameLoadToBank` covers both plans. +4. **Plan A flaw (if runtime conversion is kept):** D47.13's staging guard +"fail if D_PC > avail" is too weak — the staging region [F+R−D_PC, F+R) +overlaps the live N64 image [F, F+D_N64) whenever S_bank < D_N64 + D_PC, and +emission would then read corrupted bytes. Correct guard: `D_N64 + D_PC ≤ +avail`. Low probability in practice (files load at stage start; ~2.3 MB bank +vs ~82 KB worst case), but the stated guard silently misses it. +5. **Process fix:** the 23 d43_*.py investigation scripts (incl. the reference +converter) lived in gitignored `build-pc/` — moved to tracked `tools_pc/` +and committed, so the spec is versioned and reviewable. d43_cover.py's one +internal path reference updated; converter re-run from new location: ALL CLEAN. +6. **Review checklist for the next session** (confirm each item against code + +ROM before executing Plan B; record results as D49): see docs/HANDOFF.md +§Task 1 — eight claims, each with a falsification criterion. If any fires, +fallback to Plan A with the corrected guard from item 4. + ### G. Phase 2 status (current) Done: PD fast3d integrated (`port/fast3d/`, replaces the deleted gfxstub.c); diff --git a/tools_pc/d43_chainbound.py b/tools_pc/d43_chainbound.py new file mode 100644 index 00000000..fef3f620 --- /dev/null +++ b/tools_pc/d43_chainbound.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 +"""Cast-chain strict overlap/fit bound (D46 follow-up). + +Formula (derived from N64 ground truth: the N64 game works, so per file +8*M_a <= R_share_N64 - B_n64 - E; PC slots are 2x wide with the same M_a): + + sum(P_final_actual) <= sum(B_pc) + 2*(R_chain_N64 - sum(B_n64)) + +where R_chain_N64 = 0x18160 (N64 cast-screen bufferRemaining). +Compares against the PC bufferRemaining = 0x1C000 currently in front.c. +""" +import csv, struct, zlib, os, re +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) + +RSZ_N64 = {1: 0x10, 2: 0x1C, 4: 0x14, 8: 0x10, 9: 0x24, 10: 0x1C, + 12: 0x28, 13: 0x20, 15: 0x1C, 17: 0x20, 18: 0x8, 21: 0x14, + 22: 0x10, 23: 2, 24: 0x20} +RSZ_PC = {1: 0x18, 2: 0x28, 4: 0x28, 8: 0x18, 9: 0x30, 10: 0x1C, + 12: 0x30, 13: 0x30, 15: 0x1C, 17: 0x28, 18: 0x10, 21: 0x14, + 22: 0x20, 23: 2, 24: 0x40} + +nsnt = {} +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + for line in open(os.path.join(root, fn)): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: + continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: + continue + try: + nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: + pass + +def be32(b, o): + return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF + +def analyze(base_want): + for r in rows: + if len(r) < 3 or not r[2]: + continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): + base = base[:-4] + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: + key = cand + break + if key != base_want: + continue + addr, size = int(r[0]), int(r[1]) + out = zlib.decompress(rom[addr:addr + size][2:], -15) + D = len(out) + NS, NT = nsnt[key] + R0n = 4 * NS + 12 * NT + stack = [R0n] + seen = set() + nodes = [] + gdls = [] + while stack: + o = stack.pop() + if o in seen or o >= D - 23: + break + seen.add(o) + op = struct.unpack_from(">H", out, o)[0] & 0xff + data = be32(out, o + 4) + nodes.append((o, op, data)) + if op in (4, 24): + p, s = be32(out, data), be32(out, data + 4) + if p: gdls.append(p) + if s: gdls.append(s) + elif op == 22: + p = be32(out, data + 8) + if p: gdls.append(p) + for q in (be32(out, o + 0x14), be32(out, o + 0xC)): + if q: + stack.append(q) + gsorted = sorted(set(gdls)) + g1 = gsorted[0] if gsorted else D + bulk = (g1 - R0n) - len(nodes) * 24 - sum(RSZ_N64[op] for _, op, _ in nodes) + B_n64 = 4 * NS + 12 * NT + len(nodes) * 24 + sum(RSZ_N64[op] for _, op, _ in nodes) + bulk + B_pc = 8 * NS + 12 * NT + len(nodes) * 48 + sum(RSZ_PC[op] for _, op, _ in nodes) + bulk + E = D - g1 + return dict(base=base_want, D=D, B_n64=B_n64, B_pc=B_pc, E=E) + +# cast-screen weapon sets (front.c ~867-891) +rifles = ["kalash", "m16", "fnp90", "autoshot", "grenadelaunch", "sniperrifle"] +pistols = ["wppk", "wppksil", "skorpion", "uzi", "tt33", "ruger", "laser", "golden"] +heads = [] +# find all head files (Chead*Z) +for r in rows: + if len(r) < 3 or not r[2]: + continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): + base = base[:-4] + if base.startswith("Chead") and base.endswith("Z"): + heads.append(base) + +def worst_in(names): + best = None + for n in names: + a = analyze(n) + if not a: + print(" !! not found:", n) + continue + # P_final(worst K) proxy: B_pc + 2E (marker slack handled separately) + pf = a["B_pc"] + 2 * a["E"] + if best is None or pf > best[0]: + best = (pf, a) + return best + +print("== worst head ==") +hp, ha = worst_in([h[1:-1] for h in heads]) +print("%s P_conv=%X B_pc=%X B_n64=%X" % (ha["base"], hp, ha["B_pc"], ha["B_n64"])) + +print("== worst rifle ==") +rp, ra = worst_in(rifles) +print("%s P_conv=%X B_pc=%X B_n64=%X" % (ra["base"], rp, ra["B_pc"], ra["B_n64"])) + +print("== worst pistol ==") +pp, pa_ = worst_in(pistols) +print("%s P_conv=%X B_pc=%X B_n64=%X" % (pa_["base"], pp, pa_["B_pc"], pa_["B_n64"])) + +# worst body: scan C* bodies (exclude heads/suits/props) +bodies = [] +for r in rows: + if len(r) < 3 or not r[2]: + continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): + base = base[:-4] + if base.startswith("C") and base.endswith("Z") and "head" not in base.lower() \ + and "suit" not in base.lower(): + bodies.append(base[1:-1]) +bp, ba = worst_in(bodies) +print("== worst body ==") +print("%s P_conv=%X B_pc=%X B_n64=%X" % (ba["base"], bp, ba["B_pc"], ba["B_n64"])) + +for label, weapon in (("rifle", ra), ("pistol", pa_)): + sBpc = ba["B_pc"] + ha["B_pc"] + weapon["B_pc"] + sBn64 = ba["B_n64"] + ha["B_n64"] + weapon["B_n64"] + bound = sBpc + 2 * (0x18160 - sBn64) + print("chain(%s): sumB_pc=%X sumB_n64=%X strict-bound=%X vs PC region 0x1C000 -> %s" + % (label, sBpc, sBn64, bound, "FITS" if bound <= 0x1C000 else "SHORT by %X" % (bound - 0x1C000))) + +# ---- single-file strict bounds: P_final_actual <= B_pc + 2*(R_share_N64 - B_n64) +print() +print("== single-file strict bounds ==") +def check(name, r_n64, region_pc): + a = analyze(name) + if not a: + print("%-16s NOT FOUND" % name); return + bound = a["B_pc"] + 2 * (r_n64 - a["B_n64"]) + ok = "OK " if bound <= region_pc else "SHORT by %X" % (bound - region_pc) + print("%-16s B_pc=%6X B_n64=%6X bound=%7X vs region %X -> %s" + % (name, a["B_pc"], a["B_n64"], bound, region_pc, ok)) + +check("suit_lf_hand", 0xBD70, 0x18000) +check("trigger", 0xAFD0, 0x17000) +check("watchlaser", 0xAFD0, 0x17000) +check("walletbond", 0xA000, 0x17000) + +# weapons: worst over all G* files with R_share_N64 = 0x7530 +weapons = [] +for r in rows: + if len(r) < 3 or not r[2]: + continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): + base = base[:-4] + if base.startswith("G") and base.endswith("Z"): + weapons.append(base[1:-1]) +worst_w = None +for w in weapons: + a = analyze(w) + if not a: + continue + bound = a["B_pc"] + 2 * (0x7530 - a["B_n64"]) + if worst_w is None or bound > worst_w[0]: + worst_w = (bound, w, a) +print("worst weapon: %-14s bound=%X vs 0xF000 -> %s" + % (worst_w[1], worst_w[0], "OK" if worst_w[0] <= 0xF000 else "SHORT by %X" % (worst_w[0]-0xF000))) + +# bondview chain: body+head+weapon, R_chain_N64 = 0x14820 (size_item_buffer) +sBpc = ba["B_pc"] + ha["B_pc"] + ra["B_pc"] +sBn64 = ba["B_n64"] + ha["B_n64"] + ra["B_n64"] +bound = sBpc + 2 * (0x14820 - sBn64) +print("bondview chain: bound=%X vs 0x23000 -> %s" + % (bound, "OK" if bound <= 0x23000 else "SHORT by %X" % (bound - 0x23000))) diff --git a/tools_pc/d43_convert.py b/tools_pc/d43_convert.py new file mode 100644 index 00000000..acdb562e --- /dev/null +++ b/tools_pc/d43_convert.py @@ -0,0 +1,260 @@ +#!/usr/bin/env python3 +"""D43 reference model-file converter (N64 BE -> PC LE). Validates on all 512 files: +- layout: [switches NS*8][texconfigs NT*12][DFS nodes(48B)+records(PC sz)][vtx arrays][blobs][GDLs 16B slots] +- every pointer remap resolves via the unified region map +- GDL spans pack tight (16B multiples), D_PC - G_PC == 16*total_slots +- reports max D_PC vs N64 size (staging headroom) +This is the spec that port/src/romdata.c romdataFixupModelFile implements.""" +import csv, struct, zlib, os, re, sys +from collections import Counter + +ROM = open('data/ge007.ntsc-final.z64','rb').read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +MASK = 0xFFFFFF +def be16(b,o): return struct.unpack_from(">h",b,o)[0] +def bu16(b,o): return struct.unpack_from(">H",b,o)[0] +def be32o(b,o): return struct.unpack_from(">I",b,o)[0] & MASK # file offset (low 24) +def be32r(b,o): return struct.unpack_from(">I",b,o)[0] # raw u32 BE + +# N64 record sizes (from bondtypes.h comments, verified vs ROM via full walk) +N64_REC = {1:0x1C, 2:0x1C, 3:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, + 13:0x20, 15:0x1C, 18:0x08, 21:0x14, 22:0x10, 23:0x02, 24:0x20} +# PC record sizes (probe-verified) +PC_REC = {1:24, 2:40, 3:40, 4:40, 8:24, 9:48, 10:28, 12:48, + 13:48, 15:28, 18:16, 21:20, 22:32, 23:2, 24:64} +PC_NODE = 48 + +def walk(src, D, NS, NT): + """DFS with LOD/SWITCH rewiring. Returns (nodes, gdls) where + nodes = [(n64_off, op)] in visit order, gdls = [gdl_off] in visit order ([Primary, Secondary]).""" + R0 = 4*NS + 12*NT + seen = set(); stack = [R0]; nodes = []; gdls = [] + while stack: + o = stack.pop() + if o in seen or o >= D: continue + seen.add(o) + op = bu16(src, o) & 0xff + data = be32o(src, o+4) + nodes.append((o, op)) + child = be32o(src, o+0x14); nxt = be32o(src, o+0xC) + if op == 8: # LOD -> Affects (data+8) + a = be32o(src, data+8) + if a: stack.append(a) + elif op == 18: # SWITCH -> Controls (data+0) + c = be32o(src, data) + if c: stack.append(c) + else: + if child: stack.append(child) + if nxt: stack.append(nxt) + # GDLs in [Primary, Secondary] order at node entry + if op == 4: + p = be32o(src, data); s = be32o(src, data+4) + if p: gdls.append(p) + if s: gdls.append(s) + elif op == 24: + p = be32o(src, data); s = be32o(src, data+4) + if p: gdls.append(p) + if s: gdls.append(s) + elif op == 22: + p = be32o(src, data+8) + if p: gdls.append(p) + return nodes, gdls + +def gdl_end(src, D, g): + o = g + while o + 8 <= D: + if (be32r(src, o) >> 24) == 0xB8: return o + 8 + o += 8 + return D + +errors = [] +stats = Counter() +max_ratio = 0; max_file = "" +blob_info = [] +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key = cand; break + if not key: + errors.append(f"{base}: no header data"); continue + NS, NT = nsnt[key] + try: addr, size = int(r[0]), int(r[1]) + except ValueError: continue + try: src = zlib.decompress(ROM[addr:addr+size][2:], -15) + except Exception as e: + errors.append(f"{base}: decompress {e}"); continue + D = len(src); stats["files"] += 1 + + nodes, gdls = walk(src, D, NS, NT) + if not nodes: + errors.append(f"{base}: empty tree"); continue + + # ---- pass 1: layout + region map ---- + regions = [] # (old_off, old_size, new_off) + def add_region(old, osz, cur): + regions.append((old, osz, cur)) + return cur + osz + + dstpos = 0 + for i in range(NS): + dstpos = add_region(4*i, 4, dstpos) # switches 4B -> 8B each + for i in range(NT): + dstpos = add_region(4*NS + 12*i, 12, dstpos) # texconfigs 12B -> 12B + + node_newoff = {}; rec_newoff = {} + gdl_spans = [] # (old_g, slots_emitted) + vtx_regions = [] # for G_VTX seg5 remap sanity (subset of regions) + zero_vtx = [] # (vo,) arrays with nv==0 but non-null ptr — sized up to next object + for (no, op) in nodes: + data = be32o(src, no+4) + dstpos = add_region(no, 24, dstpos) # node 24B -> 48B + node_newoff[no] = dstpos - 24 + if op not in N64_REC: + errors.append(f"{base}: unknown opcode {op} at {no:#x}") + continue + nsz, psz = N64_REC[op], PC_REC[op] + rec_newoff[data] = dstpos + dstpos = add_region(data, nsz, dstpos) + # vertex arrays right after the record + # (nv==0 with non-null ptr: real data exists, GDL uploads via abs offset — + # size it up to the next object; see PexplosionbitZ) + if op == 4: + nv = bu16(src, data+0x10); vo = be32o(src, data+0xC) + if vo: + sz = 16*nv if nv else None + if sz is None: zero_vtx.append((vo,)) + else: vtx_regions.append((vo, nv)) + if nv: dstpos = add_region(vo, sz, dstpos) + elif op == 24: + nv = be16(src, data+0xC); ncv = be16(src, data+0xE) + vo = be32o(src, data+8); cvo = be32o(src, data+0x10); puo = be32o(src, data+0x14) + if nv and vo: + vtx_regions.append((vo, nv)); dstpos = add_region(vo, 16*nv, dstpos) + if ncv and cvo: + vtx_regions.append((cvo, ncv)); dstpos = add_region(cvo, 16*ncv, dstpos) + if nv and puo: + dstpos = add_region(puo, 2*nv, dstpos) # PointUsage s16 x numVertices + elif op == 22: + nv = struct.unpack_from(">i", src, data)[0]; vo = be32o(src, data+4) + if nv and vo: + vtx_regions.append((vo, nv)); dstpos = add_region(vo, 16*nv, dstpos) + + # zero-count vertex arrays: size up to the next higher object offset + obj_offs_now = sorted(set([no for no,_ in nodes]) | set(rec_newoff.keys()) | + set(v[0] for v in vtx_regions) | set(gdls)) + for (vo,) in zero_vtx: + nxt = min((o for o in obj_offs_now if o > vo), default=D) + sz = nxt - vo + if sz <= 0 or sz % 16: errors.append(f"{base}: zero-vtx array {vo:#x} bad size {sz:#x}") + else: + vtx_regions.append((vo, sz // 16)) + dstpos = add_region(vo, sz, dstpos) + + # embedded texture blobs (texconfig TextureID seg5) + blob_old = [] + for i in range(NT): + tid = be32r(src, 4*NS + 12*i) + if (tid >> 28) == 5: + blob_old.append(tid & MASK) + # blob size: up to the next higher object offset + all_offs = sorted(set([no for no,_ in nodes]) | set(rec_newoff.keys()) | + set(v[0] for v in vtx_regions) | set(gdls) | set(4*NS+12*i for i in range(NT))) + for bo in blob_old: + nxt = min((o for o in all_offs if o > bo), default=D) + dstpos = add_region(bo, nxt - bo, dstpos) + blob_info.append((base, hex(bo), nxt - bo)) + + # GDLs last, tight-packed 16B slots (up to and incl. ENDDL) + gdl_newoff = {} + total_slots = 0 + for g in gdls: + end = gdl_end(src, D, g) + nslots = (end - g) // 8 + if (end - g) % 8: errors.append(f"{base}: GDL {g:#x} not 8B aligned end") + gdl_newoff[g] = dstpos + dstpos += 16 * nslots + total_slots += nslots + regions.append((g, end - g, gdl_newoff[g])) + + D_PC = dstpos + G_PC = min(gdl_newoff.values()) if gdl_newoff else D_PC + if gdl_newoff and (D_PC - G_PC) % 16: + errors.append(f"{base}: D_PC-G_PC not multiple of 16") + ratio = D_PC / D if D else 0 + if ratio > max_ratio: max_ratio, max_file = ratio, base + stats["dpc_max"] = max(stats["dpc_max"], D_PC) + + # ---- pass 2: remap sanity (every pointer field must resolve) ---- + regions.sort() + import bisect + rold = [x[0] for x in regions]; rmap = {x[0]: x for x in regions} + def remap(off): + if off == 0: return 0 + i = bisect.bisect_right(rold, off) - 1 + if i < 0: return None + o, osz, n = regions[i] + if o <= off < o + osz: return n + (off - o) + return None + + def check(tag, off): + if off == 0: return + if remap(off) is None: + errors.append(f"{base}: {tag} -> {off:#x} not in map") + + for (no, op) in nodes: + data = be32o(src, no+4) + for f in (4, 8, 0xC, 0x10, 0x14): check(f"node{no:#x}+{f:#x}", be32o(src, no+f)) + if op not in N64_REC: continue + ptrfields = { + 1: [4], 2: [0x14], 3: [0x14], 8: [8], 9: [0x18, 0x1C], + 12: [0x18], 13: [0x10, 0x14], 18: [0], 24: [8, 0x10, 0x14], + 4: [0xC], 22: [4], + }.get(op, []) + for f in ptrfields: check(f"rec{op}@{data:#x}+{f:#x}", be32o(src, data+f)) + if op == 24: + ncv = be16(src, data+0xE) + cvo = be32o(src, data+0x10) + for i in range(ncv): + check(f"colvtx{cvo:#x}[{i}].LinkedTo", be32r(src, cvo + 16*i + 8) & MASK if (be32r(src, cvo+16*i+8)>>24)==5 else 0) + for i in range(NS): check(f"switch[{i}]", be32o(src, 4*i)) + + # GDL w1 seg5 targets must resolve — ONLY for address-bearing opcodes. + # (G_TRI4/G_TRI1/G_TEXTURE/SETOTHERMODE/etc carry index data in w1, not addresses.) + ADDR_OPS = {0x04, 0xFD, 0xF3} # G_VTX, G_SETTIMG, G_LOADBLOCK + for g in gdls: + o = g + while o + 8 <= D: + w0 = be32r(src, o); c = w0 >> 24 + if c == 0xB8: break + w1 = be32r(src, o+4) + if c in ADDR_OPS and (w1 >> 28) == 5: + check(f"gdl{g:#x}@{o:#x} op{c:#x} w1", w1 & MASK) + o += 8 + +stats["max_ratio"] = round(max_ratio, 3) +print("files:", stats["files"], " max D_PC:", hex(stats["dpc_max"]), f" max D_PC/D_N64 ratio: {max_ratio:.2f} ({max_file})") +print("embedded blobs:", blob_info) +if errors: + print(f"\n{len(errors)} ERRORS:") + for e in errors[:30]: print(" ", e) +else: + print("ALL CLEAN: every remap target resolves, layout invariants hold") diff --git a/tools_pc/d43_cover.py b/tools_pc/d43_cover.py new file mode 100644 index 00000000..e781ff43 --- /dev/null +++ b/tools_pc/d43_cover.py @@ -0,0 +1,83 @@ +# Verify every seg-5 VTX offset and SETTIMG w1 target falls inside a modeled object. +import struct, zlib +exec(open("tools_pc/d43_sizes.py").read().split("stats = []")[0]) + +RSZ = {1:0x10,2:0x1C,4:0x14,8:0x10,9:0x24,10:0x1C,12:0x28,13:0x20, + 15:0x1C,17:0x20,18:0x8,21:0x14,22:0x10,23:2,24:0x20} + +def be32(b,o): return struct.unpack_from(">I",b,o)[0] +def off(p): # vma 0x05xxxxxx -> file offset; non-vma pointers are invalid + return (p & 0xFFFFFF) if (p >> 24) == 5 else None + +def walk(out,D,NS,NT): + R0n=4*NS+12*NT + stack=[R0n]; seen=set(); objs=[]; gdls=[] + while stack: + o=stack.pop() + if o in seen or o>=D-23: break + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4) + objs.append((o,24,"node")) + rec=off(data) + if rec is not None and rech",out,rec+0xC)[0] if op==24 else struct.unpack_from(">H",out,rec+0x10)[0] + if nv>0 and v is not None and vh",out,rec+0xE)[0] + cv=off(be32(out,rec+0x10)); pu=off(be32(out,rec+0x14)) + if nc>0 and cv is not None and cv0 and pu is not None and pui",out,rec)[0]; v=off(be32(out,rec+4)) + p=off(be32(out,rec+8)) + if nv>0 and v is not None and v>24 + w1=be32(out,o+4) + top=(w1>>24)&0xff + if t==4 and top==5: + tot_vtx+=1 + x=w1&0xffffff + if not any(s<=x +import csv, struct, sys, zlib + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() + +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +entries = [] +for r in rows: + if len(r) < 3 or not r[2]: + continue + try: + addr, size = int(r[0]), int(r[1]) + except ValueError: + continue + entries.append((addr, size, r[2])) + +flt = sys.argv[1] if len(sys.argv) > 1 else "" +want = None +for addr, size, name in sorted(entries): + base = name.rsplit("/", 1)[-1] + if base.endswith(".bin"): + base = base[:-4] + if flt and flt.lower() in base.lower(): + want = (addr, size, base) + break +if not want: + sys.exit("no file matching %r" % flt) + +addr, size, base = want +data = rom[addr:addr + size] +out = zlib.decompress(data[2:], -15) +print("%s: rom=%d decomp=%d (0x%X)" % (base, size, len(out), len(out))) + +# N64 header-less layout: switches[NS] then textures[NT] then RootNode. +# We don't know NS/NT from the file alone; find them via the exe record is +# not available here, so heuristic: scan for a plausible node tree start. +# Instead: dump raw words and let the eye decode. Print first 64 bytes. +print("head:", out[:64].hex()) + +def be16(b, o): return struct.unpack_from(">H", b, o)[0] +def be32(b, o): return struct.unpack_from(">I", b, o)[0] + +# Try to find GDL streams: sequences of 8-byte words whose w0>>24 is a known cmd. +CMDS = {0xc0:"NOOP",0x05:"VTX",0x0f:"MTX",0x13:"DL",0x15:"MOVEWORD",0x19:"POPMTX", + 0x02:"MATRIXLOAD",0x07:"CLIPRATIO",0x0b:"SPSETOTHERMODE?",0x0d:"ENVMODE?", + 0x23:"RDPPPIPESYNC",0x25:"RDPSETSCISSOR?",0x26:"RDPSETOPTIONS?",0x27:"RDPLOADTLUT?", + 0x28:"RDPSETCONVERT?",0x29:"RDPSETSCENECULL?",0x2a:"RDPFILLRECT",0x2b:"RDPSETFOGCOLOR?", + 0x2c:"RDPSETPRIMCOLOR",0x2d:"RDPSETPROJECT?",0x2e:"RDPSETBADEVENTS?",0x2f:"RDPLOADUTILE?", + 0x31:"RDPSETENVCOLOR",0x32:"RDPSETCOMBINE",0xba:"RDPSETOTHERMODE_H",0xbb:"RDPSETOTHERMODE_L", + 0xbf:"RDPSETTEXIMAGE",0xb1:"RDPSETTILESIZE?",0xbbb:None} +def cmdname(c): + return CMDS.get(c, "0x%02x" % c) + +# scan for the longest run of plausible Gfx words +best = (0, 0, 0) +i = 0 +while i + 8 <= len(out): + w0 = be32(out, i) + c = w0 >> 24 + if c in CMDS: + j = i + while j + 8 <= len(out) and (be32(out, j) >> 24) in CMDS: + j += 8 + L = j - i + if L > best[0]: + best = (L, i, c) + i += 4 + +L, off, c0 = best +print("\nlongest GDL-like run: %d bytes at offset 0x%X (starts cmd %s)" % (L, off, cmdname(c0))) +# dump first 24 words of that run +for k in range(min(24, L // 8)): + w0 = be32(out, off + 8*k) + w1 = be32(out, off + 8*k + 4) + print(" [%2d] %s w0=%08X w1=%08X" % (k, cmdname(w0 >> 24), w0, w1)) diff --git a/tools_pc/d43_fullwalk.py b/tools_pc/d43_fullwalk.py new file mode 100644 index 00000000..e55d6620 --- /dev/null +++ b/tools_pc/d43_fullwalk.py @@ -0,0 +1,143 @@ +# D43: definitive N64 model-file layout analysis. +# Walks nodes via Child(+0x14) AND Next(+0x0C). Records all objects with sizes. +# Checks: full tiling [0,D), GDL contiguity at tail, node/record invariants. +import csv, struct, zlib, os, re, sys +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) + +# N64 record sizes per opcode (bondtypes.h, verified: no rec-overflow in 512 files) +RSZ = {1:0x10, 2:0x1C, 3:0x1C, 4:0x14, 5:0x1AC, 6:0x18, 7:0x1B0, 8:0x10, + 9:0x24, 10:0x1C, 11:0x50, 12:0x28, 13:0x20, 14:0x10, 15:0x1C, + 16:0x18, 17:0x20, 18:0x08, 21:0x14, 22:0x10, 23:0x02, 24:0x20} + +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be16(b, o): return struct.unpack_from(">H", b, o)[0] +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +def be32s(b, o): + v = struct.unpack_from(">i", b, o)[0] + return v & 0xFFFFFF if v >= 0 else -v + +nfiles = 0; fails = []; gaphist = Counter(); gdlnotatend = [] +nodecount_total = 0; sibling_only = 0 +onlychild_counts = Counter() + +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size = int(r[0]),int(r[1]) + except ValueError: continue + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key = cand; break + if not key: fails.append((base,"nohdr")); continue + NS,NT = nsnt[key] + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: fails.append((base,"decomp")); continue + D = len(out); nfiles += 1 + R0 = 4*NS + 12*NT + + # --- walk nodes: Child + Next --- + objs = [(0, R0, "hdr")] + stack = [(R0, 0)]; seen = set() + while stack: + o, via = stack.pop() + if o in seen or not (0 <= o < D): fails.append((base,f"badnode {o:#x}")); break + seen.add(o) + op = be16(out, o) & 0xff + data = be32(out, o+4) + child = be32(out, o+0x14); nxt = be32(out, o+0xC) + objs.append((o, o+24, "node")) + rsz = RSZ.get(op, 0) + if not rsz: fails.append((base,f"noRSZ op{op} @ {o:#x}")); break + if data + rsz > D: fails.append((base,f"recovf op{op} {data:#x}+{rsz:#x}>{D:#x}")); break + objs.append((data, data+rsz, f"rec{op}")) + # per-opcode sub-objects + if op == 4: # DL + p = be32(out, data); s = be32(out, data+4) + v = be32(out, data+0xC); nv = be16(out, data+0x10) + if p: objs.append((p, None, "gdl")) + if s: objs.append((s, None, "gdl")) + if v and nv: objs.append((v, v+16*nv, "vtx")) + elif op == 24: # DLCOLLISION + p = be32(out, data); s = be32(out, data+4) + v = be32(out, data+8); nv = be16(out, data+0xC); nc = be16(out, data+0xE) + cv = be32(out, data+0x10); puo = be32(out, data+0x14) + if p: objs.append((p, None, "gdl")) + if s: objs.append((s, None, "gdl")) + if v and nv: objs.append((v, v+16*nv, "vtx")) + if cv and nc: objs.append((cv, cv+16*nc, "coll")) + if puo: objs.append((puo, None, "pu")) + elif op == 22: # DLPRIMARY + nv = be32s(out, data) + v = be32(out, data+4); p = be32(out, data+8) + if p: objs.append((p, None, "gdl")) + if v and nv > 0: objs.append((v, v+16*nv, "vtx22")) + elif op == 9: # BSP: check for embedded pointers? (size 0x24) + pass + if child: stack.append((child, 1)) + if nxt: stack.append((nxt, 2)) + else: + nodecount_total += len(seen) + # count nodes not reachable as anyone's Child (sibling-only, excl. root) + childtargets = set() + for o in seen: + c = be32(out, o+0x14) + if c: childtargets.add(c) + sibling_only += len(seen - childtargets - {R0}) + # resolve open-ended (gdl/pu): sorted starts + starts = sorted(set(s for s,e,k in objs)) + fixed = [] + for (s,e,k) in objs: + if e is None: + nxts = [x for x in starts if x > s] + fixed.append((s, min(nxts) if nxts else D, k)) + else: + fixed.append((s,e,k)) + # tiling + ivs = sorted((s,e) for s,e,k in fixed) + pos = 0; gaps = [] + for s,e in ivs: + if s < pos: fails.append((base,f"overlap {s:#x} pos: gaps.append(s-pos) + pos = e + else: + if pos != D: fails.append((base,f"tail {pos:#x}!={D:#x}")) + for g in gaps: gaphist[g] += 1 + # GDL tail contiguity: all gdl objs must form one contiguous run ending at D + gs = sorted(set(s for s,e,k in fixed if k=="gdl")) + if gs: + # check union of gdl intervals == [min(gs), D) with no non-gdl inside + merged = [] + for s,e,k in sorted(fixed): + if k == "gdl": + if merged and s <= merged[-1][1]: merged[-1] = (merged[-1][0], max(merged[-1][1], e)) + else: merged.append([s,e]) + if len(merged) > 1 or merged[0][1] != D: + gdlnotatend.append((base, [f"{a:#x}-{b:#x}" for a,b in merged])) + +print(f"files: {nfiles} fails: {len(fails)}") +for f in fails[:20]: print(" ", f) +print(f"total nodes (Child+Next walk): {nodecount_total}") +print(f"nodes reached only via Next (not first sibling): {sibling_only}") +print(f"gap-size histogram: {dict(sorted(gaphist.items()))}") +print(f"GDL-not-single-tail-run files: {len(gdlnotatend)}") +for b, m in gdlnotatend[:10]: print(" ", b, m) diff --git a/tools_pc/d43_gdldump.py b/tools_pc/d43_gdldump.py new file mode 100644 index 00000000..718c4f11 --- /dev/null +++ b/tools_pc/d43_gdldump.py @@ -0,0 +1,92 @@ +# D43 helper: dump real GDL command streams from model files (N64 layout), +# showing w1 top-byte distribution per opcode. Usage: python d43_gdldump.py +import csv, struct, sys, zlib, os, re +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) + +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF + +flt = sys.argv[1] if len(sys.argv) > 1 else "" +limit = int(sys.argv[2]) if len(sys.argv) > 2 else 4 +shown = 0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + if flt and flt.lower() not in base.lower(): continue + try: addr, size = int(r[0]), int(r[1]) + except ValueError: continue + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key = cand; break + if not key: continue + NS, NT = nsnt[key] + try: out = zlib.decompress(rom[addr:addr + size][2:], -15) + except Exception: continue + D = len(out) + R0 = 4 * NS + 12 * NT + + # collect DL / DLCOLLISION record gdl offsets (full walk: Child+Next+Parent) + stack = [R0]; seen = set(); gdls = [] + while stack: + o = stack.pop() + if o in seen or o >= D: continue + seen.add(o) + op = struct.unpack_from(">H", out, o)[0] & 0xff + data = be32(out, o + 4) + parent = be32(out, o + 8); nxt = be32(out, o + 0xC) + child = be32(out, o + 0x14) + if op in (4, 24): + p = be32(out, data); s = be32(out, data + 4) + if p: gdls.append((p, "P")) + if s: gdls.append((s, "S")) + for q in (child, nxt, parent): + if q: stack.append(q) + + print("== %s D=0x%X NS=%d NT=%d gdls=%d" % (base, D, NS, NT, len(gdls))) + topbytes = Counter(); opcounts = Counter() + for goff, tag in gdls[:2]: + # walk the GDL: commands are 8B slots; stop at ENDDL or invalid opcode + i = goff; n = 0 + while i + 8 <= D and n < 400: + w0 = be32(out, i); w1 = be32(out, i + 4) + c = w0 >> 24 + opcounts[c] += 1 + if c == 0x1d: break # ENDDL + i += 8; n += 1 + print(" %s @0x%X (%d slots, ends at 0x%X)" % (tag, goff, n, i)) + for k in range(min(n, 14)): + w0 = be32(out, goff + 8 * k); w1 = be32(out, goff + 8 * k + 4) + print(" [%2d] c=%02X w0=%08X w1=%08X" % (k, w0 >> 24, w0, w1)) + for goff, tag in gdls: + i = goff; n = 0 + while i + 8 <= D and n < 400: + w0 = be32(out, i); w1 = be32(out, i + 4) + c = w0 >> 24 + if c == 0x1d: break + topbytes[(c, (w1 >> 24) & 0xff)] += 1 + i += 8; n += 1 + print(" opcode/topbyte histogram:") + for (c, tb), cnt in sorted(topbytes.items()): + print(" c=%02X top(w1)=%02X x%d" % (c, tb, cnt)) + shown += 1 + if shown >= limit: break diff --git a/tools_pc/d43_gdlhist.py b/tools_pc/d43_gdlhist.py new file mode 100644 index 00000000..f2bb7fb7 --- /dev/null +++ b/tools_pc/d43_gdlhist.py @@ -0,0 +1,80 @@ +# D43: histogram of (opcode, w1 top byte) across ALL model GDLs, plus per-file +# max GDL extent checks. Also verifies every GDL ends in ENDDL (0xB8). +import csv, struct, zlib, os, re +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +def be32raw(b, o): return struct.unpack_from(">I", b, o)[0] + +hist = Counter(); seg5 = Counter(); problems = [] +nfiles = 0; ngdl = 0; maxslots = 0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size = int(r[0]),int(r[1]) + except ValueError: continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: problems.append((base,"nohdr")); continue + NS,NT = nsnt[key] + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: problems.append((base,"decomp")); continue + D = len(out); R0 = 4*NS + 12*NT + stack=[R0]; seen=set(); gdls=[] + while stack: + o=stack.pop() + if o in seen or o>=D: problems.append((base,"badnode")); break + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4) + ch=be32(out,o+0x14) + if op in (4,24): + p=be32(out,data); s=be32(out,data+4) + if p: gdls.append(p) + if s: gdls.append(s) + # Child+Next only: matches modelIterateDisplayLists reachability + for q in (ch, be32(out,o+0xC)): + if q: stack.append(q) + nfiles += 1 + for g in set(gdls): + ngdl += 1 + i=g; n=0; ended=False + while i+8 <= D and n < 100000: + w0=be32raw(out,i); w1=be32raw(out,i+4) + c=w0>>24 + hist[(c,(w1>>24)&0xff)] += 1 + if (w1>>24)&0xff == 5 and c in (1,3,4,6,7,0xfd,0xfe,0xff): + seg5[(c,w1&0xffffff)] += 1 + if c==0xB8: ended=True; break + i+=8; n+=1 + maxslots=max(maxslots,n) + if not ended: problems.append((base,"gdl %X no ENDDL (n=%d)"%(g,n))) +print("files:",nfiles,"gdls:",ngdl,"max slots in one gdl:",maxslots) +print("\n(opcode, w1 top byte) histogram:") +for (c,tb),cnt in sorted(hist.items()): + print(" c=%02X seg=%02X x%d" % (c,tb,cnt)) +print("\nseg-5 address commands (opcode, offset):") +for (c,off),cnt in sorted(seg5.items()): + print(" c=%02X off=%06X x%d" % (c,off,cnt)) +print("\nproblems:",len(problems)) +for p in problems[:10]: print(" ",p) diff --git a/tools_pc/d43_gdlorder.py b/tools_pc/d43_gdlorder.py new file mode 100644 index 00000000..c5232b4a --- /dev/null +++ b/tools_pc/d43_gdlorder.py @@ -0,0 +1,144 @@ +# D43: EXACT modelIterateDisplayLists visit-order simulation (mutable nodes, +# LOD/SWITCH rewiring, BSP sibling splice). Verify per file that consecutive +# visited GDLs are offset-adjacent (count == cur size), i.e. a converter may +# emit GDLs in visit order and compaction will work. +import csv, struct, zlib, os, re + +ROM=r"data/ge007.ntsc-final.z64" +rom=open(ROM,"rb").read() +rows=list(csv.reader(open(r"scripts/filelist.u.csv"))) + +hdrfiles=set() +for root,dirs,files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root,fn)) +nsnt={} +for f in sorted(hdrfiles): + for line in open(f): + m=re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$",line) + if not m: continue + a=[x.strip() for x in m.group(1).split(",")] + if len(a)<9: continue + try: nsnt[a[0]]=(int(a[4],0),int(a[8],0)) + except ValueError: pass + +def be16(b,o): return struct.unpack_from(">H",b,o)[0] +def be32(b,o): return struct.unpack_from(">I",b,o)[0]&0xFFFFFF + +class N: # mutable node + __slots__=("op","data","parent","next","prev","child") + +bad=0; total=0; gdls_total=0 +for r in rows: + if len(r)<3 or not r[2]: continue + base=r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base=base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size=int(r[0]),int(r[1]) + except ValueError: continue + key=None + for cand in (base,base[1:],base[:-1],base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT=nsnt[key] + try: out=zlib.decompress(rom[addr:addr+size][2:],-15) + except Exception: continue + D=len(out); R0=4*NS+12*NT; total+=1 + + nodes={} + stack=[R0] + while stack: + o=stack.pop() + if o in nodes: continue + n=N(); n.op=be16(out,o)&0xff; n.data=be32(out,o+4) + n.parent=be32(out,o+8); n.next=be32(out,o+0xC); n.prev=be32(out,o+0x10); n.child=be32(out,o+0x14) + nodes[o]=n + # Child+Next closure (mutable walk reaches siblings via Next), plus + # LOD.Affects / SWITCH.Controls targets in case they lie outside it. + for q in (n.child, n.next): + if q: stack.append(q) + if n.op==8: + q=be32(out,n.data+8) + if q: stack.append(q) + elif n.op==18: + q=be32(out,n.data+0) + if q: stack.append(q) + + def gdl_of(n): + if n.op==4 or n.op==24: + p=be32(out,n.data+0); s=be32(out,n.data+4) + return (p,s) + if n.op==22: # DLPRIMARY N64: numVertices s32@0, Vertices@4, Primary@8, BaseAddr@0xC + return (be32(out,n.data+8), 0) + return (0,0) + + # full visit sequence: iterate the function to exhaustion + seq=[] + node=R0; prev_node=None; prev_gdl=0; guard=0 + while node and guard<200000: + guard+=1 + n=nodes[node] + gdl=0 + if n.op in (4,22,24): + p,s=gdl_of(n) + if node!=prev_node: + gdl=p + elif s and s!=prev_gdl: + gdl=s + elif n.op==8: # LOD rewire + aff=be32(out,n.data+8) + n.child=aff + elif n.op==18: # SWITCH rewire + ctl=be32(out,n.data+0) + n.child=ctl + elif n.op==9: # BSP splice (visible=TRUE) + lc=be32(out,n.data+0x18); rc=be32(out,n.data+0x1C) + node1,node2=lc,rc + if node1: + n.child=node1 + nodes[node1].prev=0 + loop=node1 + while nodes[loop].next and nodes[loop].next!=node2: + loop=nodes[loop].next + nodes[loop].next=node2 + if node2: + nodes[node2].prev=loop + loop=node2 + while nodes[loop].next and nodes[loop].next!=node1: + loop=nodes[loop].next + nodes[loop].next=0 + else: + n.child=node2 + if node2: nodes[node2].prev=0 + if gdl: + seq.append(gdl) + prev_node=node; prev_gdl=gdl + # next call resumes at same node (nodeptr unchanged) -> loop continues with same node + continue + prev_node=None + if n.child: + node=n.child + else: + while node: + nn=nodes[node].next + if nn: node=nn; break + node=nodes[node].parent + # verification: strictly ascending, no duplicates, each GDL ENDDLs before + # the next visited start (so a converter may pack them contiguously). + ok=True; why="" + for i,g in enumerate(seq): + end = seq[i+1] if i+1I",out,j)[0]>>24 + if c==0xB8: ended=True; break + j+=8; n+=1 + if not ended: ok=False; why="no ENDDL in [%X,%X)"%(g,end); break + if not ok: + bad+=1 + if bad<=5: print("BAD",base,why,["%X"%v for v in seq[:10]],"D=%X"%D) + gdls_total+=len(seq) + +print("files:",total,"bad:",bad,"total gdls:",gdls_total) diff --git a/tools_pc/d43_gdlseq.py b/tools_pc/d43_gdlseq.py new file mode 100644 index 00000000..5221cf3d --- /dev/null +++ b/tools_pc/d43_gdlseq.py @@ -0,0 +1,56 @@ +import csv, struct, zlib, os, re, sys +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +def be32raw(b, o): return struct.unpack_from(">I", b, o)[0] +flt = sys.argv[1] if len(sys.argv) > 1 else "CboilerbondZ" +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if flt.lower() not in base.lower(): continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + addr,size=int(r[0]),int(r[1]) + out=zlib.decompress(rom[addr:addr+size][2:],-15) + D=len(out); R0=4*NS+12*NT + stack=[R0]; seen=set(); gdls=[] + while stack: + o=stack.pop() + if o in seen or o>=D: break + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4); ch=be32(out,o+0x14) + if op in (4,24): + p=be32(out,data); s=be32(out,data+4) + if p: gdls.append((p,"P",o)) + if s: gdls.append((s,"S",o)) + for q in (ch, be32(out,o+0xC)): + if q: stack.append(q) + print("== %s D=0x%X gdls=%d" % (base,D,len(gdls))) + g = min(x[0] for x in gdls) + i=g; n=0 + while i+8 <= D and n < 120: + w0=be32raw(out,i); w1=be32raw(out,i+4) + c=w0>>24 + print(" [%3d] c=%02X w0=%08X w1=%08X" % (n,c,w0,w1)) + if c==0xB8: break + i+=8; n+=1 + break diff --git a/tools_pc/d43_invariants.py b/tools_pc/d43_invariants.py new file mode 100644 index 00000000..07fc3fa7 --- /dev/null +++ b/tools_pc/d43_invariants.py @@ -0,0 +1,111 @@ +# D43: full ownership tiling check per file. +# objects: switches[0,4NS) textures[4NS,4NS+12NT) nodes(20B) records(RSZ) +# vtx(nv*16) coll(nc*16) gdl(up to next gdl start or D) +# PointUsage: [puo, min(next owned start)) -- verify even & >0 +# then verify union == [0,D) exactly (no gaps/overlaps). +import csv, struct, zlib, os, re +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +RSZ={1:0x10,2:0x1C,4:0x14,8:0x10,9:0x24,10:0x1C,18:0x08,21:0x14,23:0x02,24:0x20} + +hdrfiles=set() +for root,dirs,files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root,fn)) +nsnt={} +for f in sorted(hdrfiles): + for line in open(f): + m=re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$",line) + if not m: continue + a=[x.strip() for x in m.group(1).split(",")] + if len(a)<9: continue + try: nsnt[a[0]]=(int(a[4],0),int(a[8],0)) + except ValueError: pass + +def be16(b,o): return struct.unpack_from(">H",b,o)[0] +def be32(b,o): return struct.unpack_from(">I",b,o)[0]&0xFFFFFF + +fails=[]; pu_sizes=Counter(); nfiles=0; gapfiles=[] +for r in rows: + if len(r)<3 or not r[2]: continue + base=r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base=base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size=int(r[0]),int(r[1]) + except ValueError: continue + key=None + for cand in (base,base[1:],base[:-1],base[1:-1]): + if cand in nsnt: key=cand; break + if not key: fails.append((base,"no header")); continue + NS,NT=nsnt[key] + try: out=zlib.decompress(rom[addr:addr+size][2:],-15) + except Exception: fails.append((base,"decomp")); continue + D=len(out); nfiles+=1 + R0=4*NS+12*NT + + objs=[] # (start,end,kind) + pu_starts=[] + stack=[R0]; nodeset=set() + while stack: + o=stack.pop() + if o in nodeset or o>=D: fails.append((base,"bad node %X"%o)); break + nodeset.add(o) + op=be16(out,o)&0xff + data=be32(out,o+4); child=be32(out,o+20) + objs.append((o,o+20,"node")) + rsz=RSZ.get(op,0) + if not rsz or data+rsz>D: fails.append((base,"bad rec op%d"%op)); break + objs.append((data,data+rsz,"rec")) + if op==4: + for f_off in (0,4): + v=be32(out,data+f_off) + if v: objs.append((v,None,"gdl")) + v=be32(out,data+0xC); nv=be16(out,data+0x10) + if v: objs.append((v,v+16*nv,"vtx")) + elif op==24: + for f_off in (0,4): + v=be32(out,data+f_off) + if v: objs.append((v,None,"gdl")) + v=be32(out,data+8); nv=be16(out,data+0xC); nc=be16(out,data+0xE) + cv=be32(out,data+0x10); puo=be32(out,data+0x14) + if v: objs.append((v,v+16*nv,"vtx")) + if cv: objs.append((cv,cv+16*nc,"coll")) + if puo: pu_starts.append(puo) + if child: stack.append(child) + else: + # gdl extents: sorted gdl starts, each extends to next gdl start or D + gs=sorted(set(s for s,e,k in objs if k=="gdl")) + fixed=[] + for i,(s,e,k) in enumerate(objs): + if k=="gdl": + end = gs[gs.index(s)+1] if gs.index(s)+1puo] + end=min(nxt) if nxt else D + sz=end-puo + if sz<=0 or sz%2: fails.append((base,"pu bad size %d @%X"%(sz,puo))); break + objs.append((puo,end,"pu")) + pu_sizes[sz]+=1 + else: + # tiling check + ivs=sorted((s,e) for s,e,k in objs) + ok=True; pos=0 + for s,e in ivs: + if spos: gapfiles.append((base,hex(pos),hex(s))); ok=False; break + pos=e + if ok and pos!=D: fails.append((base,"short end %X vs D=%X"%(pos,D))) + +print("files:",nfiles,"fails:",len(fails)) +for f in fails[:15]: print(" ",f) +print("gap files:",len(gapfiles)) +for g in gapfiles[:10]: print(" ",g) +print("\nPointUsage sizes:",dict(sorted(pu_sizes.items()))) diff --git a/tools_pc/d43_layout.py b/tools_pc/d43_layout.py new file mode 100644 index 00000000..f658ee4a --- /dev/null +++ b/tools_pc/d43_layout.py @@ -0,0 +1,106 @@ +# D43: full layout analysis of all model files with CORRECT N64 sizes. +# node = 24B; records per opcode; vertex/coll arrays; GDLs. Report gaps/overlaps. +import csv, struct, zlib, os, re +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +RSZ = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:0x2, 24:0x20} +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +def be16(b, o): return struct.unpack_from(">H", b, o)[0] +def be32raw(b, o): return struct.unpack_from(">I", b, o)[0] + +gapstats = Counter(); problems = [] +nfiles = 0; totalnodes = 0; opcodes = Counter() +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size = int(r[0]),int(r[1]) + except ValueError: continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: problems.append((base,"nohdr")); continue + NS,NT = nsnt[key] + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: problems.append((base,"decomp")); continue + D = len(out); R0 = 4*NS + 12*NT; nfiles += 1 + stack=[R0]; seen=set(); objs=[] + while stack: + o=stack.pop() + if o in seen or o>=D-23: problems.append((base,"badnode %X"%o)); break + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + opcodes[op]+=1 + data=be32(out,o+4) + objs.append((o,o+24,"node")) + if op in RSZ: + rsz = RSZ[op] + if data + rsz > D: problems.append((base,"rec %X+%X>D"%(data,rsz))) + else: objs.append((data,data+rsz,"rec%d"%op)) + if op==22: + nv=struct.unpack_from(">i",out,data)[0] + v=be32(out,data+4) + p=be32(out,data+8) + if p: objs.append((p,None,"gdl")) + if v and nv>0: objs.append((v,v+16*nv,"vtx22")) + if op==4: + p=be32(out,data); s=be32(out,data+4) + v=be32(out,data+0xC) + if p: objs.append((p,None,"gdl")) + if s: objs.append((s,None,"gdl")) + if v: objs.append((v,None,"vtx?")) # size unknown (numVertices may be 0) + elif op==24: + p=be32(out,data); s=be32(out,data+4) + v=be32(out,data+8); nv=be16(out,data+0xC) + cv=be32(out,data+0x10); nc=be16(out,data+0xE) + pu=be32(out,data+0x14) + if p: objs.append((p,None,"gdl")) + if s: objs.append((s,None,"gdl")) + if v and nv: objs.append((v,v+16*nv,"vtx")) + if cv and nc: objs.append((cv,cv+16*nc,"coll")) + if pu: objs.append((pu,None,"pointusage?")) + for q in (be32(out,o+0x14), be32(out,o+0xC)): + if q: stack.append(q) + totalnodes += len(seen) + # resolve open-ended objects using sorted starts + known = sorted(set(s for s,e,k in objs if e is not None)) + for i,(s,e,k) in enumerate(objs): + if e is None: + nxt = [x for x in known if x > s] + objs[i] = (s, min(nxt) if nxt else D, k) + ivs = sorted((s,e) for s,e,k in [(0,R0,"hdr")]+objs) + pos = 0 + for s,e in ivs: + if s < pos: problems.append((base,"overlap %X<%X"%(s,pos))); break + if s > pos: + gapstats[(s-pos)] += 1 + if len(problems) < 40: + problems.append((base, "gap %X..%X (%d B)" % (pos, s, s - pos))) + break + pos = e + else: + if pos != D: problems.append((base,"tail gap %X!=D=%X"%(pos,D))) + +print("files:",nfiles,"nodes:",totalnodes) +print("opcodes:",dict(sorted(opcodes.items()))) +print("gap-size histogram (first gap per file):",dict(sorted(gapstats.items())[:20])) +print("problems:",len(problems)) +for p in problems[:40]: print(" ",p) diff --git a/tools_pc/d43_layout1.py b/tools_pc/d43_layout1.py new file mode 100644 index 00000000..38421074 --- /dev/null +++ b/tools_pc/d43_layout1.py @@ -0,0 +1,77 @@ +import csv, struct, zlib, os, re, sys +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +RSZ = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:0x2, 24:0x20} +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +def be16(b, o): return struct.unpack_from(">H", b, o)[0] +flt = sys.argv[1] +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if flt.lower() not in base.lower(): continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + addr,size=int(r[0]),int(r[1]) + out=zlib.decompress(rom[addr:addr+size][2:],-15) + D=len(out); R0=4*NS+12*NT + print("== %s NS=%d NT=%d R0=%X D=%X" % (base,NS,NT,R0,D)) + stack=[R0]; seen=set(); objs=[] + while stack: + o=stack.pop() + if o in seen or o>=D-23: break + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4) + objs.append((o,o+24,"node%d"%op)) + if op in RSZ: objs.append((data,data+RSZ[op],"rec%d"%op)) + if op==4: + p=be32(out,data); s=be32(out,data+4); v=be32(out,data+0xC) + if p: objs.append((p,None,"gdlP")) + if s: objs.append((s,None,"gdlS")) + if v: objs.append((v,None,"vtx?")) + elif op==24: + p=be32(out,data); s=be32(out,data+4) + v=be32(out,data+8); nv=be16(out,data+0xC) + cv=be32(out,data+0x10); nc=be16(out,data+0xE) + pu=be32(out,data+0x14) + if p: objs.append((p,None,"gdlP")) + if s: objs.append((s,None,"gdlS")) + if v and nv: objs.append((v,v+16*nv,"vtx(%d)"%nv)) + if cv and nc: objs.append((cv,cv+16*nc,"coll(%d)"%nc)) + if pu: objs.append((pu,None,"pointusage?")) + for q in (be32(out,o+0x14), be32(out,o+0xC)): + if q: stack.append(q) + known = sorted(set(s for s,e,k in objs if e is not None)) + res=[] + for (s,e,k) in objs: + if e is None: + nxt=[x for x in known if x>s] + e=min(nxt) if nxt else D + res.append((s,e,k)) + res.sort() + pos=0 + for s,e,k in res: + mark = "" + if spos: mark=" <<< GAP %d B"% (s-pos) + print(" %05X..%05X %-14s%s" % (s,e,k,mark)) + pos=max(pos,e) + break diff --git a/tools_pc/d43_lutscan.py b/tools_pc/d43_lutscan.py new file mode 100644 index 00000000..8d16e049 --- /dev/null +++ b/tools_pc/d43_lutscan.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +"""Scan all model files' texture tables -> image headers -> format/maxlod. +Determines whether any LUT textures (formats 9-12) exist, which affects +worst-case marker expansion sizing (D45).""" +import csv, struct, zlib, os, re +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() + +imgrows = list(csv.reader(open("imagelist.u.csv"))) +img_off = [int(r[0]) for r in imgrows] +img_sz = [int(r[1]) for r in imgrows] + +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) + +nsnt = {} +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + f = os.path.join(root, fn) + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: + continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: + continue + try: + nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: + pass + +def bitread(buf, pos, n): + val = 0 + for i in range(n): + byte = buf[pos >> 3] + bit = 7 - ((pos & 7) % 8) + val = (val << 1) | ((byte >> bit) & 1) + pos += 1 + return val, pos + +fmtall = Counter() +fmtlut = Counter() +maxlod_dist = Counter() +nimg = 0 +bad = 0 +zc = 0 +matched_files = 0 +lut_examples = [] + +for r in rows: + if len(r) < 3 or not r[2]: + continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): + base = base[:-4] + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: + key = cand + break + if not key: + continue + matched_files += 1 + NS, NT = nsnt[key] + addr, size = int(r[0]), int(r[1]) + out = zlib.decompress(rom[addr:addr + size][2:], -15) + D = len(out) + texoff = 4 * NS + 12 * NT + for i in range(NT): + tid = struct.unpack_from(">I", out, texoff + i * 12)[0] + if tid >= len(img_off): + bad += 1 + continue + nimg += 1 + o = img_off[tid] + s = img_sz[tid] + hdr = rom[o] + iszlib = (hdr & 0x40) >> 6 + lod = hdr & 0x3f + maxlod_dist[min(lod, 7)] += 1 + if iszlib: + zc += 1 + fmt = (rom[o + 1] >> 4) & 0xf + else: + b2 = rom[o + 1:o + s] + fmt, _ = bitread(b2, 0, 4) + fmtall[fmt] += 1 + if fmt in (9, 10, 11, 12): + fmtlut[fmt] += 1 + if len(lut_examples) < 10: + lut_examples.append((key, i, tid, fmt)) + +print("nsnt entries:", len(nsnt)) +print("matched model files:", matched_files, "tex refs:", nimg, "bad:", bad, "zlib:", zc) +print("formats used:", dict(sorted(fmtall.items()))) +print("LUT formats (9-12):", dict(fmtlut)) +print("maxlod dist:", dict(sorted(maxlod_dist.items()))) +for e in lut_examples: + print(" LUT example:", e) diff --git a/tools_pc/d43_pointusage.py b/tools_pc/d43_pointusage.py new file mode 100644 index 00000000..dca7e74a --- /dev/null +++ b/tools_pc/d43_pointusage.py @@ -0,0 +1,85 @@ +# D43: for every op24 record, compare PointUsage byte extent (to next owned object) +# against numVertices*2 and other candidates. +import csv, struct, zlib, os, re +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +RSZ = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:0x2, 24:0x20} +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +def be16(b, o): return struct.unpack_from(">H", b, o)[0] +from collections import Counter +rel = Counter(); examples=[] +nrec=0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size = int(r[0]),int(r[1]) + except ValueError: continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: continue + D = len(out); R0 = 4*NS + 12*NT + stack=[R0]; seen=set(); objs=[]; recs=[] + while stack: + o=stack.pop() + if o in seen or o>=D-23: break + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4) + objs.append((o,o+24,"node")) + if op in RSZ: objs.append((data,data+RSZ[op],"rec")) + if op==4: + p=be32(out,data); s=be32(out,data+4); v=be32(out,data+0xC) + if p: objs.append((p,None,"gdl")) + if s: objs.append((s,None,"gdl")) + if v: objs.append((v,None,"vtx")) + elif op==24: + p=be32(out,data); s=be32(out,data+4) + v=be32(out,data+8); nv=be16(out,data+0xC) + nc=be16(out,data+0xE); cv=be32(out,data+0x10) + pu=be32(out,data+0x14) + if p: objs.append((p,None,"gdl")) + if s: objs.append((s,None,"gdl")) + if v and nv: objs.append((v,v+16*nv,"vtx")) + if cv and nc: objs.append((cv,cv+16*nc,"coll")) + if pu: objs.append((pu,None,"pu")) + recs.append((base,pu,nv,nc)) + for q in (be32(out,o+0x14), be32(out,o+0xC)): + if q: stack.append(q) + known = sorted(set(s for s,e,k in objs if e is not None)) + ends={} + for (s,e,k) in objs: + if e is None: + nxt=[x for x in known if x>s] + e=min(nxt) if nxt else D + ends[s]=e + for (base,pu,nv,nc) in recs: + nrec+=1 + ext = ends[pu]-pu + rel[ext - 2*nv]+=1 + if ext != 2*nv and len(examples)<10: + examples.append((base,hex(pu),ext,nv,nc)) +print("op24 records:",nrec) +print("(PointUsage extent - 2*numVertices) histogram:") +for k,c in sorted(rel.items()): print(" %+d x%d" % (k,c)) +print("mismatch examples (base, pu, extent, nv, nc):") +for e in examples: print(" ",e) diff --git a/tools_pc/d43_seg5.py b/tools_pc/d43_seg5.py new file mode 100644 index 00000000..80f2a91a --- /dev/null +++ b/tools_pc/d43_seg5.py @@ -0,0 +1,86 @@ +# D43: for every seg-5 VTX/MOVEMEM/MTX/DL reference in every GDL, check whether +# the offset falls inside a known vertex/coll array or elsewhere. +import csv, struct, zlib, os, re +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +RSZ={1:0x10,2:0x1C,4:0x14,8:0x10,9:0x24,10:0x1C,18:0x08,21:0x14,23:0x02,24:0x20} +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +def be16(b, o): return struct.unpack_from(">H", b, o)[0] +def be32raw(b, o): return struct.unpack_from(">I", b, o)[0] + +where = Counter(); examples = [] +nfiles=0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size = int(r[0]),int(r[1]) + except ValueError: continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: continue + D = len(out); R0 = 4*NS + 12*NT; nfiles+=1 + stack=[R0]; seen=set(); gdls=[]; arrays=[] + while stack: + o=stack.pop() + if o in seen or o>=D: break + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4); ch=be32(out,o+0x14) + if op==4: + p=be32(out,data); s=be32(out,data+4) + if p: gdls.append(p) + if s: gdls.append(s) + v=be32(out,data+0xC); nv=be16(out,data+0x10) + if v: arrays.append((v, v+16*nv, "vtx(op4)")) + elif op==24: + p=be32(out,data); s=be32(out,data+4) + if p: gdls.append(p) + if s: gdls.append(s) + v=be32(out,data+8); nv=be16(out,data+0xC) + cv=be32(out,data+0x10); nc=be16(out,data+0xE) + if v: arrays.append((v, v+16*nv, "vtx(op24)")) + if cv: arrays.append((cv, cv+16*nc, "coll(op24)")) + for q in (ch, be32(out,o+0xC)): + if q: stack.append(q) + for g in set(gdls): + i=g; n=0 + while i+8 <= D and n < 100000: + w0=be32raw(out,i); w1=be32raw(out,i+4) + c=w0>>24 + if c in (1,3,4,6,7) and ((w1>>24)&0xff)==5: + off = w1 & 0xFFFFFF + hit=None + for a,b,k in arrays: + if a <= off < b: hit=k; break + where[(c,hit)] += 1 + if hit is None and len(examples) < 8: + examples.append((base, c, hex(off))) + if c==0xB8: break + i+=8; n+=1 +print("files:",nfiles) +print("(opcode, region) for seg-5 refs:") +for (c,hit),cnt in sorted(where.items(), key=lambda x:(x[0][0],str(x[0][1]))): + print(" c=%02X %-10s x%d" % (c, str(hit), cnt)) +print("non-array examples:", examples) diff --git a/tools_pc/d43_seg5ops.py b/tools_pc/d43_seg5ops.py new file mode 100644 index 00000000..19b6d189 --- /dev/null +++ b/tools_pc/d43_seg5ops.py @@ -0,0 +1,88 @@ +#!/usr/bin/env python3 +"""Dump opcode + words for every seg-5 w1 slot in model GDLs, grouped by opcode.""" +import csv, struct, zlib, os, re +from collections import Counter, defaultdict + +ROM = open('data/ge007.ntsc-final.z64','rb').read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +MASK = 0xFFFFFF +def be32o(b,o): return struct.unpack_from(">I",b,o)[0] & MASK +def be32r(b,o): return struct.unpack_from(">I",b,o)[0] + +by_op = defaultdict(Counter) # opcode -> Counter of w1 values (seg5 only) +inrange = Counter(); outrange = Counter() +samples = defaultdict(list) +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + try: addr,size=int(r[0]),int(r[1]) + except ValueError: continue + try: src=zlib.decompress(ROM[addr:addr+size][2:],-15) + except Exception: continue + D=len(src); R0=4*NS+12*NT + seen=set(); stack=[R0]; gdls=[] + while stack: + o=stack.pop() + if o in seen or o>=D: continue + seen.add(o) + op=struct.unpack_from(">H",src,o)[0]&0xff + data=be32o(src,o+4) + child=be32o(src,o+0x14); nxt=be32o(src,o+0xC) + if op==8: + a=be32o(src,data+8) + if a: stack.append(a) + elif op==18: + c=be32o(src,data) + if c: stack.append(c) + else: + if child: stack.append(child) + if nxt: stack.append(nxt) + if op in (4,24): + p=be32o(src,data); s=be32o(src,data+4) + gdls += [x for x in (p,s) if x] + elif op==22: + p=be32o(src,data+8) + if p: gdls.append(p) + for g in sorted(set(gdls)): + o=g + while o+8<=D: + w0=be32r(src,o); c=w0>>24 + if c==0xB8: break + w1=be32r(src,o+4) + if (w1>>28)==5: + off=w1&MASK + by_op[c][off]+=1 + if off < D: inrange[c]+=1 + else: outrange[c]+=1 + if len(samples[c])<6 and off>=D: samples[c].append((base,hex(o),hex(w0),hex(w1))) + o+=8 + +print("opcode -> (in-range count, out-of-range count):") +for c in sorted(set(list(inrange)+list(outrange))): + print(f" 0x{c:02X}: in={inrange[c]} out={outrange[c]} distinct_vals={len(by_op[c])}") +print() +for c,v in samples.items(): + print(f"samples opcode 0x{c:02X} (file, slot_off, w0, w1):") + for s in v: print(" ", s) diff --git a/tools_pc/d43_seg5vtx.py b/tools_pc/d43_seg5vtx.py new file mode 100644 index 00000000..b9414444 --- /dev/null +++ b/tools_pc/d43_seg5vtx.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +"""Analyze G_VTX w1 seg-5 usage: do offsets match vertex-array file offsets? +Also: quantify trailing junk after ENDDL within each GDL span (for the converter's +packing decision), and per-opcode record counts across all 512 files.""" +import csv, struct, zlib, os, re +from collections import Counter + +rom = open('data/ge007.ntsc-final.z64','rb').read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b,o): return struct.unpack_from(">I",b,o)[0] & 0xFFFFFF +def be32r(b,o): return struct.unpack_from(">I",b,o)[0] + +seg5_offs = Counter() +matched = 0; unmatched = []; total_seg5 = 0 +files_with_seg5 = set() +opcounts = Counter() +junk_total = 0; junk_files = 0; tight = 0; loose = 0 +nsz_bad = [] + +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + try: addr,size=int(r[0]),int(r[1]) + except ValueError: continue + try: out=zlib.decompress(rom[addr:addr+size][2:],-15) + except Exception: continue + D=len(out); R0=4*NS+12*NT + seen=set(); stack=[R0]; vtoffs=set(); gdls=[] + while stack: + o=stack.pop() + if o in seen or o>=D: continue + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4) + opcounts[op]+=1 + if op==4: + vtoffs.add(be32(out,data+0xC)) + gdls += [x for x in (be32(out,data), be32(out,data+4)) if x] + elif op==24: + vtoffs.add(be32(out,data+8)); vtoffs.add(be32(out,data+0x10)) + gdls += [x for x in (be32(out,data), be32(out,data+4)) if x] + elif op==22: + vtoffs.add(be32(out,data+4)) + p=be32(out,data+8) + if p: gdls.append(p) + for q in (be32(out,o+0xC), be32(out,o+0x14)): + if q: stack.append(q) + # GDL spans in visit order = sorted unique offsets + gs = sorted(set(gdls)) + for i,g in enumerate(gs): + end = gs[i+1] if i+1 < len(gs) else D + n=0; en=None + o=g + while o+8<=end: + w0=be32r(out,o); c=w0>>24 + if c==0x04 and ((w1:=be32r(out,o+4))>>24)&0xf == 5: + off = w1 & 0xFFFFFF + seg5_offs[off]+=1; total_seg5+=1 + files_with_seg5.add(base) + if off in vtoffs: matched+=1 + elif len(unmatched)<10: unmatched.append((base, hex(w1), sorted(vtoffs)[:6])) + if c==0xB8: en=o+8; break + o+=8; n+=1 + if en is None: + junk_total += (end-g) + elif en < end: + loose += 1; junk_total += end-en + else: + tight += 1 + # N64 record size check for op5/op11 trailing bytes + # (sizes already known from RSZ map; skip here) + +print("G_VTX seg5 total:", total_seg5, "in files:", len(files_with_seg5)) +print("matched to a vertex-array offset in same file:", matched) +print("unmatched samples:", unmatched[:6]) +print("seg5 offset histogram (top 10):", [(hex(k),v) for k,v in seg5_offs.most_common(10)]) +print() +print("GDL span tightness: tight=%d loose=%d total_trailing_bytes=%d" % (tight, loose, junk_total)) +print() +print("opcode record counts across all files:") +for op in sorted(opcounts): print(f" op{op}: {opcounts[op]}") diff --git a/tools_pc/d43_seg5vtx2.py b/tools_pc/d43_seg5vtx2.py new file mode 100644 index 00000000..8739c493 --- /dev/null +++ b/tools_pc/d43_seg5vtx2.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +"""Seg-5 G_VTX analysis WITH LOD/SWITCH rewiring (as in d43_gdlorder.py). +Question: does every seg-5 G_VTX offset point INSIDE a vertex array of the same file? +(vo + d where d % 16 == 0 and d < 16*nv) — i.e. is it (array_base_file_offset + displacement)?""" +import csv, struct, zlib, os, re +from collections import Counter + +rom = open('data/ge007.ntsc-final.z64','rb').read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b,o): return struct.unpack_from(">I",b,o)[0] & 0xFFFFFF +def be32r(b,o): return struct.unpack_from(">I",b,o)[0] + +total_seg5 = 0; inside = 0; outside = [] +files_with = set() +vtx_regions = Counter() # (d mod 16) histogram of displacement +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + try: addr,size=int(r[0]),int(r[1]) + except ValueError: continue + try: out=zlib.decompress(rom[addr:addr+size][2:],-15) + except Exception: continue + D=len(out); R0=4*NS+12*NT + + # walk with LOD/SWITCH rewiring; collect vertex regions (vo, nv) and GDL offsets per record + seen=set(); stack=[R0]; regions=[]; gdls=[] + while stack: + o=stack.pop() + if o in seen or o>=D: continue + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4) + child = be32(out,o+0x14); nxt = be32(out,o+0xC) + if op==8: # LOD: Child := Affects (data+8) + a = be32(out,data+8) + if a: stack.append(a) + elif op==18: # SWITCH: Child := Controls (data+0) + c = be32(out,data) + if c: stack.append(c) + else: + if child: stack.append(child) + if nxt: stack.append(nxt) + if op==4: + nv=struct.unpack_from(">H",out,data+0x10)[0] + vo=be32(out,data+0xC) + regions.append((vo,nv)) + gdls += [x for x in (be32(out,data), be32(out,data+4)) if x] + elif op==24: + nv=struct.unpack_from(">h",out,data+0xC)[0] + cv=struct.unpack_from(">h",out,data+0xE)[0] + vo=be32(out,data+8); cvo=be32(out,data+0x10) + regions.append((vo,nv)) + if cv: regions.append((cvo,cv)) + gdls += [x for x in (be32(out,data), be32(out,data+4)) if x] + elif op==22: + nv=struct.unpack_from(">i",out,data)[0] + vo=be32(out,data+4) + regions.append((vo,nv)) + p=be32(out,data+8) + if p: gdls.append(p) + + gs = sorted(set(gdls)) + for i,g in enumerate(gs): + end = gs[i+1] if i+1 < len(gs) else D + o=g + while o+8<=end: + w0=be32r(out,o); c=w0>>24 + if c==0x04: + w1=be32r(out,o+4) + if (w1>>24)&0xf == 5: + off = w1 & 0xFFFFFF + total_seg5 += 1; files_with.add(base) + hit=False + for vo,nv in regions: + d = off - vo + if 0 <= d < 16*nv and nv>0: + vtx_regions[d & 0xf]+=1 + hit=True; break + if hit: inside+=1 + elif len(outside)<12: outside.append((base, hex(off), [(hex(a),b) for a,b in regions[:8]])) + o+=8 + +print("seg5 G_VTX total:", total_seg5, "files:", len(files_with)) +print("pointing inside a vertex array (vo+d, d<16nv):", inside) +print("displacement mod 16 histogram:", dict(vtx_regions)) +print("outside samples:") +for s in outside: print(" ", s) diff --git a/tools_pc/d43_sizes.py b/tools_pc/d43_sizes.py new file mode 100644 index 00000000..f72b8005 --- /dev/null +++ b/tools_pc/d43_sizes.py @@ -0,0 +1,125 @@ +# D43: PC image size budget (final). +# +# Converter output layout (PC image, little-endian): +# [switches NS*8][textures NT*12][nodes 48B + records pc-size, DFS preorder] +# [bulk arrays: vertex/coll/pointusage, verbatim sizes, s16/f32 bswapped] +# [GDLs at tail in visit order, 16B Gfx slots, no inter-GDL padding] +# +# sub_GAME_7F0762E0 mirror+expand contract: +# D = pc_remaining (PC image size), g1 = first GDL offset, R = buffer. +# texCopyGdls mirrors [g1,D) to [R-D+g1,R); texLoadFromGdl expands each GDL +# rightward from g1 reading the mirror. Each N64 8B slot becomes one PC 16B +# slot, so the GDL region doubles: input extent of GDL k = 2*E_k (E_k = N64 +# extent bytes). Fit and no-clobber both reduce to: +# R >= P_final = B_pc + sum(2*E_k) + 16 * sum_markers(K_t - 1) +# K_t = worst-case output slots a marker of type t emits (tex.c analysis; +# only types 2/3/4 occur in ROM): +RSZ_N64 = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, + 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:2, 24:0x20} +RSZ_PC = {1:0x18, 2:0x28, 4:0x28, 8:0x18, 9:0x30, 10:0x1C, 12:0x30, + 13:0x30, 15:0x1C, 17:0x28, 18:0x10, 21:0x14, 22:0x20, 23:2, 24:0x40} +NODE_N64, NODE_PC = 24, 48 +# exact worst-case output slots per marker (tex.c helpers, maxlod<=8, +# all texTry* state guards emitting, valid=FALSE branch, water check on): +# preamble = PipeSync(<=1) + texWriteTextureCmd(<=1) = 2 +# LoadToTmemAddr/Zero <= 9, TileFromDef <= 4, TileLods <= 3*maxlod, +# Tile <= 3, light begin/end = 0, water sub = 1 +K_BOUND = {0:2+9+4+3*7+1, 1:2+9+1+9+4+3*7+1, 2:2+9+3*8+1, 3:2+9+3+3+1, 4:2+9+3+1} +R_BUF = 0x14820 # size_item_buffer (gun.c), shared body/head/props per hand + +import csv, struct, zlib, os, re +from collections import Counter +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF + +def gdl_marker_slots(out, g, e): + # count texture markers (G_NOOP w1 type field) in [g,e); returns {type:count} + c = Counter() + j = g + while j + 8 <= e: + w0 = struct.unpack_from(">I", out, j)[0] + if (w0 >> 24) == 0xC0: + t = w0 & 7 + c[t] += 1 + j += 8 + return c + +stats = [] +nfiles = 0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr, size = int(r[0]), int(r[1]) + except ValueError: continue + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key = cand; break + if not key: continue + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: continue + D = len(out); NS, NT = nsnt[key] + R0n = 4*NS + 12*NT + # walk nodes (DFS preorder), collect records + GDLs + stack = [R0n]; seen = set(); nodes = []; gdls = [] + while stack: + o = stack.pop() + if o in seen or o >= D-23: break + seen.add(o) + op = struct.unpack_from(">H", out, o)[0] & 0xff + data = be32(out, o+4) + nodes.append((o, op, data)) + if op in (4, 24): + p, s = be32(out, data), be32(out, data+4) + if p: gdls.append(p) + if s: gdls.append(s) + elif op == 22: + p = be32(out, data+8) + if p: gdls.append(p) + for q in (be32(out, o+0x14), be32(out, o+0xC)): + if q: stack.append(q) + gsorted = sorted(set(gdls)) + g1 = gsorted[0] if gsorted else D + # bulk arrays (vertex/coll/pointusage + alignment padding) keep N64 byte + # counts on PC: 16B/vertex, 16B/coll vertex, 2B/PU entry. + bulk = (g1 - R0n) - len(nodes)*NODE_N64 - sum(RSZ_N64[op] for _, op, _ in nodes) + B_pc = 8*NS + 12*NT + len(nodes)*NODE_PC + sum(RSZ_PC[op] for _, op, _ in nodes) + bulk + G_in = 0; Kslack = 0 + for i, g in enumerate(gsorted): + e = gsorted[i+1] if i+1 < len(gsorted) else D + assert (e - g) % 8 == 0, (base, hex(g), e - g) + G_in += 2 * (e - g) # each N64 8B slot -> one PC 16B slot + for t, n in gdl_marker_slots(out, g, e).items(): + Kslack += n * (K_BOUND.get(t, 44) - 1) + P_final = B_pc + G_in + 16*Kslack + stats.append((base, D, B_pc, G_in, 16*Kslack, P_final)) + nfiles += 1 + +print("files:", nfiles) +worst = sorted(stats, key=lambda s: -s[5])[:15] +for b, D, Bp, Gi, K, Pf in worst: + flag = "OVER" if Pf > R_BUF else " " + print("%-22s N64=%7X B_pc=%7X G_in=%7X Kslack=%7X P_final=%8X %s (R=%X)" + % (b, D, Bp, Gi, K, Pf, flag, R_BUF)) +over = [s for s in stats if s[5] > R_BUF] +print("files over budget:", len(over), "of", nfiles) +totD = sum(s[1] for s in stats); totP = sum(s[5] for s in stats) +print("sum N64 bytes: %X sum P_final: %X" % (totD, totP)) diff --git a/tools_pc/d43_sizes2.py b/tools_pc/d43_sizes2.py new file mode 100644 index 00000000..0909df63 --- /dev/null +++ b/tools_pc/d43_sizes2.py @@ -0,0 +1,125 @@ +# D43: PC image size budget (final). +# +# Converter output layout (PC image, little-endian): +# [switches NS*8][textures NT*12][nodes 48B + records pc-size, DFS preorder] +# [bulk arrays: vertex/coll/pointusage, verbatim sizes, s16/f32 bswapped] +# [GDLs at tail in visit order, 16B Gfx slots, no inter-GDL padding] +# +# sub_GAME_7F0762E0 mirror+expand contract: +# D = pc_remaining (PC image size), g1 = first GDL offset, R = buffer. +# texCopyGdls mirrors [g1,D) to [R-D+g1,R); texLoadFromGdl expands each GDL +# rightward from g1 reading the mirror. Each N64 8B slot becomes one PC 16B +# slot, so the GDL region doubles: input extent of GDL k = 2*E_k (E_k = N64 +# extent bytes). Fit and no-clobber both reduce to: +# R >= P_final = B_pc + sum(2*E_k) + 16 * sum_markers(K_t - 1) +# K_t = worst-case output slots a marker of type t emits (tex.c analysis; +# only types 2/3/4 occur in ROM): +RSZ_N64 = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, + 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:2, 24:0x20} +RSZ_PC = {1:0x18, 2:0x28, 4:0x28, 8:0x18, 9:0x30, 10:0x1C, 12:0x30, + 13:0x30, 15:0x1C, 17:0x28, 18:0x10, 21:0x14, 22:0x20, 23:2, 24:0x40} +NODE_N64, NODE_PC = 24, 48 +# exact worst-case output slots per marker (tex.c helpers, maxlod<=8, +# all texTry* state guards emitting, valid=FALSE branch, water check on): +# preamble = PipeSync(<=1) + texWriteTextureCmd(<=1) = 2 +# LoadToTmemAddr/Zero <= 9, TileFromDef <= 4, TileLods <= 3*maxlod, +# Tile <= 3, light begin/end = 0, water sub = 1 +K_BOUND = {0:2+9+4+3*7+1, 1:2+9+1+9+4+3*7+1, 2:2+9+3*8+1, 3:2+9+3+3+1, 4:2+9+3+1} +R_BUF = 0x14820 # size_item_buffer (gun.c), shared body/head/props per hand + +import csv, struct, zlib, os, re +from collections import Counter +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF + +def gdl_marker_slots(out, g, e): + # count texture markers (G_NOOP w1 type field) in [g,e); returns {type:count} + c = Counter() + j = g + while j + 8 <= e: + w0 = struct.unpack_from(">I", out, j)[0] + if (w0 >> 24) == 0xC0: + t = w0 & 7 + c[t] += 1 + j += 8 + return c + +stats = [] +nfiles = 0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/", 1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr, size = int(r[0]), int(r[1]) + except ValueError: continue + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key = cand; break + if not key: continue + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: continue + D = len(out); NS, NT = nsnt[key] + R0n = 4*NS + 12*NT + # walk nodes (DFS preorder), collect records + GDLs + stack = [R0n]; seen = set(); nodes = []; gdls = [] + while stack: + o = stack.pop() + if o in seen or o >= D-23: break + seen.add(o) + op = struct.unpack_from(">H", out, o)[0] & 0xff + data = be32(out, o+4) + nodes.append((o, op, data)) + if op in (4, 24): + p, s = be32(out, data), be32(out, data+4) + if p: gdls.append(p) + if s: gdls.append(s) + elif op == 22: + p = be32(out, data+8) + if p: gdls.append(p) + for q in (be32(out, o+0x14), be32(out, o+0xC)): + if q: stack.append(q) + gsorted = sorted(set(gdls)) + g1 = gsorted[0] if gsorted else D + # bulk arrays (vertex/coll/pointusage + alignment padding) keep N64 byte + # counts on PC: 16B/vertex, 16B/coll vertex, 2B/PU entry. + bulk = (g1 - R0n) - len(nodes)*NODE_N64 - sum(RSZ_N64[op] for _, op, _ in nodes) + B_pc = 8*NS + 12*NT + len(nodes)*NODE_PC + sum(RSZ_PC[op] for _, op, _ in nodes) + bulk + G_in = 0; Kslack = 0 + for i, g in enumerate(gsorted): + e = gsorted[i+1] if i+1 < len(gsorted) else D + assert (e - g) % 8 == 0, (base, hex(g), e - g) + G_in += 2 * (e - g) # each N64 8B slot -> one PC 16B slot + for t, n in gdl_marker_slots(out, g, e).items(): + Kslack += n * (K_BOUND.get(t, 44) - 1) + P_final = B_pc + G_in + 16*Kslack + R_need2 = B_pc + G_in + 32*Kslack + stats.append((base, D, B_pc, G_in, 16*Kslack, P_final, Kslack, R_need2)) + nfiles += 1 + +print("files:", nfiles) +worst = sorted(stats, key=lambda s: -s[5])[:15] +for b, D, Bp, Gi, K, Pf, Ks, R2 in worst: + print("%-22s N64=%7X B_pc=%7X G_in=%7X Mtot=%5d P_final=%8X R_need2=%8X" % (b, D, Bp, Gi, Ks, Pf, R2)) +over = [s for s in stats if s[5] > R_BUF] +print("max 16*M_total:", max(s[7]-s[5] for s in stats), "hex %X" % max(s[7]-s[5] for s in stats)) +print("files over budget:", len(over), "of", nfiles) +totD = sum(s[1] for s in stats); totP = sum(s[5] for s in stats) +print("sum N64 bytes: %X sum P_final: %X" % (totD, totP)) diff --git a/tools_pc/d43_tree_dump.py b/tools_pc/d43_tree_dump.py new file mode 100644 index 00000000..e8ee4d86 --- /dev/null +++ b/tools_pc/d43_tree_dump.py @@ -0,0 +1,48 @@ +import csv, struct, sys, zlib, os, re +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass +def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF +flt = sys.argv[1] +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if flt.lower() not in base.lower(): continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + addr,size=int(r[0]),int(r[1]) + out=zlib.decompress(rom[addr:addr+size][2:],-15) + D=len(out); R0=4*NS+12*NT + print("== %s D=0x%X NS=%d NT=%d R0=0x%X" % (base,D,NS,NT,R0)) + seen=set() + def show(o, depth): + if o in seen or o>=D: return + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4); par=be32(out,o+8); nxt=be32(out,o+0xC); prv=be32(out,o+0x10); ch=be32(out,o+0x14) + print(" %snode op=%-2d @%05X data=%05X par=%05X next=%05X prev=%05X child=%05X" % (" "*depth, op, o, data, par, nxt, prv, ch)) + if data < D: + # print first 12 words of the record + ws=[be32(out,data+4*k) for k in range(min(6,(D-data)//4))] + print(" rec: "+ " ".join("%06X"%w for w in ws)) + if ch: show(ch, depth+1) + if nxt: show(nxt, depth+1) + show(R0, 0) + break diff --git a/tools_pc/d43_vtxfmt.py b/tools_pc/d43_vtxfmt.py new file mode 100644 index 00000000..94dc4f75 --- /dev/null +++ b/tools_pc/d43_vtxfmt.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +"""Verify collision-vertex format: @8 is 0x05xxxxxx-or-0 (LinkedTo node ptr), +@C is a small s16 index. Also verify normal-vertex arrays never have 0x05xxxxxx at @8.""" +import csv, struct, zlib, os, re +from collections import Counter + +rom = open('data/ge007.ntsc-final.z64','rb').read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0)) + except ValueError: pass + +def be32(b,o): return struct.unpack_from(">I",b,o)[0] & 0xFFFFFF +def be32r(b,o): return struct.unpack_from(">I",b,o)[0] + +coll_bad8 = Counter(); coll_idx_range = [1<<30, -1<<30] +norm_weird8 = 0; norm_total = 0; coll_total = 0 +files_checked = 0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + key=None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: key=cand; break + if not key: continue + NS,NT = nsnt[key] + try: addr,size=int(r[0]),int(r[1]) + except ValueError: continue + try: out=zlib.decompress(rom[addr:addr+size][2:],-15) + except Exception: continue + D=len(out); R0=4*NS+12*NT + files_checked += 1 + seen=set(); stack=[R0] + while stack: + o=stack.pop() + if o in seen or o>=D: continue + seen.add(o) + op=struct.unpack_from(">H",out,o)[0]&0xff + data=be32(out,o+4) + child = be32(out,o+0x14); nxt = be32(out,o+0xC) + if op==8: + a = be32(out,data+8) + if a: stack.append(a) + elif op==18: + c = be32(out,data) + if c: stack.append(c) + else: + if child: stack.append(child) + if nxt: stack.append(nxt) + if op==4 or op==22: + if op==4: + nv=struct.unpack_from(">H",out,data+0x10)[0]; vo=be32(out,data+0xC) + else: + nv=struct.unpack_from(">i",out,data)[0]; vo=be32(out,data+4) + for i in range(nv): + v = vo + 16*i + w8 = be32r(out, v+8) + norm_total += 1 + if w8 != 0 and (w8 >> 24) != 5: + norm_weird8 += 1 + elif op==24: + nv=struct.unpack_from(">h",out,data+0xC)[0] + ncv=struct.unpack_from(">h",out,data+0xE)[0] + vo=be32(out,data+8); cvo=be32(out,data+0x10) + for i in range(nv): + v = vo + 16*i + w8 = be32r(out, v+8) + norm_total += 1 + if w8 != 0 and (w8 >> 24) != 5: + norm_weird8 += 1 + for i in range(ncv): + v = cvo + 16*i + w8 = be32r(out, v+8) + idx = struct.unpack_from(">h", out, v+0xC)[0] + coll_total += 1 + if w8 != 0 and (w8 >> 24) != 5: + coll_bad8[w8 & 0xFFFFFF] += 1 + coll_idx_range[0] = min(coll_idx_range[0], idx) + coll_idx_range[1] = max(coll_idx_range[1], idx) + +print("files:", files_checked, " normal vtx:", norm_total, " weird @8 (non-0/non-seg5):", norm_weird8) +print("collision vtx:", coll_total, " bad @8 values:", dict(list(coll_bad8.items())[:10])) +print("CollisionRelatedIndex range:", coll_idx_range) diff --git a/tools_pc/d43_walk.py b/tools_pc/d43_walk.py new file mode 100644 index 00000000..aca8afa0 --- /dev/null +++ b/tools_pc/d43_walk.py @@ -0,0 +1,97 @@ +# D43 helper: walk every model file's node tree (N64 layout) and report +# opcode histogram + structural sanity. Usage: python d43_walk.py +import csv, struct, zlib, sys +from collections import Counter + +ROM = r"data/ge007.ntsc-final.z64" +rom = open(ROM, "rb").read() +rows = list(csv.reader(open(r"scripts/filelist.u.csv"))) + +OP={1:"HEADER",2:"GROUP",3:"OP03",4:"DL",5:"OP05",6:"OP06",7:"OP07",8:"LOD",9:"BSP",10:"BOX",11:"OP11",12:"GUNFIRE",13:"SHADOW",14:"OP14",15:"INTERLINK",16:"OP16",17:"OP17",18:"SWITCH",21:"GROUPSIMPLE",22:"DLPRIMARY",23:"HEADPH",24:"DLCOLLISION"} + +# N64 record sizes (bytes) per opcode, from bondtypes.h +RSZ={1:0x10,2:0x1C,3:0x1C,4:0x14,5:0x1AC,6:0x18,7:0x1B0,8:0x10,9:0x24,10:0x1C, + 11:0x50,12:0x28,13:0x20,14:0x10,15:0x1C,16:0x18,17:0x20,18:0x08,21:0x14, + 22:0x10,23:0x02,24:0x20} + +# (numSwitches, numtextures) per model from the exe headers would be ideal; +# instead parse assets/obseg/**/ModelFileHeader.inc.c + body records. +import re, glob +nsnt = {} +import os +hdrfiles = set() +for root, dirs, files in os.walk("assets"): + for fn in files: + if fn.lower().endswith("modelfileheader.inc.c"): + hdrfiles.add(os.path.join(root, fn)) +nsnt = {} +for f in sorted(hdrfiles): + for line in open(f): + m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line) + if not m: continue + a = [x.strip() for x in m.group(1).split(",")] + if len(a) < 9: continue + name = a[0] + try: + NS = int(a[4], 0); NT = int(a[8], 0) + except ValueError: + continue + nsnt[name] = (NS, NT) + +# body/chr records live in other .inc.c files with the same macro? check c_item_entries etc. +print("headers parsed:", len(nsnt)) + +opcount = Counter() +files_ok = 0 +files_bad = [] +total_nodes = 0 +maxP = 0 +for r in rows: + if len(r) < 3 or not r[2]: continue + base = r[2].rsplit("/",1)[-1] + if base.endswith(".bin"): base = base[:-4] + if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue + try: addr,size=int(r[0]),int(r[1]) + except ValueError: continue + try: out = zlib.decompress(rom[addr:addr+size][2:], -15) + except Exception: files_bad.append((base,"decomp")); continue + D = len(out) + key = None + for cand in (base, base[1:], base[:-1], base[1:-1]): + if cand in nsnt: + key = cand; break + NS,NT = nsnt.get(key, (0,0)) if key else (0,0) + if NT is None: + # try to find NT from a record table elsewhere; fallback scan + files_bad.append((base,"no-NT")) + continue + root_off = 4*NS + 12*NT + def be16(o): return struct.unpack_from(">H", out, o)[0] + def be32(o): return struct.unpack_from(">I", out, o)[0] & 0xFFFFFF + seen = set() + ok = True + n = 0 + stack = [root_off] if root_off < D else [] + while stack: + o = stack.pop() + if o in seen or o >= D: ok=False; break + seen.add(o) + op = be16(o) & 0xff + opcount[op] += 1 + n += 1 + data = be32(o+4); child = be32(o+20) + if not (0 < data < D): ok=False; break + # validate record fits + rsz = RSZ.get(op, 0) + if rsz and data + rsz > D: ok=False; files_bad.append((base,"rec-overflow op%d @%X+%X>%X"%(op,data,rsz,D))); break + if child: stack.append(child) + total_nodes += n + if ok: files_ok += 1 + else: files_bad.append((base,"walk-fail root=%X NS=%d NT=%d D=%X"%(root_off,NS,NT,D))) + +print("files ok:", files_ok, " bad:", len(files_bad)) +for b in files_bad[:20]: print(" ", b) +print("total nodes:", total_nodes) +print("opcode histogram:") +for op in sorted(opcount): + print(" op%-3d %-12s x%d" % (op, OP.get(op,"??"), opcount[op]))