forked from Ports/goldeneye-pc-port
PC port: phase 2 — D48 review: re-plan D43 as offline pre-conversion (Plan B)
- D48 finding in PCPortResearch.md §F: d43_convert.py does not emit bytes (layout+remap validation only); Plan B = Python emit pass -> data/pcmodels-<region>/ RZ sidecars served via port-layer hw_address/rom_size table patch (no C converter); Plan A kept as fallback with corrected staging guard (D_N64+D_PC<=avail) - Moved all 23 d43_*.py investigation scripts from gitignored build-pc/ to tracked tools_pc/ (spec is now versioned); d43_cover.py path ref updated; converter re-run from new location: ALL CLEAN - HANDOFF.md rewritten as two-session prompt: Session A = independent review of Plan B (8 claims, each with falsification criterion -> finding D49), Session B = execute (emit pass + round-trip validator, romdata plumbing, one-line hook, pixel-assert soak); verified-facts converter spec preserved - AGENTS.md phase status updated to D48 re-plan
This commit is contained in:
James Dansereau committed
1 parent
181f6db07d
commit
73ea03734d
26 files changed
+2839
-216
No files matched your search
@@ -48,6 +48,6 @@ Run `/linkcheck` for this sweep. Record new findings in `docs/PCPortResearch.md`
|
||||
- **Compile+link milestone:** all ~235 TUs compile and `ge007.x86_64.exe` links clean (`ninja ge007 -k 0`, 236/236). Findings D7–D17 in `docs/PCPortResearch.md` §11; asset stubs in `port/src/assetstubs.c`, fast3d no-ops in `port/src/gfxstub.c` (delete when real fast3d lands).
|
||||
- **Phase 1 (boot to window):** done — ROM loads/validates/maps at cart base 0x10000000 (`romdata.c`), SDL2 window + GL clear loop (`video.c`), absolute asset symbols from `scripts/gen_romassets.py` → `port/src/romassets_<r>.s` (findings D18–D23).
|
||||
- **Phase 1.5 (boot to first frame):** DONE — boot chain complete AND real frames render (`[NOTE] frame N rendered in … us`). D34–D42 closed the blockers on top of D31–D33: ANIM_DATA lvalues, music seq-table ABI+endianness, music-heap sizing, libaudio bank-tree re-layout, implicit-declaration prototype shim (D38), Globalimagetable rebasing + true segment size (D39), ModelHitEntry pool sizing (D40), cross-thread GL context release (D41), rsp task-settings toggle (D42). ROM dump verified against the No-Intro good dump (byte-identical, header CRCs re-checked — `tools_pc/romverify.c`).
|
||||
- **Phase 2 (rendering):** IN PROGRESS — fast3d integrated + first frames render; current blocker is **D43** (model-file loading ABI mismatch: N64-layout ROM model files read as 8-byte-pointer PC structs → D37-style re-layout in `port/src/romdata.c`; a Rare-validated near-analogue exists in the PD port's `preprocess/filemodel.c` — §2.4). **D43 status:** all 512 model files validated; buffer sizing finalized by the D46 overlap-safety analysis (strict bound from N64 ground truth); step-2 mechanical game-code edits APPLIED (uncommitted); **D47 closed every open converter question** — compaction state values, G_VTX encoding, opcode-aware remap rule, vertex formats, staging contract — and the full conversion spec is implemented + validated 512/512 by `build-pc/d43_convert.py` (max D_PC = 0xB7E0, ratio ≤ 1.31). Done this session: D46 buffer corrections (front.c 0x25000 / initmenus.c 0x85000) + fast3d seg_addr seg-5 raw-VMA case. Remaining: C implementation of `romdataFixupModelFile` in `port/src/romdata.c` (port d43_convert.py 1:1) + hook in load_object_fill_header (+ poolRemaining=0 reset), build + soak — full checklist in `docs/HANDOFF.md`; findings D47 in §F. Recurring sub-task: ABI/layout fixes per the D32 procedure (§H) as more asset types are first touched (PD's `preprocess/` module already solves most of them).
|
||||
- **Phase 2 (rendering):** IN PROGRESS — fast3d integrated + first frames render; current blocker is **D43** (model-file loading ABI mismatch: N64-layout ROM model files read as 8-byte-pointer PC structs → D37-style re-layout). **D43 status:** all 512 model files validated; buffer sizing finalized by the D46 overlap-safety analysis; step-2 mechanical game-code edits APPLIED + committed (ecefffdb); **D47 closed every open converter question** — compaction state values, G_VTX encoding, opcode-aware remap rule, vertex formats — spec implemented + validated 512/512 by `tools_pc/d43_convert.py` (max D_PC = 0xB7E0, ratio ≤ 1.31). **D48 review re-plan:** the old "port d43_convert.py 1:1 to C" task is discarded — that script does not emit bytes, and **Plan B (offline pre-conversion)** is now the default: Python emit pass → `data/pcmodels-<region>/` RZ sidecars served via a port-layer `hw_address`/`rom_size` table patch (no C converter; full mechanics in §F/D48). Next session independently reviews Plan B (HANDOFF Task 1 → finding D49), then executes (Task 2); fallback Plan A = D47.13 C converter with the corrected staging guard (D48.4: `D_N64 + D_PC ≤ avail`). All 23 d43_*.py scripts moved from gitignored build-pc/ to tracked `tools_pc/`. Recurring sub-task: ABI/layout fixes per the D32 procedure (§H) as more asset types are first touched (PD's `preprocess/` module already solves most of them; prefer the offline pattern for format work — D48).
|
||||
- **Phase 3 (audio + input):** libaudio → SDL — largely copy-and-adapt from PD (`mixer.c` 722 lines vs our 31-line stub; `input.c` SDL_GameController backend; §2.4); decide the ASP strategy (C2: `aspMain*` dummies in `ucode.c`).
|
||||
- **Phase 4 (saves + polish):** file-backed EEPROM — build on PD's `fs.c` (file-backed save dir); PFS/motor already no-op stubs.
|
||||
+273
-215
@@ -1,110 +1,254 @@
|
||||
# Handoff brief — GoldenEye 007 PC port (Phase 1.5 done, Phase 2: first frames render)
|
||||
# Handoff brief — GoldenEye 007 PC port (Phase 2: first frames render; D43 re-planned as offline pre-conversion)
|
||||
|
||||
_Paste-ready briefing for the next agent session. The authoritative detail lives in
|
||||
`AGENTS.md` and `docs/PCPortResearch.md` (§H handoff, §F/D31–D46 findings); this is
|
||||
the summary + the immediate tasks._
|
||||
_Paste-ready briefing. Authoritative detail lives in `AGENTS.md` and
|
||||
`docs/PCPortResearch.md` (§F/D31–D48 findings, §H handoff); this is the summary
|
||||
+ the immediate tasks._
|
||||
|
||||
## Your job
|
||||
One task left on D43 (everything else is done and validated):
|
||||
1. ~~**D44:** one-line fix to the crash handler~~ — **DONE, committed.** Every
|
||||
crash now auto-logs a `BACKTRACE:` section to stdout and `ge007.crash.log`.
|
||||
2. **D43 (main task):** model-file loading ABI mismatch — stage object load faults
|
||||
in `modelPromoteNodeOffsetsToPointers()` (src/game/model.c:5688) because ROM model
|
||||
files are serialized with N64 layout (4-byte pointer fields) but read on PC as
|
||||
8-byte-pointer structs. **Investigation COMPLETE; the converter spec is fully
|
||||
pinned down and validated 512/512 by `build-pc/d43_convert.py`** (see §F/D47).
|
||||
What remains: **the C implementation of `romdataFixupModelFile()` in
|
||||
`port/src/romdata.c`, the hook in `load_object_fill_header`, build + soak.**
|
||||
After that keep pushing rendering forward (more asset types will fault the
|
||||
same way — Tbg_*_stanZ backgrounds next).
|
||||
## Your job — TWO sessions, in order
|
||||
|
||||
Work agentically — fix → build (~5 s) → verify → commit at each working milestone.
|
||||
Commit message style: `PC port: <phase> — <what>`.
|
||||
- **Session A (next): INDEPENDENT REVIEW of Plan B (finding D48).** Do NOT
|
||||
implement anything. Verify the 8 claims in §Task 1 against code + ROM, each
|
||||
with its falsification criterion. Record results as finding **D49** in
|
||||
`docs/PCPortResearch.md` §F (per-item CONFIRMED/REFUTED + evidence), update
|
||||
this file's status, commit. If every item is confirmed, Session B is green-lit.
|
||||
If any is refuted: record why, then either adjust Plan B per the finding or
|
||||
fall back to Plan A (§Task 3). The goal does not change: get stage geometry
|
||||
on screen.
|
||||
- **Session B (after D49 confirms): EXECUTE Plan B** per §Task 2 — Python emit
|
||||
pass + sidecar generation, ~60 lines of port-layer plumbing, one-line hook,
|
||||
build + pixel-assert soak. Commit at each working checkpoint
|
||||
(`PC port: phase 2 — <what>`).
|
||||
|
||||
Work agentically — fix → build (~5 s) → verify → commit at each milestone.
|
||||
|
||||
## Read first (authoritative, in order)
|
||||
1. `AGENTS.md` — non-negotiables, critical files, build, verification ritual, phase status.
|
||||
2. `docs/PCPortResearch.md` §H (handoff + plan), then **§F/D47** (converter
|
||||
contract finalized — read before writing the C converter), **D46** (overlap
|
||||
safety + final buffer sizing), **D45** (buffer constants), **D43** (the open
|
||||
model-file task), **§2.4** (PD-port copy-candidate audit —
|
||||
read before starting D43 or Phase 3/4), **D37** (the libaudio bank-tree
|
||||
re-layout pattern), D39–D42 for the first-render fixes, D31–D38 for background.
|
||||
1. `AGENTS.md` — non-negotiables, critical files, build, verification ritual.
|
||||
2. `docs/PCPortResearch.md` **§F/D48** (the plan under review — read fully),
|
||||
then **D47** (converter contract — still ground truth for the emit pass),
|
||||
**D46** (overlap safety + buffer sizing), **D45** (buffer constants),
|
||||
**D43** (the model-file task), §2.4 (PD-port copy-candidate audit).
|
||||
3. The code paths D48 cites: `port/src/rzdecomp.c`, `port/src/libultra.c`
|
||||
(`piServiceDma`), `port/src/romdata.c` (`romdataInit`,
|
||||
`romdataCartAddrValid`), `src/ramrom.c`, `src/game/ob.c` (`load_resource`,
|
||||
`fileIndexLoadToBank`, `fileSetSize`, `obInit`), `src/game/objecthandler_2.c`
|
||||
(`load_object_fill_header`).
|
||||
|
||||
## Non-negotiables (the ones that bite)
|
||||
- N64 build files (`Makefile`, `tools/`, `rsp/`, `ld/`) — never touch.
|
||||
- **Game logic is unmodified.** Hardware deps go in `port/`. Narrow exception: mechanical,
|
||||
semantics-preserving ABI/layout fixes forced by the 32→64-bit transition (embedded
|
||||
pointers → u32 + cast at use; PC-guarded pool sizing / idiom branches), each documented
|
||||
as a D3x finding. No behavior changes.
|
||||
- **Game logic is unmodified.** Hardware deps go in `port/`. Narrow exception:
|
||||
mechanical, semantics-preserving ABI/layout fixes forced by the 32→64-bit
|
||||
transition (embedded pointers → u32 + cast at use; PC-guarded pool sizing /
|
||||
idiom branches), each documented as a D3x finding. No behavior changes.
|
||||
(Plan B's only game-code change is the PORT-guarded hook block in
|
||||
`load_object_fill_header` — bookkeeping reset + one-shot patch call, no logic.)
|
||||
- Region macros in `CMakeLists.txt` must mirror the N64 Makefile exactly.
|
||||
|
||||
## Current state
|
||||
- **D43 status (latest session, §F/D47):**
|
||||
- DONE + APPLIED (uncommitted): all step-2 mechanical game-code edits
|
||||
(bondtypes.h Vertex u32 union; model.c PROMOTE32; chr.c casts;
|
||||
objecthandler_2.c Textures offset; tex.c Gfx-size/dispatch/byte-check/
|
||||
copy-count fixes; gun.c/front.c/initmenus.c D45 buffer constants).
|
||||
- DONE (this session): **D46 buffer corrections applied** (front.c
|
||||
bufferRemaining 0x1C000→0x25000, initmenus.c logo 0x7C000→0x85000) and the
|
||||
**fast3d seg_addr seg-5 raw-VMA case** (port/fast3d/gfx_pc.cpp).
|
||||
- DONE (this session): every open converter question resolved — compaction
|
||||
state values (delta = R−P; accessors at 0x80090000), G_VTX w0 encoding
|
||||
(GBI1 style; fast3d already correct), G_TRI4 false-positive eliminated
|
||||
(opcode-aware remap rule), vertex formats verified, no embedded blobs,
|
||||
zero-vtx-array rule, reload hazard + poolRemaining-reset decision.
|
||||
- **VALIDATED:** `build-pc/d43_convert.py` implements the full conversion spec
|
||||
and runs it on all 512 ROM files: ALL CLEAN (every pointer remap resolves;
|
||||
max D_PC = 0xB7E0; max ratio D_PC/D_N64 = 1.31).
|
||||
- **REMAINING:** port that validated logic to C as
|
||||
`romdataFixupModelFile()` in port/src/romdata.c (two-pass: walk+layout,
|
||||
then emit PC image staged at the tail of the reserved block and memmove to
|
||||
F), hook it into load_object_fill_header (+ poolRemaining=0 reset before
|
||||
_fileNameLoadToBank, per D47.4), set poolRemaining = D_PC after conversion,
|
||||
build + run + 30 s soak.
|
||||
- (Earlier session, verified live 2026-08-22)
|
||||
- **Re-ran the committed build:** frames 1–4 render (`[NOTE] frame N rendered in … us`,
|
||||
frame 1 ~21 ms), then FATAL `0xc0000005` at PC `0x14007a31a` =
|
||||
`modelPromoteNodeOffsetsToPointers()` model.c:5688 — exactly D43. The black screen you
|
||||
see is the clear color only: display lists execute but no stage geometry has loaded yet.
|
||||
- **Dev loop measured (16-core box):** full clean rebuild ≈ **5 s** wall; launch → crash
|
||||
≈ **3.7 s**. Neither is a bottleneck — *diagnosis* is (gdb launch-only mode + manual
|
||||
addr2line arithmetic per fault). ccache and boot fast-forward are NOT worth it.
|
||||
- **D44 fixed + committed:** `crashStackTraceSym()` in
|
||||
`port/src/crash.c` calls `GetCurrentThreadStackLimits(low, high)` — passing NULL pointer
|
||||
*values* instead of `&low, &high`. The API writes to address 0x0 *inside the SEH filter*
|
||||
→ the process dies before printing the backtrace. That's why `ge007.crash.log` contains
|
||||
only Phase 1 (EXCEPTION/PC/MODULE lines) and no `BACKTRACE:` section, and why every D3x
|
||||
fault so far cost a gdb session. The build already uses `-fno-omit-frame-pointer`
|
||||
(CMakeLists.txt:193), so the EBP-chain walk will work once the call is fixed.
|
||||
- **ROM dump verified:** byte-identical to the No-Intro good dump, header CRCs re-compute
|
||||
clean (`tools_pc/romverify.c`). Fingerprint + provenance in Environment below.
|
||||
- Frames render because of D31–D42 (all documented in §F): boot chain (real-zlib
|
||||
decompression, animation-table ABI+endianness, ANIM_DATA lvalues, music seq-table fixup,
|
||||
music-heap sizing, libaudio bank-tree re-layout), D38 prototype shim (warning-clean
|
||||
build), D39 Globalimagetable rebasing (`GIMG_OFF()` idiom), D40 ModelHitEntry pool
|
||||
sizing, D41 WGL single-thread context currency (`gfx_sdl_release_context()`), D42
|
||||
rsp.c pointer-toggle truncation.
|
||||
- **D43 status (latest review session, §F/D48):** investigation complete;
|
||||
converter spec finalized (D47) and validated 512/512 (`tools_pc/d43_convert.py`,
|
||||
max D_PC = 0xB7E0, ratio ≤ 1.31). **The remaining task was re-planned:** the
|
||||
old plan ("port d43_convert.py 1:1 to C as `romdataFixupModelFile`") is
|
||||
discarded because (a) d43_convert.py does not emit bytes — the emission code
|
||||
is unimplemented in any language, and (b) **Plan B (offline pre-conversion)**
|
||||
reuses the existing ROM load chain and eliminates the C converter entirely:
|
||||
a Python emit pass writes 512 PC-layout RZ sidecars to
|
||||
`data/pcmodels-<region>/`; the port layer serves them by patching
|
||||
`file_resource_table[i].hw_address` + `resource_lookup_data_array[i].rom_size`
|
||||
(full mechanics in D48.2). Plan A (D47.13 C converter) remains the fallback,
|
||||
with a corrected staging guard (D48.4: `D_N64 + D_PC ≤ avail`, not
|
||||
`D_PC ≤ avail`).
|
||||
- DONE + committed (ecefffdb, pushed): all step-2 mechanical game-code edits
|
||||
(bondtypes.h Vertex u32 union; model.c PROMOTE32; chr.c casts;
|
||||
objecthandler_2.c Textures offset; tex.c Gfx-size/dispatch/byte-check/
|
||||
copy-count fixes), D45/D46 buffer constants (gun.c/front.c/initmenus.c),
|
||||
fast3d seg_addr seg-5 raw-VMA case (port/fast3d/gfx_pc.cpp).
|
||||
- DONE this review session: **all 23 `d43_*.py` scripts moved from gitignored
|
||||
`build-pc/` to tracked `tools_pc/` and committed** — the spec is now
|
||||
versioned. Converter re-run from new location: ALL CLEAN.
|
||||
- Verified environment facts (do not rediscover): frames 1–4 render then FATAL
|
||||
at `modelPromoteNodeOffsetsToPointers()` (model.c:5688) = D43; black screen
|
||||
is clear color only (no stage geometry loaded yet). ROM dump byte-identical
|
||||
to the No-Intro good dump (`tools_pc/romverify.exe`). Crash log with real
|
||||
backtraces works (D44). Build clean 237/237; full rebuild ~5 s.
|
||||
|
||||
## ~~Immediate task 1 — D44~~ (DONE — committed)
|
||||
The one-line fix (`GetCurrentThreadStackLimits(&low, &high)` in
|
||||
`crashStackTraceSym()`) is in. Verified: `BACKTRACE:` with #00 at model.c:5688
|
||||
and #01 `load_object_fill_header` appears on stdout + in `ge007.crash.log`.
|
||||
Caveats learned: (a) frames past the true chain can be stale stack data —
|
||||
validate a frame's address is inside `.text` before trusting it; (b) addr2line
|
||||
wants the full absolute hex address directly (`addr2line -e
|
||||
build-pc/ge007.x86_64.exe -f -C 0x14007a31a`).
|
||||
## Task 1 — Plan B review checklist (Session A)
|
||||
|
||||
## Immediate task 2 — D43: model-file loading ABI mismatch (INVESTIGATION DONE — implement)
|
||||
Verify each claim. For each: state CONFIRMED or REFUTED in D49 with the
|
||||
evidence you used (file:line, command output). If a falsification criterion
|
||||
fires, stop and record it — do not paper over it.
|
||||
|
||||
### Verified facts (all checked against ROM ground truth, all 512 files / 1661 nodes — do NOT re-derive)
|
||||
**R1. RZ format is reproducible.** Claim: every model ROM region is
|
||||
`[0x11 0x72][raw deflate]`; `decompressdata()` (port/src/rzdecomp.c) skips
|
||||
exactly those 2 bytes and inflates raw (`inflateInit2(-15)`) with a generous
|
||||
avail_in bound, so a sidecar file in that format works through
|
||||
`load_resource` unmodified. Verify: read rzdecomp.c; confirm the 2-byte prefix
|
||||
is constant across all 512 (one-liner over `scripts/filelist.u.csv` + ROM —
|
||||
`tools_pc/d43_convert.py` already decompresses all 512 with `[2:], -15`, re-run
|
||||
it). **Falsified if:** the prefix varies per file, or decompressdata depends on
|
||||
header state beyond 2 bytes.
|
||||
|
||||
**R2. The DMA path is a plain host memcpy.** Claim: `romCopy` (src/ramrom.c) →
|
||||
`osPiStartDma` → port shim `piServiceDma` (port/src/libultra.c) = `memcpy` from
|
||||
`(void*)srcPA`, gated ONLY by `romdataCartAddrValid()` (port/src/romdata.c:212,
|
||||
checks `addr ≥ CART_BASE && off+size ≤ romSize`). The ROM is VirtualAlloc'd at
|
||||
CART_BASE 0x10000000 sized exactly romSize — so extending the reservation by
|
||||
the sidecar total and extending the validity bound serves sidecars with zero
|
||||
other changes. Verify: read both shims end-to-end; confirm no other gating
|
||||
(alignment, hardcoded 0x10C00000 limits elsewhere, cache effects). Note
|
||||
`romCopyAligned` exists but is NOT used by this load path. **Falsified if:**
|
||||
any additional gate rejects non-ROM host pointers or sizes we can't satisfy.
|
||||
|
||||
**R3. The resource table is the single choke point and is writable.** Claim:
|
||||
`file_resource_table` is a plain (non-const) global defined via include in
|
||||
src/game/ob.c:22; ALL 512 model loads go through `load_object_fill_header`
|
||||
(dst==0 → `_fileNameLoadToBank`, custom-buffer callers in front.c/gun.c/
|
||||
bondview2.c/initmenus.c → `_fileNameLoadToAddr`), both reading `hw_address` via
|
||||
`load_resource`; no game code dereferences C*/G*/P*Z asset symbols directly
|
||||
outside the table. Verify: grep callers of `fileLoad` /
|
||||
`load_object_fill_header`; grep every model symbol (e.g. `CcamguardZ`,
|
||||
`PflagZ`) outside `assets/obseg/file_resource_table.inc.c`,
|
||||
`port/src/romassets_*.s`, and N64-only files (`obseg.h` decls, `ob_seg.s`).
|
||||
**Falsified if:** any compiled game code reads model bytes straight from the
|
||||
cart (it would bypass the sidecar and crash on N64 layout).
|
||||
|
||||
**R4. Patch timing: after obInit, before first model load.** Claim: `obInit()`
|
||||
(src/boss.c:179) runs before the first model load and computes `rom_size` from
|
||||
adjacent-entry hw_address DELTAS (ob.c:122) — so the patch must run after it
|
||||
and set `rom_size` explicitly; nothing between obInit and the first model load
|
||||
consumes model entries' rom_size/hw_address. Verify: trace the init chain
|
||||
(bossInitMainthreadData → … → first stage object load; the D43 crash stack
|
||||
shows the order); grep `rom_size` uses (ob.c:122, 253, 291;
|
||||
`obLoadBGFileBytesAtOffset` is BG-only). **Falsified if:** anything in that
|
||||
window reads model entries' rom_size/hw_address, or obInit can run after a
|
||||
model load.
|
||||
|
||||
**R5. Load-flow fit for pre-converted data.** Claim: with sidecars, a fresh
|
||||
dst==0 load allocates S_bank = whole remaining stage bank; the compressed
|
||||
source placement `round8(compressed)+8 ≤ S_bank` holds (max single-file
|
||||
compressed ≪ ~2.3 MB bank); `load_resource` sets poolRemaining := D_PC
|
||||
automatically; compaction headroom R = S_bank satisfies D46's strict bound with
|
||||
P_conv = D_PC (D46 already proved the buffers against these sizes). ADDITIONALLY:
|
||||
for every dst!=0 caller, the patched compressed size + 8 ≤ the caller's buffer
|
||||
size (gun.c 0x18000/0xBD70/0x17000, front.c wallet 0x17000 / cast screen
|
||||
0x25000, initmenus.c logo 0x85000, bondview2.c weaponbuf `size0`). Verify: have
|
||||
the emit pass (or a quick Python pass) print per-file compressed sizes and
|
||||
cross-check against the D45/D46 constants; re-run `tools_pc/d43_chainbound.py`.
|
||||
**Falsified if:** any buffer/chain is undersized for its file.
|
||||
|
||||
**R6. Reload hazard + the one-line reset.** Claim: `fileSetSize` (ob.c:346-347)
|
||||
leaves poolRemaining = rom_remaining = post-compaction size; without a reset, a
|
||||
same-stage reload allocates S_bank' = P_old which can be < round8(compressed)+8
|
||||
→ `load_resource` hits the `source − ptrdata < 8` branch → poolRemaining = 0 →
|
||||
silent load failure. A PORT-guarded `poolRemaining = 0` reset immediately before
|
||||
`_fileNameLoadToBank` restores fresh-bank semantics every load with identical
|
||||
steady-state usage. Verify: read `fileSetSize`, `fileIndexLoadToBank`,
|
||||
`load_resource`; grep consumers of `get_pc_remaining_buffer_for_index` (should
|
||||
be compaction only) to confirm the reset clobbers nothing. **Falsified if:**
|
||||
the reset breaks a consumer, or the hazard cannot actually occur (then drop the
|
||||
reset and say so in D49).
|
||||
|
||||
**R7. Region handling.** Claim: sidecars are derived from a specific region ROM;
|
||||
the generator takes the region, reads `data/ge007.<region>.z64`, writes
|
||||
`data/pcmodels-<region>/`; romdataInit picks the directory matching the ROM it
|
||||
loaded (same token logic as ROM discovery in `romdataInit`). Verify: read
|
||||
`build-pc.sh` + `romdataInit`'s ROM discovery; confirm name sets are identical
|
||||
across regions is NOT required (the generator maps whatever files exist).
|
||||
**Falsified if:** the build flow can't regenerate sidecars per region, or
|
||||
romdataInit can't tell which directory matches.
|
||||
|
||||
**R8. Emission spec completeness.** Claim: the "Verified facts" section below +
|
||||
D47.13 cover every field of every opcode present in ROM (1,2,4,8,9,10,12,13,
|
||||
15,18,21,22,23,24), and the round-trip validator (§Task 2 step 1) catches any
|
||||
miss. Verify: walk each `ModelRoData_*Record` in src/bondtypes.h and confirm an
|
||||
emission rule exists for every pointer/float/int field; spot-check 2–3 PC
|
||||
record sizes against a fresh C probe (the Python `PC_REC` table was
|
||||
probe-verified once — re-confirm it hasn't drifted). **Falsified if:** any
|
||||
field has no rule, or a PC record size is wrong.
|
||||
|
||||
**Review output:** D49 in §F (8 items adjudicated + evidence), HANDOFF status
|
||||
updated, committed + pushed. If all CONFIRMED → Session B executes §Task 2.
|
||||
|
||||
## Task 2 — Execution checklist (Session B; only after D49 confirms)
|
||||
|
||||
1. **Emit pass in Python** (`tools_pc/d43_emit.py`, importing/reusing
|
||||
`d43_convert.py`'s walk + region map). For each of the 512 files, emit the
|
||||
PC image per D47.13 + Verified facts below: switches (NS×8B LE) →
|
||||
texconfigs (NT×12B; bswap TextureID u32) → nodes+records in DFS preorder
|
||||
(PC layout mirroring bondtypes.h; every promoted pointer field =
|
||||
zero-extended u64 of `0x05|new_off`; Primary/Secondary GDL ptrs remapped but
|
||||
NOT promoted; BaseAddr emitted 0) → vertex arrays immediately after their
|
||||
record (normal: bswap s16 x,y,z,index,s,t, bytes @C–F raw; collision: bswap
|
||||
s16 x,y,z,index + remap LinkedTo u32@8 + bswap CollisionRelatedIndex s16@C /
|
||||
reserved s16@E; nv==0 with non-null ptr → size up to next object offset,
|
||||
D47.10) → PointUsage (2×numVertices s16, after op24 collision verts) →
|
||||
**GDLs LAST, contiguous, visit order, 16B LE slots**: w0' = bswap32(w0);
|
||||
w1' = bswap32(raw) EXCEPT seg-5 of {G_VTX=0x04, G_SETTIMG=0xFD,
|
||||
G_LOADBLOCK=0xF3} → remap low24 via the region map; **no LSB set**; emit up
|
||||
to and including ENDDL. Write each file as `[0x11 0x72] + raw deflate(PC
|
||||
image)` (Python: `zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION,
|
||||
zlib.DEFLATED, -15)`) into `data/pcmodels-<region>/` plus a
|
||||
`manifest.csv` (`name,offset,size`) with offsets assigned contiguously in
|
||||
`file_resource_table` order (parse the quoted names from
|
||||
`assets/obseg/file_resource_table.inc.c`).
|
||||
**Round-trip validator (mandatory before trusting output):** re-parse each
|
||||
emitted image as PC layout — 48B node stride, `PC_REC` record sizes, LE Gfx
|
||||
slots; assert every promoted pointer equals the expected new offset, every
|
||||
seg-5 GDL w1 target resolves inside the image, D_PC matches the pass-1 size.
|
||||
Report per-file compressed sizes (feed R5's cross-check).
|
||||
2. **Port-layer plumbing in `port/src/romdata.c`** (~60 lines):
|
||||
- `romdataInit`: if `data/pcmodels-<region>/manifest.csv` exists for the
|
||||
loaded ROM, reserve the extended region — VirtualAlloc at CART_BASE sized
|
||||
`romSize + sidecarTotal` (aligned), memcpy the ROM as today, then load the
|
||||
sidecar blob(s) at `[CART_BASE + romSize, …)` per manifest offsets. Log
|
||||
count + total bytes. Missing directory → warn and continue (ROM-only
|
||||
mode = current behavior).
|
||||
- Extend `romdataCartAddrValid` to accept up to
|
||||
`CART_BASE + romSize + sidecarTotal`.
|
||||
- New `void romdataPatchModelSidecars(void)` — one-shot (static flag): for
|
||||
each `file_resource_table[i]` whose name is in the manifest, set
|
||||
`hw_address = (u8*)(CART_BASE + romSize + off)` and
|
||||
`resource_lookup_data_array[i].rom_size = size`. `extern` both arrays from
|
||||
ob.c (same pattern as the existing `resource_lookup_data_array` extern).
|
||||
3. **Hook in `load_object_fill_header`** (src/game/objecthandler_2.c, ONE
|
||||
`#if defined(PORT)` block, D48-class bookkeeping — document as D50):
|
||||
(a) call `romdataPatchModelSidecars()` (lazy one-shot; obInit has run by
|
||||
first model load — R4); (b) reset
|
||||
`resource_lookup_data_array[fileGetIndex(name)].poolRemaining = 0`
|
||||
immediately before the `_fileNameLoadToBank`/`_fileNameLoadToAddr` call
|
||||
(R6). No other game-code changes.
|
||||
4. **Build, run, verify + pixel assert (build it NOW — dev-loop item #1):**
|
||||
env-gated framebuffer PPM dump every N frames in `port/video.c`
|
||||
(e.g. `GE_PCDUMP=1` → `build-pc/frame_%05d.ppm`) + `tools_pc/pixcount.py`
|
||||
(counts non-clear-color pixels; diffs two captures). Milestone: no SIGSEGV
|
||||
in `modelPromoteNodeOffsetsToPointers`; pixel count rises above a small
|
||||
threshold and stays up through a 30 s+ soak. If geometry is wrong/missing,
|
||||
the converted sidecars are inspectable OFFLINE (Python) — that's the point
|
||||
of Plan B; don't gdb the game first.
|
||||
5. **Commit checkpoints** (`PC port: phase 2 — …`); record findings D49/D50+
|
||||
in §F; update AGENTS.md phase status + this file. Then move on to the next
|
||||
asset type (Tbg_*_stanZ backgrounds) with the SAME offline pattern — PD's
|
||||
`port/src/preprocess/filebg.c` is the per-field reference (§2.4).
|
||||
|
||||
## Task 3 — Plan A fallback (only if D49 refutes Plan B)
|
||||
|
||||
D47.13 C converter in `port/src/romdata.c`: two-pass walk+layout then emit,
|
||||
staged at `F + R − D_PC`, memmove to F; hook after the filedata assignment and
|
||||
before Switches/Textures/RootNode setup; set `poolRemaining = D_PC` exactly
|
||||
(never rom_remaining); poolRemaining=0 reset before `_fileNameLoadToBank`.
|
||||
**Use the corrected guard (D48.4): fail if `D_N64 + D_PC > avail`** — the
|
||||
staging region overlaps the live N64 image otherwise. The Python emit pass from
|
||||
Task 2 step 1 is still worth writing first: it becomes the byte-level oracle to
|
||||
diff the C output against (convert one file in C, compare with Python output).
|
||||
|
||||
## Verified facts (converter spec — ground truth for the emit pass; do NOT re-derive)
|
||||
|
||||
**Format / layout:**
|
||||
- All 512 C*/G*/P*Z model files (Tbg_*_stanZ backgrounds are a different format/path — later) decompress fine; each file starts directly with the switches array (no in-file header — NS/NT come from the exe `ModelFileHeader` record via `objheader->numSwitches/numtextures`).
|
||||
- File layout: `[switches NS×4B][textures NT×12B][root node @ 4NS+12NT]…`; all embedded pointers are vma `0x05xxxxxx` (mask 0xFFFFFF).
|
||||
- **Tiling invariant verified, 0 failures in all 512 files** (`build-pc/d43_invariants.py`): `[0,D)` is exactly tiled by {switches, textures, nodes(24B), records(RSZ per opcode), vertex arrays(nv×16B), collision arrays(nc×16B), PointUsage (exactly numVertices s16 entries — +2 cases are pure alignment padding), GDLs (each extends to the next GDL start or D; 8-byte aligned, inter-GDL gaps copied as NOOP)}. Records INTERLEAVE with vertex arrays in character bodies — do not assume "all records before bulk".
|
||||
- **Tiling invariant verified, 0 failures in all 512 files** (`tools_pc/d43_invariants.py`): `[0,D)` is exactly tiled by {switches, textures, nodes(24B), records(RSZ per opcode), vertex arrays(nv×16B), collision arrays(nc×16B), PointUsage (exactly numVertices s16 entries — +2 cases are pure alignment padding), GDLs (each extends to the next GDL start or D; 8-byte aligned, inter-GDL gaps copied as NOOP)}. Records INTERLEAVE with vertex arrays in character bodies — do not assume "all records before bulk".
|
||||
- **ModelNode = 24B N64** (u16 op + 2 pad + 5×4B ptrs: Data@4, Parent@8, Next@0xC, Prev@0x10, Child@0x14) / **48B PC**. Older "20B/40B" notes are wrong — verified against raw hex stride 0x18.
|
||||
- **Opcodes present: 1,2,4,8,9,10,12,13,15,18,21,22,23,24** (HEADER, GROUP, DL, LOD, BSP, BOX, GUNFIRE, SHADOW, INTERLINK, SWITCH, GROUPSIMPLE, DLPRIMARY, HEADPH, DLCOLLISION). op22 N64: numVertices s32@0, Vertices@4, Primary@8, BaseAddr@0xC (size 0x10); only 5 files have it.
|
||||
- Record sizes N64/PC: OP01 0x10/0x18, OP02 0x1C/0x28, OP04 0x14/0x28, OP08 0x10/0x18, OP09 0x24/0x30, OP10 0x1C/0x1C, OP12 0x28/0x30, OP13 0x20/0x30, OP15 0x1C/0x1C, OP17 0x20/0x28, OP18 0x08/0x10, OP21 0x14/0x14, OP22 0x10/0x20, OP23 0x02/0x02, OP24 0x20/0x40 (compiler-verified). Field layouts in `src/bondtypes.h` (ModelRoData_*Record); read N64 fields from packed offsets, emit via real PC structs — don't hand-compute PC offsets.
|
||||
- Record sizes N64/PC: OP01 0x10/0x18, OP02 0x1C/0x28, OP04 0x14/0x28, OP08 0x10/0x18, OP09 0x24/0x30, OP10 0x1C/0x1C, OP12 0x28/0x30, OP13 0x20/0x30, OP15 0x1C/0x1C, OP17 0x20/0x28, OP18 0x08/0x10, OP21 0x14/0x14, OP22 0x10/0x20, OP23 0x02/0x02, OP24 0x20/0x40 (compiler-verified). Field layouts in `src/bondtypes.h` (ModelRoData_*Record); read N64 fields from packed offsets, emit via the real PC layout — don't hand-compute PC offsets.
|
||||
- **f32 fields are big-endian in ROM** — bswap all f32 (record origins, BoundingVolumeRadius, Scale, Min/MaxDistance) and every s16 in vertex arrays; also bswap32 the TextureID u32 in the texture table (texLoad reads `*updateword & 0xffff` as the texture number).
|
||||
- **Vertex arrays must stay 16B-stride** (GDL gSPVertex hardcodes 16B/vertex; propobj.c copies with sizeof(Vertex); model.c:4360 allocates numVertices×4 words). bswap every s16 field BE→LE. Main Vertices array is pure data; **CollisionVertices carry LinkedTo@8 = vma of a ModelNode** — remap to the new node offset (PROMOTE promotes them after rebase).
|
||||
- **GDLs:** 8B BE word pairs → 16B LE slots: `w0=bswap32(rom_w0); w1=bswap32(rom_w1)` — **leave w1 raw, NO LSB set** (ROM convention; fast3d's extended seg_addr case handles unmarked segmented addresses — D47.13). GE uses STANDARD libultra GBI numbering (include/PR/gbi.h); opcodes live in the TOP byte of w0. **Opcode-aware remap: only {G_VTX=0x04, G_SETTIMG=0xFD, G_LOADBLOCK=0xF3} carry addresses in w1** (seg-5 → remap low24 via region map). All other opcodes get bswap-only — G_TRI4 (GE extension 0xB1) w1 is 4-bit vertex-index data, NOT an address (D47.6); same for TRI1/TEXTURE/SET*MODE/CLEARGEO/SETGEO/syncs. **G_VTX w0 is GBI1-style** (`04<<24 | dst<<16 | 16·n`, dst=((n−1)<<4)|v0; all 2826 have v0=0, n≤16) — bswap only, no field remap (D47.5). ROM scan: all 2805 seg-5 VTX refs have even offsets → lossless; no nested G_DL in model files.
|
||||
@@ -113,131 +257,40 @@ build-pc/ge007.x86_64.exe -f -C 0x14007a31a`).
|
||||
- PROMOTE contract: emit every promoted pointer as zero-extended u32 (low word of the 8B slot, high word 0); `PROMOTE` (`(u32)var + diff`, vma 0x5000000) then works unmodified. Fields promoted per opcode (model.c switch): all nodes Data/Parent/Next/Prev/Child; OP01 FirstGroup; OP02 ChildGroup; OP04 Vertices; OP08 Affects; OP09 left/rightChild; OP18 Controls; OP24 Vertices/CollisionVertices/PointUsage/CollisionVertices[i].LinkedTo. **NOT promoted:** GDL Primary/Secondary (stay raw 0x05xxxxxx), BaseAddr (game overwrites with fileramaddr).
|
||||
|
||||
**Load path (single hook point):**
|
||||
- ALL model loads go through `load_object_fill_header()` (objecthandler_2.c:97): dst≠0 → `_fileNameLoadToAddr(name,0,dst,size)` (region = caller's bytes — see D45 for the grown per-region sizes); else `_fileNameLoadToBank(name,0,0x100,4)`. **Fresh-load semantics (D47.2-4):** `fileIndexLoadToBank` takes ALL remaining MEMPOOL_STAGE space (`S_bank`) only when poolRemaining==0; then F := alloc(S_bank), R := S_bank, load_resource decompresses D into F and sets P := D_N64. At compaction: **delta = R − P** (positive on fresh loads — the GDL mirror moves FORWARD). `fileSetSize(reallocate=1)` → `mempAddEntryOfSizeToBank` rewinds the bank cursor to the post-compaction size, giving the tail back.
|
||||
- **Reload hazard + PC fix (D47.4):** fileSetSize leaves P=R=post-compaction size, so a same-stage reload would alloc(P_old) which can be < D_N64. PORT-guarded reset of poolRemaining=0 before _fileNameLoadToBank forces fresh-bank semantics every load (staging space = whole remaining bank; steady-state usage identical).
|
||||
- **The fixup must set ONLY poolRemaining (= D_PC exactly — D46/D47), never rom_remaining.** Do NOT use fileSetSize for this — it clobbers rom_remaining too. Set `resource_lookup_data_array[fileGetIndex(name)].poolRemaining` directly (array defined in ob.c:14, not extern'd — declare `extern` in romdata.c; note the accessors actually index **0x80090000 + idx*20** — D47.2).
|
||||
- **Hook location:** inside load_object_fill_header — (a) poolRemaining=0 reset immediately BEFORE `_fileNameLoadToBank`; (b) `u32 romdataFixupModelFile(u8 *blob, const char *name, s16 NS, s16 NT)` (returns D_PC, 0 on failure; reads entry.rom_remaining itself for the staging-space guard) called immediately after the filedata assignment and BEFORE Switches/Textures/RootNode setup + sub_GAME_7F075A90 — PROMOTE walks with PC ModelNode stride (48B), so the file must already be re-laid-out. One `#if defined(PORT)` call site.
|
||||
- ALL model loads go through `load_object_fill_header()` (objecthandler_2.c:97): dst≠0 → `_fileNameLoadToAddr(name,0,dst,size)` (region = caller's bytes — see D45 for the grown per-region sizes); else `_fileNameLoadToBank(name,0,0x100,4)`. **Fresh-load semantics (D47.2-4):** `fileIndexLoadToBank` takes ALL remaining MEMPOOL_STAGE space (`S_bank`) only when poolRemaining==0; then F := alloc(S_bank), R := S_bank, load_resource decompresses D into F and sets P := decompressed size (D_N64 from ROM; **D_PC directly for sidecars — no fixup needed**). At compaction: **delta = R − P** (positive on fresh loads — the GDL mirror moves FORWARD). `fileSetSize(reallocate=1)` → `mempAddEntryOfSizeToBank` rewinds the bank cursor to the post-compaction size, giving the tail back.
|
||||
- **Reload hazard + PC fix (D47.4, needed by BOTH plans):** fileSetSize leaves P=R=post-compaction size, so a same-stage reload would alloc(P_old) which can be too small for the compressed source placement → `load_resource` sets poolRemaining=0 → silent failure. PORT-guarded reset of poolRemaining=0 before the load call forces fresh-bank semantics every load (staging space = whole remaining bank; steady-state usage identical).
|
||||
- **Plan A only (fallback):** the fixup must set ONLY poolRemaining (= D_PC exactly), never rom_remaining; stage at F+R−D_PC with guard `D_N64 + D_PC ≤ avail` (D48.4); hook after filedata assignment, before Switches/Textures/RootNode setup + sub_GAME_7F075A90 (PROMOTE walks PC ModelNode stride — the image must already be re-laid-out).
|
||||
|
||||
**Compaction contract (sub_GAME_7F0762E0, objecthandler_2.c:24-83) — the tightest constraint:**
|
||||
**Compaction contract (sub_GAME_7F0762E0, objecthandler_2.c:24-83) — the tightest constraint (applies to both plans):**
|
||||
- It mirrors the GDL region [first_gdl_off, P_conv) into the caller-region's tail headroom (texCopyGdls to [A+R−P_conv+g, A+R)), then rewrites each GDL in place via texLoadFromGdl (expands texture markers into RDP commands), writing back starting at the original first-GDL offset; final fileSetSize = end of last rewritten GDL (= P_conv + 16·M_actual). **D46: fit and no-clobber are the SAME condition — R ≥ P_conv + 16×Σ(K_t−1) worst case — and all buffers satisfy it (see §F/D46 for the strict bound + the two cast-screen corrections).**
|
||||
- Per-GDL `count` = byte distance between CONSECUTIVELY VISITED gdls → **GDLs must be packed contiguously at the tail of the PC image, in modelIterateDisplayLists visit order.**
|
||||
- `modelIterateDisplayLists` (model.c:6254) **MUTATES the tree while walking**: LOD → `node->Child = LOD.Affects`; SWITCH → `node->Child = Switch.Controls`; BSP → `modelApplyReorderRelationsByArg(node,TRUE)` splices siblings (leftChild group before rightChild). Visit order per DL node: Primary, then Secondary (if non-null and ≠ previous), then DFS Child/Next/Parent.
|
||||
- `build-pc/d43_gdlorder.py` implements this exact simulation to verify visit-order adjacency — **bug fixed + verified**: node collection now pushes child+next plus LOD.Affects/SWITCH.Controls targets; all 512 files pass (total GDLs 2301).
|
||||
- Primary/Secondary are NOT promoted (raw 0x05xxxxxx), so modelNodeReplaceGdl's `Primary == find` comparisons work unchanged; after compaction Primary = replacementgdl (raw value, arbitrary parity — see next item).
|
||||
- `tools_pc/d43_gdlorder.py` implements this exact simulation to verify visit-order adjacency — all 512 files pass (total GDLs 2301).
|
||||
- Primary/Secondary are NOT promoted (raw 0x05xxxxxx), so modelNodeReplaceGdl's `Primary == find` comparisons work unchanged; after compaction Primary = replacementgdl (raw value, arbitrary parity).
|
||||
|
||||
**Render-time GDL resolution:**
|
||||
- The game emits `gSPDisplayList(gdl, rwdata->gdl)` with w1 = raw 0x05xxxxxx; segment 5 (SPSEGMENT_MODEL_COL1) was just set to BaseAddr (= file base) by the preceding gSPSegment. fast3d's `seg_addr` (port/fast3d/gfx_pc.cpp:2282) currently sends 0x05xxxxxx down the "direct pointer" path (invalid). **Port-layer fix:** add a case — top byte 0x05 → `segmentPointers[5] + (w1 & 0xFFFFFF)`.
|
||||
- The game emits `gSPDisplayList(gdl, rwdata->gdl)` with w1 = raw 0x05xxxxxx; segment 5 (SPSEGMENT_MODEL_COL1) is set to BaseAddr (= file base) by the preceding gSPSegment. fast3d's `seg_addr` (port/fast3d/gfx_pc.cpp:2282) resolves raw 0x05xxxxxx via `segmentPointers[5] + (w1 & 0xFFFFFF)` — **DONE, committed.**
|
||||
- Segments 3/4 (MTX table / vertex buffer) are set by the game per-render via gSPSegment ✓. BG segments (13/14/15) belong to the background path — later task.
|
||||
|
||||
### Implementation checklist (in order)
|
||||
0. ~~**D46 buffer corrections (2 lines)**~~ — **DONE this session:** front.c
|
||||
`bufferRemaining = 0x25000`, initmenus.c logo buffer `0x85000` (comments cite D46).
|
||||
1. ~~Fix + run `build-pc/d43_gdlorder.py`~~ — DONE: all 512 files pass, op22 added.
|
||||
2. ~~**Mechanical game-code edits (D32 class)**~~ — **APPLIED (uncommitted):**
|
||||
bondtypes.h Vertex LinkedTo/CollisionRelatedNode → u32 under `#ifdef PORT`;
|
||||
model.c PROMOTE32 macro + use at line ~5735; chr.c:3257-3259 cast + !=0 check;
|
||||
objecthandler_2.c Textures offset via `sizeof(struct ModelNode*) * numSwitches`;
|
||||
tex.c (4 fixes: texCopyGdls count, dispatch `(u8)(in->words.w0>>24)`,
|
||||
0xba byte check `(s8)((w0>>8)&0xff)`, texLoadFromGdl count `srcsize/sizeof(Gfx)`);
|
||||
gun.c size_item_buffer {0x23000,0x23000}, D_80032464 {0xF000,0xF000}, SUIT R
|
||||
0x18000 (pool = size−0x18000), TRIGGER/WATCHLASER R 0x17000; front.c wallet R
|
||||
0x17000 + cast bufferRemaining (now corrected to 0x25000 per item 0);
|
||||
initmenus.c logo buffer (now 0x85000 per item 0). All PORT-guarded with D45/D46
|
||||
comments.
|
||||
- `src/bondtypes.h` Vertex: union members `struct Vertex *LinkedTo; void *CollisionRelatedNode;` → `u32 LinkedTo; u32 CollisionRelatedNode;` (sizeof(Vertex) stays 16 both platforms; game heap <4GiB so lossless on PC).
|
||||
- `src/game/model.c`: add `#define PROMOTE32(var) if (var) var = (u32)((u32)var + diff)` beside PROMOTE (~line 5677); line 5735 `PROMOTE(...LinkedTo)` → `PROMOTE32(...)`.
|
||||
- `src/game/chr.c:3257`: cast the now-u32 field `(ModelNode *)(uintptr_t)…`; line 3259 `!= NULL` → `!= 0`. (These are the only 3 use sites of the union.)
|
||||
- `src/game/objecthandler_2.c:103`: `&((s32*)filedata)[objheader->numSwitches]` → `((u8*)filedata + sizeof(ModelNode*) * objheader->numSwitches)` (N64: 4B stride, identical).
|
||||
- `src/game/tex.c`: texCopyGdls `arg2 = (arg2 >> 3)` → `arg2 / sizeof(Gfx)`; texLoadFromGdl `count = srcsize >> 3` → `srcsize / sizeof(Gfx)` and dispatch `switch (*(u8*)in)` → `switch ((u8)(in->words.w0 >> 24))` (N64-identical; those are the ONLY byte-level accesses in that function — everything else already uses words.w0/w1, and the default case copies whole slots).
|
||||
3. ~~**fast3d seg_addr**~~ — **DONE this session:** port/fast3d/gfx_pc.cpp now
|
||||
resolves raw 0x05xxxxxx (no LSB) via segmentPointers[5] + low24 BEFORE the LSB
|
||||
path; also handles unmarked segmented addresses generally (nibble 24-27 = seg,
|
||||
w1 < 0x10000000). Segment 5 is set per-render by the game's gSPSegment
|
||||
(BaseAddr = live host file base).
|
||||
4. **`romdataFixupModelFile` in `port/src/romdata.c`** (PC-only; bswap helpers +
|
||||
extern `resource_lookup_data_array` from ob.c are already there). **Port the
|
||||
validated logic of `build-pc/d43_convert.py` 1:1 — it IS the spec** (512/512
|
||||
clean; D47.12). Two-pass: (a) DFS walk collecting nodes/records/GDLs + build
|
||||
the region map (old_off, old_size → new_off); (b) emit the PC image into a
|
||||
STAGING area at `blob + avail − D_PC` (avail = entry.rom_remaining; guard:
|
||||
fail with an error if D_PC > avail), then memmove to blob. Emission order:
|
||||
switches(NS×8B LE) → texconfigs(NT×12B; bswap TextureID only — no seg-5 TIDs
|
||||
exist in any model file, D47.9) → nodes+records in DFS preorder (PC layout via
|
||||
struct assignment; every promoted pointer field = zero-extended u64 of
|
||||
0x05|new_off — full per-opcode field map in the facts above + D47.13;
|
||||
Primary/Secondary GDL ptrs remapped but NOT promoted; BaseAddr emitted 0) →
|
||||
vertex arrays immediately after their record (normal: bswap s16 x,y,z,index,s,t,
|
||||
bytes @C-F raw; collision: bswap s16 x,y,z,index + remap LinkedTo u32@8 +
|
||||
bswap CollisionRelatedIndex s16@C/reserved@E; **nv==0 with non-null ptr → size
|
||||
up to next object offset**, D47.10) → PointUsage (2×numVertices s16, after
|
||||
op24 CollisionVerts) → **GDLs LAST, contiguous, visit order, 16B LE slots**:
|
||||
w0'=bswap32(w0); w1' = bswap32(raw) EXCEPT for seg-5 of {G_VTX=0x04,
|
||||
G_SETTIMG=0xFD, G_LOADBLOCK=0xF3} → remap low24 via the region map (sorted
|
||||
regions + binary search); **no LSB set** (ROM convention; D47.6/13). Emit each
|
||||
GDL up to and including its ENDDL (drop trailing junk — fast3d returns at
|
||||
ENDDL, D47.11). Set `resource_lookup_data_array[fileGetIndex(name)].poolRemaining`
|
||||
= D_PC EXACTLY after conversion (load_resource left it at D_N64). NEVER touch
|
||||
rom_remaining. Assert `D_PC + 16×Σ_markers(K_t−1) ≤ avail` (K_BOUND
|
||||
{0:38,1:46,2:36,3:18,4:15}) with a clear error.
|
||||
5. **Hook** into load_object_fill_header (`#if defined(PORT)`): (a) reset
|
||||
`resource_lookup_data_array[fileGetIndex(name)].poolRemaining = 0` immediately
|
||||
BEFORE `_fileNameLoadToBank` (D47.4 — forces fresh-bank allocation so staging
|
||||
space = whole remaining bank; avoids the reload-under-alloc hazard);
|
||||
(b) call romdataFixupModelFile after the filedata assignment and BEFORE
|
||||
Switches/Textures/RootNode setup + sub_GAME_7F075A90 (PROMOTE walks PC
|
||||
ModelNode stride so the image must be re-laid-out first).
|
||||
6. **Build, run, verify:** no SIGSEGV in modelPromoteNodeOffsetsToPointers; scene
|
||||
shows actual geometry (not clear color); 30 s+ soak stable. Sizing is already
|
||||
proven by D46's strict bound + D47.12's max-ratio 1.31 — if anything still
|
||||
overflows, the converter has a bug, not the buffers.
|
||||
|
||||
### Helper scripts (build-pc/, written across D43 sessions)
|
||||
- `d43_decode.py` — decompress + decode one model's node tree + GDL command stream.
|
||||
- `d43_sizes.py` / `d43_sizes2.py` — decompress all 512, report N64/PC sizes +
|
||||
per-file worst-case P_final (B_pc + 2E + 16×Σ(K_t−1)).
|
||||
- `d43_walk.py` — walks all trees using exe NS/NT; opcode histogram. Note: header
|
||||
names lack the C/G/P prefix and trailing Z (file `CcamguardZ` → header `camguard`).
|
||||
## Helper scripts (tools_pc/, tracked; run from repo root)
|
||||
- `d43_convert.py` — **reference layout + validator (D47):** DFS walk with LOD/SWITCH rewiring, region map, per-opcode record/vertex/GDL analysis, remap checks; runs on all 512 ROM files (ALL CLEAN). Lays the foundation for the emit pass (Task 2 step 1) — import/reuse its walk + region map.
|
||||
- `d43_invariants.py` — full ownership tiling check (0 failures / 512).
|
||||
- `d43_gdlorder.py` — exact visit-order simulation; **bug fixed, all 512 pass**.
|
||||
- `d43_gdldump.py` / `d43_gdlhist.py` / `d43_gdlseq.py` / `d43_seg5.py` — GDL
|
||||
command dumps/histograms/sequences; seg-5 VTX reference analysis (all 2805 point
|
||||
into op4/op22 vertex arrays → must remap).
|
||||
- `d43_cover.py` — verifies every seg-5 VTX + SETTIMG target falls inside a modeled
|
||||
object (2804/2805 covered; the one exception is PexplosionbitZ's orphan array at
|
||||
offset 0x98 — handled by the converter's implicit-object rule). SETTIMG targets in
|
||||
logo files are vestigial (in gaps, never dereferenced).
|
||||
- `d43_lutscan.py` — scans all model texture tables → image headers (via
|
||||
imagelist.u.csv, order = assets/images.def): proves NO LUT textures (formats 9-12)
|
||||
and maxlod distribution {0:1063, 6:7, 7:1} across all 512 files (D46).
|
||||
- `d43_convert.py` — **THE reference converter + validator (D47):** implements
|
||||
the full N64→PC conversion spec (DFS walk with LOD/SWITCH rewiring, region map,
|
||||
per-opcode record/vertex/GDL conversion, remap checks) and runs it on all 512
|
||||
ROM files — ALL CLEAN. This IS the spec to port to C in romdata.c.
|
||||
- `d43_chainbound.py` — computes the D46 strict bound B_pc + 2×(R_share_N64 − B_n64)
|
||||
for every buffer/chain and compares against the PC region sizes.
|
||||
- Gotcha: MODELFILEHEADER's name is field a[0] of the macro args (NOT the filename —
|
||||
all header files are literally named `modelFileHeader.inc.c` in per-model dirs).
|
||||
- `d43_gdlorder.py` — exact visit-order simulation; all 512 pass.
|
||||
- `d43_sizes.py` / `d43_sizes2.py` — decompress all 512, report N64/PC sizes + per-file worst-case P_final.
|
||||
- `d43_walk.py` / `d43_fullwalk.py` / `d43_tree_dump.py` — tree walks, opcode histogram. (Header names lack the C/G/P prefix and trailing Z: file `CcamguardZ` → header `camguard`; all header files are literally named `modelFileHeader.inc.c` in per-model dirs.)
|
||||
- `d43_gdldump.py` / `d43_gdlhist.py` / `d43_gdlseq.py` — GDL command dumps/histograms/sequences.
|
||||
- `d43_seg5.py` / `d43_seg5ops.py` / `d43_seg5vtx.py` / `d43_seg5vtx2.py` — seg-5 VTX reference analysis (all 2805 point into op4/op22 vertex arrays → must remap).
|
||||
- `d43_cover.py` — verifies every seg-5 VTX + SETTIMG target falls inside a modeled object (2804/2805; the one exception is PexplosionbitZ's orphan array at 0x98 — handled by the zero-vtx rule).
|
||||
- `d43_lutscan.py` — proves NO LUT textures (formats 9-12) and maxlod distribution across all 512 files (D46).
|
||||
- `d43_chainbound.py` — computes D46's strict bound for every buffer/chain vs the PC region sizes.
|
||||
- `d43_decode.py` / `d43_layout*.py` / `d43_pointusage.py` / `d43_vtxfmt.py` — earlier investigation (format pinning).
|
||||
- Gotcha: MODELFILEHEADER's name is field a[0] of the macro args (NOT the filename).
|
||||
|
||||
### Follow-on
|
||||
Expect later faults in other asset types (BG/stan/propobj tables) — same D32/D37 procedure; PD's `preprocess/` module already solves most of them (`filebg.c`, `filetiles.c`, `filepads.c`, `filesetup.c`, …) — see the PD reference section below.
|
||||
|
||||
## Dev-loop speedups (build these when they pay off, not before)
|
||||
Priority order from the last session's measurements:
|
||||
1. **D44** (above) — automatic backtraces; biggest win per diagnosis.
|
||||
2. **`tools_pc/sym.sh`** — tiny wrapper: takes a PC or rel offset, adds image base
|
||||
`0x140000000`, runs `addr2line -e build-pc/ge007.x86_64.exe -f -C`. Kills the manual
|
||||
arithmetic; pairs with the crash log (one command per crash). Re-verify the image base
|
||||
with `info address` after any link-layout change.
|
||||
3. **`tools_pc/romdump.py`** — Python ROM/asset inspector: load `data/ge007.ntsc-final.z64`,
|
||||
parse ModelFileHeader/ModelNode trees using the bondtypes.h layout, print opcode
|
||||
sequences / node links / Data records for a given cart address. Accelerates D43 and
|
||||
every follow-on asset type; validates re-layout output against ROM ground truth without
|
||||
running the game. (Python here: no f-strings with nested quotes — use `%` formatting.)
|
||||
4. **Frame capture + pixel assert** — env-gated framebuffer PPM dump every N frames in
|
||||
`port/`, plus a script that counts non-clear-color pixels / diffs two captures. Turns
|
||||
"the scene contains actual model geometry" into an assert, makes the 30 s soak
|
||||
scriptable, and enables screenshot diffs against an emulator (Mupen64Plus with the
|
||||
verified No-Intro dump as ground truth).
|
||||
## Dev-loop speedups (priority order)
|
||||
1. **Frame capture + pixel assert** — NOW (Task 2 step 4): env-gated PPM dump every N frames in `port/video.c` + `tools_pc/pixcount.py` (non-clear-color pixel count / diff two captures). Turns "the scene contains actual model geometry" into an assert, makes the 30 s soak scriptable, and enables screenshot diffs against Mupen64Plus with the verified No-Intro dump as ground truth.
|
||||
2. **`tools_pc/sym.sh`** — tiny wrapper: takes a PC or rel offset, adds image base `0x140000000`, runs `addr2line -e build-pc/ge007.x86_64.exe -f -C`. One command per crash; re-verify the image base with `info address` after any link-layout change.
|
||||
3. **`tools_pc/romdump.py`** — Python ROM/asset inspector: parse ModelFileHeader/ModelNode trees using the bondtypes.h layout, print opcode sequences / node links / Data records for a given cart address; also re-parse SIDECAR files as PC layout (post-Plan B this is the first stop for "geometry looks wrong").
|
||||
4. D44 (automatic backtraces) — DONE, committed.
|
||||
|
||||
## PD port as standing reference (always available)
|
||||
`PD_PORT_CHECKOUT` is the Perfect Dark PC port — same Rare engine
|
||||
@@ -245,8 +298,9 @@ family, and the **standing reference/guidance source** for this project (noted i
|
||||
AGENTS.md). Consult it whenever a remaining work item has a PD analogue:
|
||||
- **Phase 2 asset conversion:** `port/src/preprocess/` (~4,100 lines) — per-ROM-segment
|
||||
N64→PC layout converters hooked from their `romdata.c`. `filemodel.c` = D43 near-analogue
|
||||
(same vma 0x5000000); `filebg.c` (expected follow-on), `segaudio.c` (cross-check for our
|
||||
D37 bank-tree fixup), `filelang.c`, `filetiles.c`, `filepads.c`, `filesetup.c`, `gbi.c`,
|
||||
(same vma 0x5000000; PD converts at LOAD time — we convert OFFLINE for speed, same field
|
||||
rules apply); `filebg.c` (expected follow-on), `segaudio.c` (cross-check for our D37
|
||||
bank-tree fixup), `filelang.c`, `filetiles.c`, `filepads.c`, `filesetup.c`, `gbi.c`,
|
||||
`segfonts.c`.
|
||||
- **Phase 3/4 (copy-and-adapt; our stubs were scaffolded for this):** `port/src/mixer.c`
|
||||
(722 lines vs our 31 — libaudio→SDL mixer), `input.c` (1,551 vs 48 — SDL_GameController
|
||||
@@ -265,7 +319,10 @@ at the use sites; verify the load-time rebase yields valid V1 addresses (< 0x800
|
||||
identity) and any fixup loop's pointer stride on x86-64**; for packed struct trees use the
|
||||
D37 re-layout pattern; for address-arithmetic idioms over exe symbols (D39/D42 class) guard
|
||||
a PC branch that reproduces the N64 32-bit value exactly. Rebuild; confirm boot advances.
|
||||
Log each as D3x in §F and note it in AGENTS.md phase status.
|
||||
Log each as D3x in §F and note it in AGENTS.md phase status. **For asset FORMAT work,
|
||||
prefer the offline pattern (D48): pin the format with Python over the ROM ground truth,
|
||||
emit + validate offline, serve through the existing load path — runtime C conversion only
|
||||
as fallback.**
|
||||
|
||||
## Environment (do not rediscover these)
|
||||
- MSYS2/MinGW tools are in `/c/msys64/mingw64/bin/` — **not** on PATH. Prefix every shell:
|
||||
@@ -286,7 +343,8 @@ Log each as D3x in §F and note it in AGENTS.md phase status.
|
||||
- gdb: **launch** mode only (`gdb -batch -ex "handle SIGSEGV stop" -ex run …`). Symbolicate
|
||||
offline with `addr2line -e build-pc/ge007.x86_64.exe -f -C <0x140000000+rel>`. Image base
|
||||
is `0x140000000` (re-verify with `info address` after a rebuild). Debug scripts live in
|
||||
`build-pc/` (`d37_tree.py`, `d39_*.gdb`, `d41_probe.gdb`, …).
|
||||
`build-pc/` (`d37_tree.py`, `d39_*.gdb`, `d41_probe.gdb`, …); the D43 investigation suite
|
||||
is in tracked `tools_pc/`.
|
||||
- **Capstone gotcha (D47.1):** the ROM is BIG-ENDIAN MIPS — always disassemble with
|
||||
`CS_MODE_MIPS32 | CS_MODE_BIG_ENDIAN`. Default (LE) mode produces garbage that looks
|
||||
plausible; an earlier session's disassembly was all LE and had to be redone.
|
||||
@@ -299,12 +357,12 @@ Log each as D3x in §F and note it in AGENTS.md phase status.
|
||||
- Many init functions use a fake RBP — compute stack offsets from the **entry RSP**, not RBP.
|
||||
- Python on this box: no f-strings with nested quotes (< 3.12) — use `%` formatting.
|
||||
|
||||
## Definition of done (this milestone)
|
||||
Checkpoint A: D44 committed — crash log shows a real backtrace for the D43 fault.
|
||||
Then the milestone proper: stage objects load without faulting and the rendered scene
|
||||
contains actual model geometry (not just the clear color / background), stable under a
|
||||
30 s+ soak. After that, Phase 2 makes it look right (fast3d CC/RM correctness vs
|
||||
`gmain.s` — GE-specific, no PD analogue) and Phase 3 adds sound + input, largely
|
||||
copy-and-adapt from the PD port (see "PD port as standing reference" above). Commit +
|
||||
push to `origin/master` at each working checkpoint using the message style
|
||||
`PC port: <phase> — <what>`.
|
||||
## Definition of done
|
||||
- **Session A:** D49 committed — all 8 checklist items adjudicated with evidence;
|
||||
HANDOFF status updated; if any item refuted, the fallback/adjustment is spelled out.
|
||||
- **Session B (the milestone proper):** stage objects load without faulting and the
|
||||
rendered scene contains actual model geometry per the pixel assert (not just the clear
|
||||
color / background), stable under a 30 s+ soak. Committed + pushed to `origin/master`
|
||||
(`PC port: phase 2 — <what>`). After that, Phase 2 makes it look right (fast3d CC/RM
|
||||
correctness vs `gmain.s` — GE-specific, no PD analogue) and Phase 3 adds sound + input,
|
||||
largely copy-and-adapt from the PD port.
|
||||
@@ -1494,6 +1494,81 @@ through the real structs (compiler packs); don't hand-compute.
|
||||
15. **PointUsage** = 2×numVertices s16 entries, indexed by MAIN-vertex index,
|
||||
chain terminated by −1 (chr.c:3309-3331).
|
||||
|
||||
**D48 (REVIEW — D43 re-plan: offline pre-conversion "Plan B"; process fix)**
|
||||
Independent review session that audited the D47 handoff against code + ROM
|
||||
before implementation. Verdict: the remaining task was mis-scoped as a 1:1 C
|
||||
port of a script that does not emit bytes; a cheaper, lower-risk path exists
|
||||
that reuses the existing ROM load chain. Plan B is now the default; Plan A
|
||||
(D47.13 C converter) is the fallback.
|
||||
|
||||
1. **`d43_convert.py` does not emit bytes.** It computes the layout/region map
|
||||
and validates that every pointer remap resolves (512/512 clean, re-run and
|
||||
confirmed this session), but there is no emission pass in ANY language. The
|
||||
byte-level contract (PC struct assignment, per-field bswaps, GDL slot writing)
|
||||
is unimplemented and unvalidated. "Port d43_convert.py 1:1 to C" would have the
|
||||
next session write ~250-300 lines of new emission code in C and debug it at
|
||||
runtime (crash → backtrace cycles). The emission must be written once either
|
||||
way — do it in Python where iteration is seconds, validate offline, then ship
|
||||
the data.
|
||||
2. **Plan B: offline pre-conversion through the existing ROM load path.**
|
||||
Generate 512 PC-layout RZ sidecar files with a Python emit pass; serve them by
|
||||
patching `file_resource_table[i].hw_address` +
|
||||
`resource_lookup_data_array[i].rom_size` from the port layer. Verified
|
||||
mechanical facts (all re-checkable in <1 h):
|
||||
- **RZ format is trivially reproducible**: 2-byte header (`0x11 0x72`) + raw
|
||||
deflate; `decompressdata()` (port/src/rzdecomp.c) skips the header and inflates
|
||||
with a generous avail_in bound. A sidecar `[0x11 0x72][raw-deflate(PC image)]`
|
||||
works through `load_resource` UNMODIFIED — and it sets `poolRemaining = D_PC`
|
||||
automatically (the decompressed size), so no fixup call is needed at all.
|
||||
- **`romCopy` on PC is a host memcpy**: src/ramrom.c → `osPiStartDma` → port
|
||||
shim `piServiceDma` (port/src/libultra.c), gated only by
|
||||
`romdataCartAddrValid()` (port/src/romdata.c:212). The ROM is VirtualAlloc'd at
|
||||
CART_BASE 0x10000000 sized romSize — extend the reservation by the sidecar
|
||||
total, place sidecars at [CART_BASE+romSize, …), extend the validity check.
|
||||
- **`file_resource_table` is a plain writable global** (included in
|
||||
src/game/ob.c:22). ALL 512 model loads funnel through
|
||||
`load_object_fill_header` (dst==0 → `_fileNameLoadToBank`; custom-buffer
|
||||
callers in front.c/gun.c/bondview2.c/initmenus.c → `_fileNameLoadToAddr`) —
|
||||
both read `hw_address` via `load_resource`. No game code dereferences C*/G*/P*Z
|
||||
symbols directly outside the table (verified by grep; symbols exist only in
|
||||
the table + romassets_<r>.s markers).
|
||||
- **The patch must run AFTER `obInit()`** (called at src/boss.c:179):
|
||||
obInit computes `rom_size` from adjacent-entry hw_address DELTAS (ob.c:122), so
|
||||
a pre-obInit patch would corrupt rom_size. Lazy one-shot at the PORT hook site
|
||||
in load_object_fill_header is simplest — by first model load, obInit has
|
||||
definitely run.
|
||||
- **Footprint**: 512 files = 1,277,088 B compressed total (1.2 MB),
|
||||
3,289,344 B decompressed N64 total; PC decompressed ≤ 1.31× per file (D47.12).
|
||||
- **The indy path is NOT used** (`resource_load_from_indy`, ob.c:56): it is
|
||||
the N64 host-protocol loader gated by `indy_ready` (src/game/indy_comms.c,
|
||||
dormant on PC), and its pPayload placement underflows on reload when
|
||||
poolRemaining == pc_size exactly. Table patching reuses the proven ROM path.
|
||||
- **What Plan B eliminates**: the C converter (~300 lines), two-pass staging
|
||||
+ memmove, staging-space guard, and the "emission bug only visible at runtime"
|
||||
risk class. What it keeps (all already committed): ABI edits, fast3d seg-5
|
||||
case, D46 buffer sizing — plus the one-line poolRemaining=0 reset (item 3).
|
||||
- **Sidecars are region-specific** (derived from the region ROM); generator
|
||||
must take the region and write `data/pcmodels-<region>/`.
|
||||
3. **The poolRemaining=0 reset is STILL needed under Plan B.** fileSetSize
|
||||
leaves P=R=post-compaction size (ob.c:346-347); a same-stage reload would then
|
||||
alloc S_bank' = P_old, which can be < round8(compressed)+8 → `load_resource`
|
||||
hits the `source − ptrdata < 8` branch → poolRemaining=0 → silent load
|
||||
failure. The D47.4 reset before `_fileNameLoadToBank` covers both plans.
|
||||
4. **Plan A flaw (if runtime conversion is kept):** D47.13's staging guard
|
||||
"fail if D_PC > avail" is too weak — the staging region [F+R−D_PC, F+R)
|
||||
overlaps the live N64 image [F, F+D_N64) whenever S_bank < D_N64 + D_PC, and
|
||||
emission would then read corrupted bytes. Correct guard: `D_N64 + D_PC ≤
|
||||
avail`. Low probability in practice (files load at stage start; ~2.3 MB bank
|
||||
vs ~82 KB worst case), but the stated guard silently misses it.
|
||||
5. **Process fix:** the 23 d43_*.py investigation scripts (incl. the reference
|
||||
converter) lived in gitignored `build-pc/` — moved to tracked `tools_pc/`
|
||||
and committed, so the spec is versioned and reviewable. d43_cover.py's one
|
||||
internal path reference updated; converter re-run from new location: ALL CLEAN.
|
||||
6. **Review checklist for the next session** (confirm each item against code +
|
||||
ROM before executing Plan B; record results as D49): see docs/HANDOFF.md
|
||||
§Task 1 — eight claims, each with a falsification criterion. If any fires,
|
||||
fallback to Plan A with the corrected guard from item 4.
|
||||
|
||||
### G. Phase 2 status (current)
|
||||
|
||||
Done: PD fast3d integrated (`port/fast3d/`, replaces the deleted gfxstub.c);
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Cast-chain strict overlap/fit bound (D46 follow-up).
|
||||
|
||||
Formula (derived from N64 ground truth: the N64 game works, so per file
|
||||
8*M_a <= R_share_N64 - B_n64 - E; PC slots are 2x wide with the same M_a):
|
||||
|
||||
sum(P_final_actual) <= sum(B_pc) + 2*(R_chain_N64 - sum(B_n64))
|
||||
|
||||
where R_chain_N64 = 0x18160 (N64 cast-screen bufferRemaining).
|
||||
Compares against the PC bufferRemaining = 0x1C000 currently in front.c.
|
||||
"""
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
|
||||
RSZ_N64 = {1: 0x10, 2: 0x1C, 4: 0x14, 8: 0x10, 9: 0x24, 10: 0x1C,
|
||||
12: 0x28, 13: 0x20, 15: 0x1C, 17: 0x20, 18: 0x8, 21: 0x14,
|
||||
22: 0x10, 23: 2, 24: 0x20}
|
||||
RSZ_PC = {1: 0x18, 2: 0x28, 4: 0x28, 8: 0x18, 9: 0x30, 10: 0x1C,
|
||||
12: 0x30, 13: 0x30, 15: 0x1C, 17: 0x28, 18: 0x10, 21: 0x14,
|
||||
22: 0x20, 23: 2, 24: 0x40}
|
||||
|
||||
nsnt = {}
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
for line in open(os.path.join(root, fn)):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m:
|
||||
continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9:
|
||||
continue
|
||||
try:
|
||||
nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def be32(b, o):
|
||||
return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
|
||||
def analyze(base_want):
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]:
|
||||
continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"):
|
||||
base = base[:-4]
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt:
|
||||
key = cand
|
||||
break
|
||||
if key != base_want:
|
||||
continue
|
||||
addr, size = int(r[0]), int(r[1])
|
||||
out = zlib.decompress(rom[addr:addr + size][2:], -15)
|
||||
D = len(out)
|
||||
NS, NT = nsnt[key]
|
||||
R0n = 4 * NS + 12 * NT
|
||||
stack = [R0n]
|
||||
seen = set()
|
||||
nodes = []
|
||||
gdls = []
|
||||
while stack:
|
||||
o = stack.pop()
|
||||
if o in seen or o >= D - 23:
|
||||
break
|
||||
seen.add(o)
|
||||
op = struct.unpack_from(">H", out, o)[0] & 0xff
|
||||
data = be32(out, o + 4)
|
||||
nodes.append((o, op, data))
|
||||
if op in (4, 24):
|
||||
p, s = be32(out, data), be32(out, data + 4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
elif op == 22:
|
||||
p = be32(out, data + 8)
|
||||
if p: gdls.append(p)
|
||||
for q in (be32(out, o + 0x14), be32(out, o + 0xC)):
|
||||
if q:
|
||||
stack.append(q)
|
||||
gsorted = sorted(set(gdls))
|
||||
g1 = gsorted[0] if gsorted else D
|
||||
bulk = (g1 - R0n) - len(nodes) * 24 - sum(RSZ_N64[op] for _, op, _ in nodes)
|
||||
B_n64 = 4 * NS + 12 * NT + len(nodes) * 24 + sum(RSZ_N64[op] for _, op, _ in nodes) + bulk
|
||||
B_pc = 8 * NS + 12 * NT + len(nodes) * 48 + sum(RSZ_PC[op] for _, op, _ in nodes) + bulk
|
||||
E = D - g1
|
||||
return dict(base=base_want, D=D, B_n64=B_n64, B_pc=B_pc, E=E)
|
||||
|
||||
# cast-screen weapon sets (front.c ~867-891)
|
||||
rifles = ["kalash", "m16", "fnp90", "autoshot", "grenadelaunch", "sniperrifle"]
|
||||
pistols = ["wppk", "wppksil", "skorpion", "uzi", "tt33", "ruger", "laser", "golden"]
|
||||
heads = []
|
||||
# find all head files (Chead*Z)
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]:
|
||||
continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"):
|
||||
base = base[:-4]
|
||||
if base.startswith("Chead") and base.endswith("Z"):
|
||||
heads.append(base)
|
||||
|
||||
def worst_in(names):
|
||||
best = None
|
||||
for n in names:
|
||||
a = analyze(n)
|
||||
if not a:
|
||||
print(" !! not found:", n)
|
||||
continue
|
||||
# P_final(worst K) proxy: B_pc + 2E (marker slack handled separately)
|
||||
pf = a["B_pc"] + 2 * a["E"]
|
||||
if best is None or pf > best[0]:
|
||||
best = (pf, a)
|
||||
return best
|
||||
|
||||
print("== worst head ==")
|
||||
hp, ha = worst_in([h[1:-1] for h in heads])
|
||||
print("%s P_conv=%X B_pc=%X B_n64=%X" % (ha["base"], hp, ha["B_pc"], ha["B_n64"]))
|
||||
|
||||
print("== worst rifle ==")
|
||||
rp, ra = worst_in(rifles)
|
||||
print("%s P_conv=%X B_pc=%X B_n64=%X" % (ra["base"], rp, ra["B_pc"], ra["B_n64"]))
|
||||
|
||||
print("== worst pistol ==")
|
||||
pp, pa_ = worst_in(pistols)
|
||||
print("%s P_conv=%X B_pc=%X B_n64=%X" % (pa_["base"], pp, pa_["B_pc"], pa_["B_n64"]))
|
||||
|
||||
# worst body: scan C* bodies (exclude heads/suits/props)
|
||||
bodies = []
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]:
|
||||
continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"):
|
||||
base = base[:-4]
|
||||
if base.startswith("C") and base.endswith("Z") and "head" not in base.lower() \
|
||||
and "suit" not in base.lower():
|
||||
bodies.append(base[1:-1])
|
||||
bp, ba = worst_in(bodies)
|
||||
print("== worst body ==")
|
||||
print("%s P_conv=%X B_pc=%X B_n64=%X" % (ba["base"], bp, ba["B_pc"], ba["B_n64"]))
|
||||
|
||||
for label, weapon in (("rifle", ra), ("pistol", pa_)):
|
||||
sBpc = ba["B_pc"] + ha["B_pc"] + weapon["B_pc"]
|
||||
sBn64 = ba["B_n64"] + ha["B_n64"] + weapon["B_n64"]
|
||||
bound = sBpc + 2 * (0x18160 - sBn64)
|
||||
print("chain(%s): sumB_pc=%X sumB_n64=%X strict-bound=%X vs PC region 0x1C000 -> %s"
|
||||
% (label, sBpc, sBn64, bound, "FITS" if bound <= 0x1C000 else "SHORT by %X" % (bound - 0x1C000)))
|
||||
|
||||
# ---- single-file strict bounds: P_final_actual <= B_pc + 2*(R_share_N64 - B_n64)
|
||||
print()
|
||||
print("== single-file strict bounds ==")
|
||||
def check(name, r_n64, region_pc):
|
||||
a = analyze(name)
|
||||
if not a:
|
||||
print("%-16s NOT FOUND" % name); return
|
||||
bound = a["B_pc"] + 2 * (r_n64 - a["B_n64"])
|
||||
ok = "OK " if bound <= region_pc else "SHORT by %X" % (bound - region_pc)
|
||||
print("%-16s B_pc=%6X B_n64=%6X bound=%7X vs region %X -> %s"
|
||||
% (name, a["B_pc"], a["B_n64"], bound, region_pc, ok))
|
||||
|
||||
check("suit_lf_hand", 0xBD70, 0x18000)
|
||||
check("trigger", 0xAFD0, 0x17000)
|
||||
check("watchlaser", 0xAFD0, 0x17000)
|
||||
check("walletbond", 0xA000, 0x17000)
|
||||
|
||||
# weapons: worst over all G* files with R_share_N64 = 0x7530
|
||||
weapons = []
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]:
|
||||
continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"):
|
||||
base = base[:-4]
|
||||
if base.startswith("G") and base.endswith("Z"):
|
||||
weapons.append(base[1:-1])
|
||||
worst_w = None
|
||||
for w in weapons:
|
||||
a = analyze(w)
|
||||
if not a:
|
||||
continue
|
||||
bound = a["B_pc"] + 2 * (0x7530 - a["B_n64"])
|
||||
if worst_w is None or bound > worst_w[0]:
|
||||
worst_w = (bound, w, a)
|
||||
print("worst weapon: %-14s bound=%X vs 0xF000 -> %s"
|
||||
% (worst_w[1], worst_w[0], "OK" if worst_w[0] <= 0xF000 else "SHORT by %X" % (worst_w[0]-0xF000)))
|
||||
|
||||
# bondview chain: body+head+weapon, R_chain_N64 = 0x14820 (size_item_buffer)
|
||||
sBpc = ba["B_pc"] + ha["B_pc"] + ra["B_pc"]
|
||||
sBn64 = ba["B_n64"] + ha["B_n64"] + ra["B_n64"]
|
||||
bound = sBpc + 2 * (0x14820 - sBn64)
|
||||
print("bondview chain: bound=%X vs 0x23000 -> %s"
|
||||
% (bound, "OK" if bound <= 0x23000 else "SHORT by %X" % (bound - 0x23000)))
|
||||
@@ -0,0 +1,260 @@
|
||||
#!/usr/bin/env python3
|
||||
"""D43 reference model-file converter (N64 BE -> PC LE). Validates on all 512 files:
|
||||
- layout: [switches NS*8][texconfigs NT*12][DFS nodes(48B)+records(PC sz)][vtx arrays][blobs][GDLs 16B slots]
|
||||
- every pointer remap resolves via the unified region map
|
||||
- GDL spans pack tight (16B multiples), D_PC - G_PC == 16*total_slots
|
||||
- reports max D_PC vs N64 size (staging headroom)
|
||||
This is the spec that port/src/romdata.c romdataFixupModelFile implements."""
|
||||
import csv, struct, zlib, os, re, sys
|
||||
from collections import Counter
|
||||
|
||||
ROM = open('data/ge007.ntsc-final.z64','rb').read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
MASK = 0xFFFFFF
|
||||
def be16(b,o): return struct.unpack_from(">h",b,o)[0]
|
||||
def bu16(b,o): return struct.unpack_from(">H",b,o)[0]
|
||||
def be32o(b,o): return struct.unpack_from(">I",b,o)[0] & MASK # file offset (low 24)
|
||||
def be32r(b,o): return struct.unpack_from(">I",b,o)[0] # raw u32 BE
|
||||
|
||||
# N64 record sizes (from bondtypes.h comments, verified vs ROM via full walk)
|
||||
N64_REC = {1:0x1C, 2:0x1C, 3:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28,
|
||||
13:0x20, 15:0x1C, 18:0x08, 21:0x14, 22:0x10, 23:0x02, 24:0x20}
|
||||
# PC record sizes (probe-verified)
|
||||
PC_REC = {1:24, 2:40, 3:40, 4:40, 8:24, 9:48, 10:28, 12:48,
|
||||
13:48, 15:28, 18:16, 21:20, 22:32, 23:2, 24:64}
|
||||
PC_NODE = 48
|
||||
|
||||
def walk(src, D, NS, NT):
|
||||
"""DFS with LOD/SWITCH rewiring. Returns (nodes, gdls) where
|
||||
nodes = [(n64_off, op)] in visit order, gdls = [gdl_off] in visit order ([Primary, Secondary])."""
|
||||
R0 = 4*NS + 12*NT
|
||||
seen = set(); stack = [R0]; nodes = []; gdls = []
|
||||
while stack:
|
||||
o = stack.pop()
|
||||
if o in seen or o >= D: continue
|
||||
seen.add(o)
|
||||
op = bu16(src, o) & 0xff
|
||||
data = be32o(src, o+4)
|
||||
nodes.append((o, op))
|
||||
child = be32o(src, o+0x14); nxt = be32o(src, o+0xC)
|
||||
if op == 8: # LOD -> Affects (data+8)
|
||||
a = be32o(src, data+8)
|
||||
if a: stack.append(a)
|
||||
elif op == 18: # SWITCH -> Controls (data+0)
|
||||
c = be32o(src, data)
|
||||
if c: stack.append(c)
|
||||
else:
|
||||
if child: stack.append(child)
|
||||
if nxt: stack.append(nxt)
|
||||
# GDLs in [Primary, Secondary] order at node entry
|
||||
if op == 4:
|
||||
p = be32o(src, data); s = be32o(src, data+4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
elif op == 24:
|
||||
p = be32o(src, data); s = be32o(src, data+4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
elif op == 22:
|
||||
p = be32o(src, data+8)
|
||||
if p: gdls.append(p)
|
||||
return nodes, gdls
|
||||
|
||||
def gdl_end(src, D, g):
|
||||
o = g
|
||||
while o + 8 <= D:
|
||||
if (be32r(src, o) >> 24) == 0xB8: return o + 8
|
||||
o += 8
|
||||
return D
|
||||
|
||||
errors = []
|
||||
stats = Counter()
|
||||
max_ratio = 0; max_file = ""
|
||||
blob_info = []
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key = cand; break
|
||||
if not key:
|
||||
errors.append(f"{base}: no header data"); continue
|
||||
NS, NT = nsnt[key]
|
||||
try: addr, size = int(r[0]), int(r[1])
|
||||
except ValueError: continue
|
||||
try: src = zlib.decompress(ROM[addr:addr+size][2:], -15)
|
||||
except Exception as e:
|
||||
errors.append(f"{base}: decompress {e}"); continue
|
||||
D = len(src); stats["files"] += 1
|
||||
|
||||
nodes, gdls = walk(src, D, NS, NT)
|
||||
if not nodes:
|
||||
errors.append(f"{base}: empty tree"); continue
|
||||
|
||||
# ---- pass 1: layout + region map ----
|
||||
regions = [] # (old_off, old_size, new_off)
|
||||
def add_region(old, osz, cur):
|
||||
regions.append((old, osz, cur))
|
||||
return cur + osz
|
||||
|
||||
dstpos = 0
|
||||
for i in range(NS):
|
||||
dstpos = add_region(4*i, 4, dstpos) # switches 4B -> 8B each
|
||||
for i in range(NT):
|
||||
dstpos = add_region(4*NS + 12*i, 12, dstpos) # texconfigs 12B -> 12B
|
||||
|
||||
node_newoff = {}; rec_newoff = {}
|
||||
gdl_spans = [] # (old_g, slots_emitted)
|
||||
vtx_regions = [] # for G_VTX seg5 remap sanity (subset of regions)
|
||||
zero_vtx = [] # (vo,) arrays with nv==0 but non-null ptr — sized up to next object
|
||||
for (no, op) in nodes:
|
||||
data = be32o(src, no+4)
|
||||
dstpos = add_region(no, 24, dstpos) # node 24B -> 48B
|
||||
node_newoff[no] = dstpos - 24
|
||||
if op not in N64_REC:
|
||||
errors.append(f"{base}: unknown opcode {op} at {no:#x}")
|
||||
continue
|
||||
nsz, psz = N64_REC[op], PC_REC[op]
|
||||
rec_newoff[data] = dstpos
|
||||
dstpos = add_region(data, nsz, dstpos)
|
||||
# vertex arrays right after the record
|
||||
# (nv==0 with non-null ptr: real data exists, GDL uploads via abs offset —
|
||||
# size it up to the next object; see PexplosionbitZ)
|
||||
if op == 4:
|
||||
nv = bu16(src, data+0x10); vo = be32o(src, data+0xC)
|
||||
if vo:
|
||||
sz = 16*nv if nv else None
|
||||
if sz is None: zero_vtx.append((vo,))
|
||||
else: vtx_regions.append((vo, nv))
|
||||
if nv: dstpos = add_region(vo, sz, dstpos)
|
||||
elif op == 24:
|
||||
nv = be16(src, data+0xC); ncv = be16(src, data+0xE)
|
||||
vo = be32o(src, data+8); cvo = be32o(src, data+0x10); puo = be32o(src, data+0x14)
|
||||
if nv and vo:
|
||||
vtx_regions.append((vo, nv)); dstpos = add_region(vo, 16*nv, dstpos)
|
||||
if ncv and cvo:
|
||||
vtx_regions.append((cvo, ncv)); dstpos = add_region(cvo, 16*ncv, dstpos)
|
||||
if nv and puo:
|
||||
dstpos = add_region(puo, 2*nv, dstpos) # PointUsage s16 x numVertices
|
||||
elif op == 22:
|
||||
nv = struct.unpack_from(">i", src, data)[0]; vo = be32o(src, data+4)
|
||||
if nv and vo:
|
||||
vtx_regions.append((vo, nv)); dstpos = add_region(vo, 16*nv, dstpos)
|
||||
|
||||
# zero-count vertex arrays: size up to the next higher object offset
|
||||
obj_offs_now = sorted(set([no for no,_ in nodes]) | set(rec_newoff.keys()) |
|
||||
set(v[0] for v in vtx_regions) | set(gdls))
|
||||
for (vo,) in zero_vtx:
|
||||
nxt = min((o for o in obj_offs_now if o > vo), default=D)
|
||||
sz = nxt - vo
|
||||
if sz <= 0 or sz % 16: errors.append(f"{base}: zero-vtx array {vo:#x} bad size {sz:#x}")
|
||||
else:
|
||||
vtx_regions.append((vo, sz // 16))
|
||||
dstpos = add_region(vo, sz, dstpos)
|
||||
|
||||
# embedded texture blobs (texconfig TextureID seg5)
|
||||
blob_old = []
|
||||
for i in range(NT):
|
||||
tid = be32r(src, 4*NS + 12*i)
|
||||
if (tid >> 28) == 5:
|
||||
blob_old.append(tid & MASK)
|
||||
# blob size: up to the next higher object offset
|
||||
all_offs = sorted(set([no for no,_ in nodes]) | set(rec_newoff.keys()) |
|
||||
set(v[0] for v in vtx_regions) | set(gdls) | set(4*NS+12*i for i in range(NT)))
|
||||
for bo in blob_old:
|
||||
nxt = min((o for o in all_offs if o > bo), default=D)
|
||||
dstpos = add_region(bo, nxt - bo, dstpos)
|
||||
blob_info.append((base, hex(bo), nxt - bo))
|
||||
|
||||
# GDLs last, tight-packed 16B slots (up to and incl. ENDDL)
|
||||
gdl_newoff = {}
|
||||
total_slots = 0
|
||||
for g in gdls:
|
||||
end = gdl_end(src, D, g)
|
||||
nslots = (end - g) // 8
|
||||
if (end - g) % 8: errors.append(f"{base}: GDL {g:#x} not 8B aligned end")
|
||||
gdl_newoff[g] = dstpos
|
||||
dstpos += 16 * nslots
|
||||
total_slots += nslots
|
||||
regions.append((g, end - g, gdl_newoff[g]))
|
||||
|
||||
D_PC = dstpos
|
||||
G_PC = min(gdl_newoff.values()) if gdl_newoff else D_PC
|
||||
if gdl_newoff and (D_PC - G_PC) % 16:
|
||||
errors.append(f"{base}: D_PC-G_PC not multiple of 16")
|
||||
ratio = D_PC / D if D else 0
|
||||
if ratio > max_ratio: max_ratio, max_file = ratio, base
|
||||
stats["dpc_max"] = max(stats["dpc_max"], D_PC)
|
||||
|
||||
# ---- pass 2: remap sanity (every pointer field must resolve) ----
|
||||
regions.sort()
|
||||
import bisect
|
||||
rold = [x[0] for x in regions]; rmap = {x[0]: x for x in regions}
|
||||
def remap(off):
|
||||
if off == 0: return 0
|
||||
i = bisect.bisect_right(rold, off) - 1
|
||||
if i < 0: return None
|
||||
o, osz, n = regions[i]
|
||||
if o <= off < o + osz: return n + (off - o)
|
||||
return None
|
||||
|
||||
def check(tag, off):
|
||||
if off == 0: return
|
||||
if remap(off) is None:
|
||||
errors.append(f"{base}: {tag} -> {off:#x} not in map")
|
||||
|
||||
for (no, op) in nodes:
|
||||
data = be32o(src, no+4)
|
||||
for f in (4, 8, 0xC, 0x10, 0x14): check(f"node{no:#x}+{f:#x}", be32o(src, no+f))
|
||||
if op not in N64_REC: continue
|
||||
ptrfields = {
|
||||
1: [4], 2: [0x14], 3: [0x14], 8: [8], 9: [0x18, 0x1C],
|
||||
12: [0x18], 13: [0x10, 0x14], 18: [0], 24: [8, 0x10, 0x14],
|
||||
4: [0xC], 22: [4],
|
||||
}.get(op, [])
|
||||
for f in ptrfields: check(f"rec{op}@{data:#x}+{f:#x}", be32o(src, data+f))
|
||||
if op == 24:
|
||||
ncv = be16(src, data+0xE)
|
||||
cvo = be32o(src, data+0x10)
|
||||
for i in range(ncv):
|
||||
check(f"colvtx{cvo:#x}[{i}].LinkedTo", be32r(src, cvo + 16*i + 8) & MASK if (be32r(src, cvo+16*i+8)>>24)==5 else 0)
|
||||
for i in range(NS): check(f"switch[{i}]", be32o(src, 4*i))
|
||||
|
||||
# GDL w1 seg5 targets must resolve — ONLY for address-bearing opcodes.
|
||||
# (G_TRI4/G_TRI1/G_TEXTURE/SETOTHERMODE/etc carry index data in w1, not addresses.)
|
||||
ADDR_OPS = {0x04, 0xFD, 0xF3} # G_VTX, G_SETTIMG, G_LOADBLOCK
|
||||
for g in gdls:
|
||||
o = g
|
||||
while o + 8 <= D:
|
||||
w0 = be32r(src, o); c = w0 >> 24
|
||||
if c == 0xB8: break
|
||||
w1 = be32r(src, o+4)
|
||||
if c in ADDR_OPS and (w1 >> 28) == 5:
|
||||
check(f"gdl{g:#x}@{o:#x} op{c:#x} w1", w1 & MASK)
|
||||
o += 8
|
||||
|
||||
stats["max_ratio"] = round(max_ratio, 3)
|
||||
print("files:", stats["files"], " max D_PC:", hex(stats["dpc_max"]), f" max D_PC/D_N64 ratio: {max_ratio:.2f} ({max_file})")
|
||||
print("embedded blobs:", blob_info)
|
||||
if errors:
|
||||
print(f"\n{len(errors)} ERRORS:")
|
||||
for e in errors[:30]: print(" ", e)
|
||||
else:
|
||||
print("ALL CLEAN: every remap target resolves, layout invariants hold")
|
||||
@@ -0,0 +1,83 @@
|
||||
# Verify every seg-5 VTX offset and SETTIMG w1 target falls inside a modeled object.
|
||||
import struct, zlib
|
||||
exec(open("tools_pc/d43_sizes.py").read().split("stats = []")[0])
|
||||
|
||||
RSZ = {1:0x10,2:0x1C,4:0x14,8:0x10,9:0x24,10:0x1C,12:0x28,13:0x20,
|
||||
15:0x1C,17:0x20,18:0x8,21:0x14,22:0x10,23:2,24:0x20}
|
||||
|
||||
def be32(b,o): return struct.unpack_from(">I",b,o)[0]
|
||||
def off(p): # vma 0x05xxxxxx -> file offset; non-vma pointers are invalid
|
||||
return (p & 0xFFFFFF) if (p >> 24) == 5 else None
|
||||
|
||||
def walk(out,D,NS,NT):
|
||||
R0n=4*NS+12*NT
|
||||
stack=[R0n]; seen=set(); objs=[]; gdls=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D-23: break
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4)
|
||||
objs.append((o,24,"node"))
|
||||
rec=off(data)
|
||||
if rec is not None and rec<D: objs.append((rec,RSZ[op],"record"))
|
||||
if op in (4,24) and rec is not None and rec<D:
|
||||
p,s=off(be32(out,rec)),off(be32(out,rec+4))
|
||||
v=off(be32(out,(rec+8) if op==24 else (rec+0xC)))
|
||||
nv=struct.unpack_from(">h",out,rec+0xC)[0] if op==24 else struct.unpack_from(">H",out,rec+0x10)[0]
|
||||
if nv>0 and v is not None and v<D: objs.append((v,16*nv,"verts"))
|
||||
if p is not None: gdls.append(p)
|
||||
if s is not None: gdls.append(s)
|
||||
if op==24:
|
||||
nc=struct.unpack_from(">h",out,rec+0xE)[0]
|
||||
cv=off(be32(out,rec+0x10)); pu=off(be32(out,rec+0x14))
|
||||
if nc>0 and cv is not None and cv<D: objs.append((cv,16*nc,"collverts"))
|
||||
if nv>0 and pu is not None and pu<D: objs.append((pu,2*nv,"pointusage"))
|
||||
elif op==22 and rec is not None and rec<D:
|
||||
nv=struct.unpack_from(">i",out,rec)[0]; v=off(be32(out,rec+4))
|
||||
p=off(be32(out,rec+8))
|
||||
if nv>0 and v is not None and v<D: objs.append((v,16*nv,"verts"))
|
||||
if p is not None: gdls.append(p)
|
||||
for q in (be32(out,o+0x14), be32(out,o+0xC)):
|
||||
x=off(q)
|
||||
if x is not None and x<D-23: stack.append(x)
|
||||
return sorted(set(gdls)), objs
|
||||
|
||||
bad_vtx=0; bad_settimg=0; tot_vtx=0; tot_settimg=0; files=0
|
||||
for r in rows:
|
||||
if len(r)<3 or not r[2]: continue
|
||||
base=r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base=base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key=None
|
||||
for cand in (base,base[1:],base[:-1],base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
try: out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
except Exception: continue
|
||||
D=len(out); NS,NT=nsnt[key]
|
||||
gs,objs=walk(out,D,NS,NT)
|
||||
for i,g in enumerate(gs):
|
||||
e=gs[i+1] if i+1<len(gs) else D
|
||||
o=g
|
||||
while o+8<=e:
|
||||
w0=be32(out,o); t=w0>>24
|
||||
w1=be32(out,o+4)
|
||||
top=(w1>>24)&0xff
|
||||
if t==4 and top==5:
|
||||
tot_vtx+=1
|
||||
x=w1&0xffffff
|
||||
if not any(s<=x<s+n for s,n,_ in objs):
|
||||
bad_vtx+=1
|
||||
if bad_vtx<6: print("VTX outside object:",base,hex(g),hex(o),hex(w1))
|
||||
elif t==0xFD and top==5:
|
||||
tot_settimg+=1
|
||||
x=w1&0xffffff
|
||||
if not any(s<=x<s+n for s,n,_ in objs):
|
||||
bad_settimg+=1
|
||||
if bad_settimg<6: print("SETTIMG outside object:",base,hex(g),hex(o),hex(w1))
|
||||
o+=8
|
||||
files+=1
|
||||
print(f"files={files} vtx_seg5={tot_vtx} (outside objs: {bad_vtx}) settimg={tot_settimg} (outside objs: {bad_settimg})")
|
||||
@@ -0,0 +1,77 @@
|
||||
# D43 helper: decode a GE model file from ROM (N64 layout) and dump its
|
||||
# node tree + GDL command streams. Usage: python d43_decode.py <name-substr>
|
||||
import csv, struct, sys, zlib
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
entries = []
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]:
|
||||
continue
|
||||
try:
|
||||
addr, size = int(r[0]), int(r[1])
|
||||
except ValueError:
|
||||
continue
|
||||
entries.append((addr, size, r[2]))
|
||||
|
||||
flt = sys.argv[1] if len(sys.argv) > 1 else ""
|
||||
want = None
|
||||
for addr, size, name in sorted(entries):
|
||||
base = name.rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"):
|
||||
base = base[:-4]
|
||||
if flt and flt.lower() in base.lower():
|
||||
want = (addr, size, base)
|
||||
break
|
||||
if not want:
|
||||
sys.exit("no file matching %r" % flt)
|
||||
|
||||
addr, size, base = want
|
||||
data = rom[addr:addr + size]
|
||||
out = zlib.decompress(data[2:], -15)
|
||||
print("%s: rom=%d decomp=%d (0x%X)" % (base, size, len(out), len(out)))
|
||||
|
||||
# N64 header-less layout: switches[NS] then textures[NT] then RootNode.
|
||||
# We don't know NS/NT from the file alone; find them via the exe record is
|
||||
# not available here, so heuristic: scan for a plausible node tree start.
|
||||
# Instead: dump raw words and let the eye decode. Print first 64 bytes.
|
||||
print("head:", out[:64].hex())
|
||||
|
||||
def be16(b, o): return struct.unpack_from(">H", b, o)[0]
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0]
|
||||
|
||||
# Try to find GDL streams: sequences of 8-byte words whose w0>>24 is a known cmd.
|
||||
CMDS = {0xc0:"NOOP",0x05:"VTX",0x0f:"MTX",0x13:"DL",0x15:"MOVEWORD",0x19:"POPMTX",
|
||||
0x02:"MATRIXLOAD",0x07:"CLIPRATIO",0x0b:"SPSETOTHERMODE?",0x0d:"ENVMODE?",
|
||||
0x23:"RDPPPIPESYNC",0x25:"RDPSETSCISSOR?",0x26:"RDPSETOPTIONS?",0x27:"RDPLOADTLUT?",
|
||||
0x28:"RDPSETCONVERT?",0x29:"RDPSETSCENECULL?",0x2a:"RDPFILLRECT",0x2b:"RDPSETFOGCOLOR?",
|
||||
0x2c:"RDPSETPRIMCOLOR",0x2d:"RDPSETPROJECT?",0x2e:"RDPSETBADEVENTS?",0x2f:"RDPLOADUTILE?",
|
||||
0x31:"RDPSETENVCOLOR",0x32:"RDPSETCOMBINE",0xba:"RDPSETOTHERMODE_H",0xbb:"RDPSETOTHERMODE_L",
|
||||
0xbf:"RDPSETTEXIMAGE",0xb1:"RDPSETTILESIZE?",0xbbb:None}
|
||||
def cmdname(c):
|
||||
return CMDS.get(c, "0x%02x" % c)
|
||||
|
||||
# scan for the longest run of plausible Gfx words
|
||||
best = (0, 0, 0)
|
||||
i = 0
|
||||
while i + 8 <= len(out):
|
||||
w0 = be32(out, i)
|
||||
c = w0 >> 24
|
||||
if c in CMDS:
|
||||
j = i
|
||||
while j + 8 <= len(out) and (be32(out, j) >> 24) in CMDS:
|
||||
j += 8
|
||||
L = j - i
|
||||
if L > best[0]:
|
||||
best = (L, i, c)
|
||||
i += 4
|
||||
|
||||
L, off, c0 = best
|
||||
print("\nlongest GDL-like run: %d bytes at offset 0x%X (starts cmd %s)" % (L, off, cmdname(c0)))
|
||||
# dump first 24 words of that run
|
||||
for k in range(min(24, L // 8)):
|
||||
w0 = be32(out, off + 8*k)
|
||||
w1 = be32(out, off + 8*k + 4)
|
||||
print(" [%2d] %s w0=%08X w1=%08X" % (k, cmdname(w0 >> 24), w0, w1))
|
||||
@@ -0,0 +1,143 @@
|
||||
# D43: definitive N64 model-file layout analysis.
|
||||
# Walks nodes via Child(+0x14) AND Next(+0x0C). Records all objects with sizes.
|
||||
# Checks: full tiling [0,D), GDL contiguity at tail, node/record invariants.
|
||||
import csv, struct, zlib, os, re, sys
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
|
||||
# N64 record sizes per opcode (bondtypes.h, verified: no rec-overflow in 512 files)
|
||||
RSZ = {1:0x10, 2:0x1C, 3:0x1C, 4:0x14, 5:0x1AC, 6:0x18, 7:0x1B0, 8:0x10,
|
||||
9:0x24, 10:0x1C, 11:0x50, 12:0x28, 13:0x20, 14:0x10, 15:0x1C,
|
||||
16:0x18, 17:0x20, 18:0x08, 21:0x14, 22:0x10, 23:0x02, 24:0x20}
|
||||
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be16(b, o): return struct.unpack_from(">H", b, o)[0]
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
def be32s(b, o):
|
||||
v = struct.unpack_from(">i", b, o)[0]
|
||||
return v & 0xFFFFFF if v >= 0 else -v
|
||||
|
||||
nfiles = 0; fails = []; gaphist = Counter(); gdlnotatend = []
|
||||
nodecount_total = 0; sibling_only = 0
|
||||
onlychild_counts = Counter()
|
||||
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size = int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key = cand; break
|
||||
if not key: fails.append((base,"nohdr")); continue
|
||||
NS,NT = nsnt[key]
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: fails.append((base,"decomp")); continue
|
||||
D = len(out); nfiles += 1
|
||||
R0 = 4*NS + 12*NT
|
||||
|
||||
# --- walk nodes: Child + Next ---
|
||||
objs = [(0, R0, "hdr")]
|
||||
stack = [(R0, 0)]; seen = set()
|
||||
while stack:
|
||||
o, via = stack.pop()
|
||||
if o in seen or not (0 <= o < D): fails.append((base,f"badnode {o:#x}")); break
|
||||
seen.add(o)
|
||||
op = be16(out, o) & 0xff
|
||||
data = be32(out, o+4)
|
||||
child = be32(out, o+0x14); nxt = be32(out, o+0xC)
|
||||
objs.append((o, o+24, "node"))
|
||||
rsz = RSZ.get(op, 0)
|
||||
if not rsz: fails.append((base,f"noRSZ op{op} @ {o:#x}")); break
|
||||
if data + rsz > D: fails.append((base,f"recovf op{op} {data:#x}+{rsz:#x}>{D:#x}")); break
|
||||
objs.append((data, data+rsz, f"rec{op}"))
|
||||
# per-opcode sub-objects
|
||||
if op == 4: # DL
|
||||
p = be32(out, data); s = be32(out, data+4)
|
||||
v = be32(out, data+0xC); nv = be16(out, data+0x10)
|
||||
if p: objs.append((p, None, "gdl"))
|
||||
if s: objs.append((s, None, "gdl"))
|
||||
if v and nv: objs.append((v, v+16*nv, "vtx"))
|
||||
elif op == 24: # DLCOLLISION
|
||||
p = be32(out, data); s = be32(out, data+4)
|
||||
v = be32(out, data+8); nv = be16(out, data+0xC); nc = be16(out, data+0xE)
|
||||
cv = be32(out, data+0x10); puo = be32(out, data+0x14)
|
||||
if p: objs.append((p, None, "gdl"))
|
||||
if s: objs.append((s, None, "gdl"))
|
||||
if v and nv: objs.append((v, v+16*nv, "vtx"))
|
||||
if cv and nc: objs.append((cv, cv+16*nc, "coll"))
|
||||
if puo: objs.append((puo, None, "pu"))
|
||||
elif op == 22: # DLPRIMARY
|
||||
nv = be32s(out, data)
|
||||
v = be32(out, data+4); p = be32(out, data+8)
|
||||
if p: objs.append((p, None, "gdl"))
|
||||
if v and nv > 0: objs.append((v, v+16*nv, "vtx22"))
|
||||
elif op == 9: # BSP: check for embedded pointers? (size 0x24)
|
||||
pass
|
||||
if child: stack.append((child, 1))
|
||||
if nxt: stack.append((nxt, 2))
|
||||
else:
|
||||
nodecount_total += len(seen)
|
||||
# count nodes not reachable as anyone's Child (sibling-only, excl. root)
|
||||
childtargets = set()
|
||||
for o in seen:
|
||||
c = be32(out, o+0x14)
|
||||
if c: childtargets.add(c)
|
||||
sibling_only += len(seen - childtargets - {R0})
|
||||
# resolve open-ended (gdl/pu): sorted starts
|
||||
starts = sorted(set(s for s,e,k in objs))
|
||||
fixed = []
|
||||
for (s,e,k) in objs:
|
||||
if e is None:
|
||||
nxts = [x for x in starts if x > s]
|
||||
fixed.append((s, min(nxts) if nxts else D, k))
|
||||
else:
|
||||
fixed.append((s,e,k))
|
||||
# tiling
|
||||
ivs = sorted((s,e) for s,e,k in fixed)
|
||||
pos = 0; gaps = []
|
||||
for s,e in ivs:
|
||||
if s < pos: fails.append((base,f"overlap {s:#x}<pos{pos:#x}")); break
|
||||
if s > pos: gaps.append(s-pos)
|
||||
pos = e
|
||||
else:
|
||||
if pos != D: fails.append((base,f"tail {pos:#x}!={D:#x}"))
|
||||
for g in gaps: gaphist[g] += 1
|
||||
# GDL tail contiguity: all gdl objs must form one contiguous run ending at D
|
||||
gs = sorted(set(s for s,e,k in fixed if k=="gdl"))
|
||||
if gs:
|
||||
# check union of gdl intervals == [min(gs), D) with no non-gdl inside
|
||||
merged = []
|
||||
for s,e,k in sorted(fixed):
|
||||
if k == "gdl":
|
||||
if merged and s <= merged[-1][1]: merged[-1] = (merged[-1][0], max(merged[-1][1], e))
|
||||
else: merged.append([s,e])
|
||||
if len(merged) > 1 or merged[0][1] != D:
|
||||
gdlnotatend.append((base, [f"{a:#x}-{b:#x}" for a,b in merged]))
|
||||
|
||||
print(f"files: {nfiles} fails: {len(fails)}")
|
||||
for f in fails[:20]: print(" ", f)
|
||||
print(f"total nodes (Child+Next walk): {nodecount_total}")
|
||||
print(f"nodes reached only via Next (not first sibling): {sibling_only}")
|
||||
print(f"gap-size histogram: {dict(sorted(gaphist.items()))}")
|
||||
print(f"GDL-not-single-tail-run files: {len(gdlnotatend)}")
|
||||
for b, m in gdlnotatend[:10]: print(" ", b, m)
|
||||
@@ -0,0 +1,92 @@
|
||||
# D43 helper: dump real GDL command streams from model files (N64 layout),
|
||||
# showing w1 top-byte distribution per opcode. Usage: python d43_gdldump.py <name-substr>
|
||||
import csv, struct, sys, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
|
||||
flt = sys.argv[1] if len(sys.argv) > 1 else ""
|
||||
limit = int(sys.argv[2]) if len(sys.argv) > 2 else 4
|
||||
shown = 0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
if flt and flt.lower() not in base.lower(): continue
|
||||
try: addr, size = int(r[0]), int(r[1])
|
||||
except ValueError: continue
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key = cand; break
|
||||
if not key: continue
|
||||
NS, NT = nsnt[key]
|
||||
try: out = zlib.decompress(rom[addr:addr + size][2:], -15)
|
||||
except Exception: continue
|
||||
D = len(out)
|
||||
R0 = 4 * NS + 12 * NT
|
||||
|
||||
# collect DL / DLCOLLISION record gdl offsets (full walk: Child+Next+Parent)
|
||||
stack = [R0]; seen = set(); gdls = []
|
||||
while stack:
|
||||
o = stack.pop()
|
||||
if o in seen or o >= D: continue
|
||||
seen.add(o)
|
||||
op = struct.unpack_from(">H", out, o)[0] & 0xff
|
||||
data = be32(out, o + 4)
|
||||
parent = be32(out, o + 8); nxt = be32(out, o + 0xC)
|
||||
child = be32(out, o + 0x14)
|
||||
if op in (4, 24):
|
||||
p = be32(out, data); s = be32(out, data + 4)
|
||||
if p: gdls.append((p, "P"))
|
||||
if s: gdls.append((s, "S"))
|
||||
for q in (child, nxt, parent):
|
||||
if q: stack.append(q)
|
||||
|
||||
print("== %s D=0x%X NS=%d NT=%d gdls=%d" % (base, D, NS, NT, len(gdls)))
|
||||
topbytes = Counter(); opcounts = Counter()
|
||||
for goff, tag in gdls[:2]:
|
||||
# walk the GDL: commands are 8B slots; stop at ENDDL or invalid opcode
|
||||
i = goff; n = 0
|
||||
while i + 8 <= D and n < 400:
|
||||
w0 = be32(out, i); w1 = be32(out, i + 4)
|
||||
c = w0 >> 24
|
||||
opcounts[c] += 1
|
||||
if c == 0x1d: break # ENDDL
|
||||
i += 8; n += 1
|
||||
print(" %s @0x%X (%d slots, ends at 0x%X)" % (tag, goff, n, i))
|
||||
for k in range(min(n, 14)):
|
||||
w0 = be32(out, goff + 8 * k); w1 = be32(out, goff + 8 * k + 4)
|
||||
print(" [%2d] c=%02X w0=%08X w1=%08X" % (k, w0 >> 24, w0, w1))
|
||||
for goff, tag in gdls:
|
||||
i = goff; n = 0
|
||||
while i + 8 <= D and n < 400:
|
||||
w0 = be32(out, i); w1 = be32(out, i + 4)
|
||||
c = w0 >> 24
|
||||
if c == 0x1d: break
|
||||
topbytes[(c, (w1 >> 24) & 0xff)] += 1
|
||||
i += 8; n += 1
|
||||
print(" opcode/topbyte histogram:")
|
||||
for (c, tb), cnt in sorted(topbytes.items()):
|
||||
print(" c=%02X top(w1)=%02X x%d" % (c, tb, cnt))
|
||||
shown += 1
|
||||
if shown >= limit: break
|
||||
@@ -0,0 +1,80 @@
|
||||
# D43: histogram of (opcode, w1 top byte) across ALL model GDLs, plus per-file
|
||||
# max GDL extent checks. Also verifies every GDL ends in ENDDL (0xB8).
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
def be32raw(b, o): return struct.unpack_from(">I", b, o)[0]
|
||||
|
||||
hist = Counter(); seg5 = Counter(); problems = []
|
||||
nfiles = 0; ngdl = 0; maxslots = 0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size = int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: problems.append((base,"nohdr")); continue
|
||||
NS,NT = nsnt[key]
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: problems.append((base,"decomp")); continue
|
||||
D = len(out); R0 = 4*NS + 12*NT
|
||||
stack=[R0]; seen=set(); gdls=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D: problems.append((base,"badnode")); break
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4)
|
||||
ch=be32(out,o+0x14)
|
||||
if op in (4,24):
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
# Child+Next only: matches modelIterateDisplayLists reachability
|
||||
for q in (ch, be32(out,o+0xC)):
|
||||
if q: stack.append(q)
|
||||
nfiles += 1
|
||||
for g in set(gdls):
|
||||
ngdl += 1
|
||||
i=g; n=0; ended=False
|
||||
while i+8 <= D and n < 100000:
|
||||
w0=be32raw(out,i); w1=be32raw(out,i+4)
|
||||
c=w0>>24
|
||||
hist[(c,(w1>>24)&0xff)] += 1
|
||||
if (w1>>24)&0xff == 5 and c in (1,3,4,6,7,0xfd,0xfe,0xff):
|
||||
seg5[(c,w1&0xffffff)] += 1
|
||||
if c==0xB8: ended=True; break
|
||||
i+=8; n+=1
|
||||
maxslots=max(maxslots,n)
|
||||
if not ended: problems.append((base,"gdl %X no ENDDL (n=%d)"%(g,n)))
|
||||
print("files:",nfiles,"gdls:",ngdl,"max slots in one gdl:",maxslots)
|
||||
print("\n(opcode, w1 top byte) histogram:")
|
||||
for (c,tb),cnt in sorted(hist.items()):
|
||||
print(" c=%02X seg=%02X x%d" % (c,tb,cnt))
|
||||
print("\nseg-5 address commands (opcode, offset):")
|
||||
for (c,off),cnt in sorted(seg5.items()):
|
||||
print(" c=%02X off=%06X x%d" % (c,off,cnt))
|
||||
print("\nproblems:",len(problems))
|
||||
for p in problems[:10]: print(" ",p)
|
||||
@@ -0,0 +1,144 @@
|
||||
# D43: EXACT modelIterateDisplayLists visit-order simulation (mutable nodes,
|
||||
# LOD/SWITCH rewiring, BSP sibling splice). Verify per file that consecutive
|
||||
# visited GDLs are offset-adjacent (count == cur size), i.e. a converter may
|
||||
# emit GDLs in visit order and compaction will work.
|
||||
import csv, struct, zlib, os, re
|
||||
|
||||
ROM=r"data/ge007.ntsc-final.z64"
|
||||
rom=open(ROM,"rb").read()
|
||||
rows=list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
|
||||
hdrfiles=set()
|
||||
for root,dirs,files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root,fn))
|
||||
nsnt={}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m=re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$",line)
|
||||
if not m: continue
|
||||
a=[x.strip() for x in m.group(1).split(",")]
|
||||
if len(a)<9: continue
|
||||
try: nsnt[a[0]]=(int(a[4],0),int(a[8],0))
|
||||
except ValueError: pass
|
||||
|
||||
def be16(b,o): return struct.unpack_from(">H",b,o)[0]
|
||||
def be32(b,o): return struct.unpack_from(">I",b,o)[0]&0xFFFFFF
|
||||
|
||||
class N: # mutable node
|
||||
__slots__=("op","data","parent","next","prev","child")
|
||||
|
||||
bad=0; total=0; gdls_total=0
|
||||
for r in rows:
|
||||
if len(r)<3 or not r[2]: continue
|
||||
base=r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base=base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key=None
|
||||
for cand in (base,base[1:],base[:-1],base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT=nsnt[key]
|
||||
try: out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
except Exception: continue
|
||||
D=len(out); R0=4*NS+12*NT; total+=1
|
||||
|
||||
nodes={}
|
||||
stack=[R0]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in nodes: continue
|
||||
n=N(); n.op=be16(out,o)&0xff; n.data=be32(out,o+4)
|
||||
n.parent=be32(out,o+8); n.next=be32(out,o+0xC); n.prev=be32(out,o+0x10); n.child=be32(out,o+0x14)
|
||||
nodes[o]=n
|
||||
# Child+Next closure (mutable walk reaches siblings via Next), plus
|
||||
# LOD.Affects / SWITCH.Controls targets in case they lie outside it.
|
||||
for q in (n.child, n.next):
|
||||
if q: stack.append(q)
|
||||
if n.op==8:
|
||||
q=be32(out,n.data+8)
|
||||
if q: stack.append(q)
|
||||
elif n.op==18:
|
||||
q=be32(out,n.data+0)
|
||||
if q: stack.append(q)
|
||||
|
||||
def gdl_of(n):
|
||||
if n.op==4 or n.op==24:
|
||||
p=be32(out,n.data+0); s=be32(out,n.data+4)
|
||||
return (p,s)
|
||||
if n.op==22: # DLPRIMARY N64: numVertices s32@0, Vertices@4, Primary@8, BaseAddr@0xC
|
||||
return (be32(out,n.data+8), 0)
|
||||
return (0,0)
|
||||
|
||||
# full visit sequence: iterate the function to exhaustion
|
||||
seq=[]
|
||||
node=R0; prev_node=None; prev_gdl=0; guard=0
|
||||
while node and guard<200000:
|
||||
guard+=1
|
||||
n=nodes[node]
|
||||
gdl=0
|
||||
if n.op in (4,22,24):
|
||||
p,s=gdl_of(n)
|
||||
if node!=prev_node:
|
||||
gdl=p
|
||||
elif s and s!=prev_gdl:
|
||||
gdl=s
|
||||
elif n.op==8: # LOD rewire
|
||||
aff=be32(out,n.data+8)
|
||||
n.child=aff
|
||||
elif n.op==18: # SWITCH rewire
|
||||
ctl=be32(out,n.data+0)
|
||||
n.child=ctl
|
||||
elif n.op==9: # BSP splice (visible=TRUE)
|
||||
lc=be32(out,n.data+0x18); rc=be32(out,n.data+0x1C)
|
||||
node1,node2=lc,rc
|
||||
if node1:
|
||||
n.child=node1
|
||||
nodes[node1].prev=0
|
||||
loop=node1
|
||||
while nodes[loop].next and nodes[loop].next!=node2:
|
||||
loop=nodes[loop].next
|
||||
nodes[loop].next=node2
|
||||
if node2:
|
||||
nodes[node2].prev=loop
|
||||
loop=node2
|
||||
while nodes[loop].next and nodes[loop].next!=node1:
|
||||
loop=nodes[loop].next
|
||||
nodes[loop].next=0
|
||||
else:
|
||||
n.child=node2
|
||||
if node2: nodes[node2].prev=0
|
||||
if gdl:
|
||||
seq.append(gdl)
|
||||
prev_node=node; prev_gdl=gdl
|
||||
# next call resumes at same node (nodeptr unchanged) -> loop continues with same node
|
||||
continue
|
||||
prev_node=None
|
||||
if n.child:
|
||||
node=n.child
|
||||
else:
|
||||
while node:
|
||||
nn=nodes[node].next
|
||||
if nn: node=nn; break
|
||||
node=nodes[node].parent
|
||||
# verification: strictly ascending, no duplicates, each GDL ENDDLs before
|
||||
# the next visited start (so a converter may pack them contiguously).
|
||||
ok=True; why=""
|
||||
for i,g in enumerate(seq):
|
||||
end = seq[i+1] if i+1<len(seq) else D
|
||||
if i and g <= seq[i-1]: ok=False; why="not ascending @%X"%g; break
|
||||
if g in seq[:i]: ok=False; why="dup %X"%g; break
|
||||
j=g; n=0; ended=False
|
||||
while j+8<=end and n<100000:
|
||||
c=struct.unpack_from(">I",out,j)[0]>>24
|
||||
if c==0xB8: ended=True; break
|
||||
j+=8; n+=1
|
||||
if not ended: ok=False; why="no ENDDL in [%X,%X)"%(g,end); break
|
||||
if not ok:
|
||||
bad+=1
|
||||
if bad<=5: print("BAD",base,why,["%X"%v for v in seq[:10]],"D=%X"%D)
|
||||
gdls_total+=len(seq)
|
||||
|
||||
print("files:",total,"bad:",bad,"total gdls:",gdls_total)
|
||||
@@ -0,0 +1,56 @@
|
||||
import csv, struct, zlib, os, re, sys
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
def be32raw(b, o): return struct.unpack_from(">I", b, o)[0]
|
||||
flt = sys.argv[1] if len(sys.argv) > 1 else "CboilerbondZ"
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if flt.lower() not in base.lower(): continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
addr,size=int(r[0]),int(r[1])
|
||||
out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
D=len(out); R0=4*NS+12*NT
|
||||
stack=[R0]; seen=set(); gdls=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D: break
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4); ch=be32(out,o+0x14)
|
||||
if op in (4,24):
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
if p: gdls.append((p,"P",o))
|
||||
if s: gdls.append((s,"S",o))
|
||||
for q in (ch, be32(out,o+0xC)):
|
||||
if q: stack.append(q)
|
||||
print("== %s D=0x%X gdls=%d" % (base,D,len(gdls)))
|
||||
g = min(x[0] for x in gdls)
|
||||
i=g; n=0
|
||||
while i+8 <= D and n < 120:
|
||||
w0=be32raw(out,i); w1=be32raw(out,i+4)
|
||||
c=w0>>24
|
||||
print(" [%3d] c=%02X w0=%08X w1=%08X" % (n,c,w0,w1))
|
||||
if c==0xB8: break
|
||||
i+=8; n+=1
|
||||
break
|
||||
@@ -0,0 +1,111 @@
|
||||
# D43: full ownership tiling check per file.
|
||||
# objects: switches[0,4NS) textures[4NS,4NS+12NT) nodes(20B) records(RSZ)
|
||||
# vtx(nv*16) coll(nc*16) gdl(up to next gdl start or D)
|
||||
# PointUsage: [puo, min(next owned start)) -- verify even & >0
|
||||
# then verify union == [0,D) exactly (no gaps/overlaps).
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
RSZ={1:0x10,2:0x1C,4:0x14,8:0x10,9:0x24,10:0x1C,18:0x08,21:0x14,23:0x02,24:0x20}
|
||||
|
||||
hdrfiles=set()
|
||||
for root,dirs,files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root,fn))
|
||||
nsnt={}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m=re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$",line)
|
||||
if not m: continue
|
||||
a=[x.strip() for x in m.group(1).split(",")]
|
||||
if len(a)<9: continue
|
||||
try: nsnt[a[0]]=(int(a[4],0),int(a[8],0))
|
||||
except ValueError: pass
|
||||
|
||||
def be16(b,o): return struct.unpack_from(">H",b,o)[0]
|
||||
def be32(b,o): return struct.unpack_from(">I",b,o)[0]&0xFFFFFF
|
||||
|
||||
fails=[]; pu_sizes=Counter(); nfiles=0; gapfiles=[]
|
||||
for r in rows:
|
||||
if len(r)<3 or not r[2]: continue
|
||||
base=r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base=base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key=None
|
||||
for cand in (base,base[1:],base[:-1],base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: fails.append((base,"no header")); continue
|
||||
NS,NT=nsnt[key]
|
||||
try: out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
except Exception: fails.append((base,"decomp")); continue
|
||||
D=len(out); nfiles+=1
|
||||
R0=4*NS+12*NT
|
||||
|
||||
objs=[] # (start,end,kind)
|
||||
pu_starts=[]
|
||||
stack=[R0]; nodeset=set()
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in nodeset or o>=D: fails.append((base,"bad node %X"%o)); break
|
||||
nodeset.add(o)
|
||||
op=be16(out,o)&0xff
|
||||
data=be32(out,o+4); child=be32(out,o+20)
|
||||
objs.append((o,o+20,"node"))
|
||||
rsz=RSZ.get(op,0)
|
||||
if not rsz or data+rsz>D: fails.append((base,"bad rec op%d"%op)); break
|
||||
objs.append((data,data+rsz,"rec"))
|
||||
if op==4:
|
||||
for f_off in (0,4):
|
||||
v=be32(out,data+f_off)
|
||||
if v: objs.append((v,None,"gdl"))
|
||||
v=be32(out,data+0xC); nv=be16(out,data+0x10)
|
||||
if v: objs.append((v,v+16*nv,"vtx"))
|
||||
elif op==24:
|
||||
for f_off in (0,4):
|
||||
v=be32(out,data+f_off)
|
||||
if v: objs.append((v,None,"gdl"))
|
||||
v=be32(out,data+8); nv=be16(out,data+0xC); nc=be16(out,data+0xE)
|
||||
cv=be32(out,data+0x10); puo=be32(out,data+0x14)
|
||||
if v: objs.append((v,v+16*nv,"vtx"))
|
||||
if cv: objs.append((cv,cv+16*nc,"coll"))
|
||||
if puo: pu_starts.append(puo)
|
||||
if child: stack.append(child)
|
||||
else:
|
||||
# gdl extents: sorted gdl starts, each extends to next gdl start or D
|
||||
gs=sorted(set(s for s,e,k in objs if k=="gdl"))
|
||||
fixed=[]
|
||||
for i,(s,e,k) in enumerate(objs):
|
||||
if k=="gdl":
|
||||
end = gs[gs.index(s)+1] if gs.index(s)+1<len(gs) else D
|
||||
fixed.append((s,end,"gdl"))
|
||||
else: fixed.append((s,e,k))
|
||||
objs=fixed
|
||||
# pointusage: [puo, min next start)
|
||||
starts=sorted(set(s for s,e,k in objs))
|
||||
for puo in pu_starts:
|
||||
nxt=[s for s in starts if s>puo]
|
||||
end=min(nxt) if nxt else D
|
||||
sz=end-puo
|
||||
if sz<=0 or sz%2: fails.append((base,"pu bad size %d @%X"%(sz,puo))); break
|
||||
objs.append((puo,end,"pu"))
|
||||
pu_sizes[sz]+=1
|
||||
else:
|
||||
# tiling check
|
||||
ivs=sorted((s,e) for s,e,k in objs)
|
||||
ok=True; pos=0
|
||||
for s,e in ivs:
|
||||
if s<pos: fails.append((base,"overlap @%X (pos %X)"%(s,pos))); ok=False; break
|
||||
if s>pos: gapfiles.append((base,hex(pos),hex(s))); ok=False; break
|
||||
pos=e
|
||||
if ok and pos!=D: fails.append((base,"short end %X vs D=%X"%(pos,D)))
|
||||
|
||||
print("files:",nfiles,"fails:",len(fails))
|
||||
for f in fails[:15]: print(" ",f)
|
||||
print("gap files:",len(gapfiles))
|
||||
for g in gapfiles[:10]: print(" ",g)
|
||||
print("\nPointUsage sizes:",dict(sorted(pu_sizes.items())))
|
||||
@@ -0,0 +1,106 @@
|
||||
# D43: full layout analysis of all model files with CORRECT N64 sizes.
|
||||
# node = 24B; records per opcode; vertex/coll arrays; GDLs. Report gaps/overlaps.
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
RSZ = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:0x2, 24:0x20}
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
def be16(b, o): return struct.unpack_from(">H", b, o)[0]
|
||||
def be32raw(b, o): return struct.unpack_from(">I", b, o)[0]
|
||||
|
||||
gapstats = Counter(); problems = []
|
||||
nfiles = 0; totalnodes = 0; opcodes = Counter()
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size = int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: problems.append((base,"nohdr")); continue
|
||||
NS,NT = nsnt[key]
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: problems.append((base,"decomp")); continue
|
||||
D = len(out); R0 = 4*NS + 12*NT; nfiles += 1
|
||||
stack=[R0]; seen=set(); objs=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D-23: problems.append((base,"badnode %X"%o)); break
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
opcodes[op]+=1
|
||||
data=be32(out,o+4)
|
||||
objs.append((o,o+24,"node"))
|
||||
if op in RSZ:
|
||||
rsz = RSZ[op]
|
||||
if data + rsz > D: problems.append((base,"rec %X+%X>D"%(data,rsz)))
|
||||
else: objs.append((data,data+rsz,"rec%d"%op))
|
||||
if op==22:
|
||||
nv=struct.unpack_from(">i",out,data)[0]
|
||||
v=be32(out,data+4)
|
||||
p=be32(out,data+8)
|
||||
if p: objs.append((p,None,"gdl"))
|
||||
if v and nv>0: objs.append((v,v+16*nv,"vtx22"))
|
||||
if op==4:
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
v=be32(out,data+0xC)
|
||||
if p: objs.append((p,None,"gdl"))
|
||||
if s: objs.append((s,None,"gdl"))
|
||||
if v: objs.append((v,None,"vtx?")) # size unknown (numVertices may be 0)
|
||||
elif op==24:
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
v=be32(out,data+8); nv=be16(out,data+0xC)
|
||||
cv=be32(out,data+0x10); nc=be16(out,data+0xE)
|
||||
pu=be32(out,data+0x14)
|
||||
if p: objs.append((p,None,"gdl"))
|
||||
if s: objs.append((s,None,"gdl"))
|
||||
if v and nv: objs.append((v,v+16*nv,"vtx"))
|
||||
if cv and nc: objs.append((cv,cv+16*nc,"coll"))
|
||||
if pu: objs.append((pu,None,"pointusage?"))
|
||||
for q in (be32(out,o+0x14), be32(out,o+0xC)):
|
||||
if q: stack.append(q)
|
||||
totalnodes += len(seen)
|
||||
# resolve open-ended objects using sorted starts
|
||||
known = sorted(set(s for s,e,k in objs if e is not None))
|
||||
for i,(s,e,k) in enumerate(objs):
|
||||
if e is None:
|
||||
nxt = [x for x in known if x > s]
|
||||
objs[i] = (s, min(nxt) if nxt else D, k)
|
||||
ivs = sorted((s,e) for s,e,k in [(0,R0,"hdr")]+objs)
|
||||
pos = 0
|
||||
for s,e in ivs:
|
||||
if s < pos: problems.append((base,"overlap %X<%X"%(s,pos))); break
|
||||
if s > pos:
|
||||
gapstats[(s-pos)] += 1
|
||||
if len(problems) < 40:
|
||||
problems.append((base, "gap %X..%X (%d B)" % (pos, s, s - pos)))
|
||||
break
|
||||
pos = e
|
||||
else:
|
||||
if pos != D: problems.append((base,"tail gap %X!=D=%X"%(pos,D)))
|
||||
|
||||
print("files:",nfiles,"nodes:",totalnodes)
|
||||
print("opcodes:",dict(sorted(opcodes.items())))
|
||||
print("gap-size histogram (first gap per file):",dict(sorted(gapstats.items())[:20]))
|
||||
print("problems:",len(problems))
|
||||
for p in problems[:40]: print(" ",p)
|
||||
@@ -0,0 +1,77 @@
|
||||
import csv, struct, zlib, os, re, sys
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
RSZ = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:0x2, 24:0x20}
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
def be16(b, o): return struct.unpack_from(">H", b, o)[0]
|
||||
flt = sys.argv[1]
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if flt.lower() not in base.lower(): continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
addr,size=int(r[0]),int(r[1])
|
||||
out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
D=len(out); R0=4*NS+12*NT
|
||||
print("== %s NS=%d NT=%d R0=%X D=%X" % (base,NS,NT,R0,D))
|
||||
stack=[R0]; seen=set(); objs=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D-23: break
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4)
|
||||
objs.append((o,o+24,"node%d"%op))
|
||||
if op in RSZ: objs.append((data,data+RSZ[op],"rec%d"%op))
|
||||
if op==4:
|
||||
p=be32(out,data); s=be32(out,data+4); v=be32(out,data+0xC)
|
||||
if p: objs.append((p,None,"gdlP"))
|
||||
if s: objs.append((s,None,"gdlS"))
|
||||
if v: objs.append((v,None,"vtx?"))
|
||||
elif op==24:
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
v=be32(out,data+8); nv=be16(out,data+0xC)
|
||||
cv=be32(out,data+0x10); nc=be16(out,data+0xE)
|
||||
pu=be32(out,data+0x14)
|
||||
if p: objs.append((p,None,"gdlP"))
|
||||
if s: objs.append((s,None,"gdlS"))
|
||||
if v and nv: objs.append((v,v+16*nv,"vtx(%d)"%nv))
|
||||
if cv and nc: objs.append((cv,cv+16*nc,"coll(%d)"%nc))
|
||||
if pu: objs.append((pu,None,"pointusage?"))
|
||||
for q in (be32(out,o+0x14), be32(out,o+0xC)):
|
||||
if q: stack.append(q)
|
||||
known = sorted(set(s for s,e,k in objs if e is not None))
|
||||
res=[]
|
||||
for (s,e,k) in objs:
|
||||
if e is None:
|
||||
nxt=[x for x in known if x>s]
|
||||
e=min(nxt) if nxt else D
|
||||
res.append((s,e,k))
|
||||
res.sort()
|
||||
pos=0
|
||||
for s,e,k in res:
|
||||
mark = ""
|
||||
if s<pos: mark=" <<< OVERLAP (prev ends %X)"%pos
|
||||
elif s>pos: mark=" <<< GAP %d B"% (s-pos)
|
||||
print(" %05X..%05X %-14s%s" % (s,e,k,mark))
|
||||
pos=max(pos,e)
|
||||
break
|
||||
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Scan all model files' texture tables -> image headers -> format/maxlod.
|
||||
Determines whether any LUT textures (formats 9-12) exist, which affects
|
||||
worst-case marker expansion sizing (D45)."""
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
|
||||
imgrows = list(csv.reader(open("imagelist.u.csv")))
|
||||
img_off = [int(r[0]) for r in imgrows]
|
||||
img_sz = [int(r[1]) for r in imgrows]
|
||||
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
|
||||
nsnt = {}
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
f = os.path.join(root, fn)
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m:
|
||||
continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9:
|
||||
continue
|
||||
try:
|
||||
nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def bitread(buf, pos, n):
|
||||
val = 0
|
||||
for i in range(n):
|
||||
byte = buf[pos >> 3]
|
||||
bit = 7 - ((pos & 7) % 8)
|
||||
val = (val << 1) | ((byte >> bit) & 1)
|
||||
pos += 1
|
||||
return val, pos
|
||||
|
||||
fmtall = Counter()
|
||||
fmtlut = Counter()
|
||||
maxlod_dist = Counter()
|
||||
nimg = 0
|
||||
bad = 0
|
||||
zc = 0
|
||||
matched_files = 0
|
||||
lut_examples = []
|
||||
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]:
|
||||
continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"):
|
||||
base = base[:-4]
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt:
|
||||
key = cand
|
||||
break
|
||||
if not key:
|
||||
continue
|
||||
matched_files += 1
|
||||
NS, NT = nsnt[key]
|
||||
addr, size = int(r[0]), int(r[1])
|
||||
out = zlib.decompress(rom[addr:addr + size][2:], -15)
|
||||
D = len(out)
|
||||
texoff = 4 * NS + 12 * NT
|
||||
for i in range(NT):
|
||||
tid = struct.unpack_from(">I", out, texoff + i * 12)[0]
|
||||
if tid >= len(img_off):
|
||||
bad += 1
|
||||
continue
|
||||
nimg += 1
|
||||
o = img_off[tid]
|
||||
s = img_sz[tid]
|
||||
hdr = rom[o]
|
||||
iszlib = (hdr & 0x40) >> 6
|
||||
lod = hdr & 0x3f
|
||||
maxlod_dist[min(lod, 7)] += 1
|
||||
if iszlib:
|
||||
zc += 1
|
||||
fmt = (rom[o + 1] >> 4) & 0xf
|
||||
else:
|
||||
b2 = rom[o + 1:o + s]
|
||||
fmt, _ = bitread(b2, 0, 4)
|
||||
fmtall[fmt] += 1
|
||||
if fmt in (9, 10, 11, 12):
|
||||
fmtlut[fmt] += 1
|
||||
if len(lut_examples) < 10:
|
||||
lut_examples.append((key, i, tid, fmt))
|
||||
|
||||
print("nsnt entries:", len(nsnt))
|
||||
print("matched model files:", matched_files, "tex refs:", nimg, "bad:", bad, "zlib:", zc)
|
||||
print("formats used:", dict(sorted(fmtall.items())))
|
||||
print("LUT formats (9-12):", dict(fmtlut))
|
||||
print("maxlod dist:", dict(sorted(maxlod_dist.items())))
|
||||
for e in lut_examples:
|
||||
print(" LUT example:", e)
|
||||
@@ -0,0 +1,85 @@
|
||||
# D43: for every op24 record, compare PointUsage byte extent (to next owned object)
|
||||
# against numVertices*2 and other candidates.
|
||||
import csv, struct, zlib, os, re
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
RSZ = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28, 13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:0x2, 24:0x20}
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
def be16(b, o): return struct.unpack_from(">H", b, o)[0]
|
||||
from collections import Counter
|
||||
rel = Counter(); examples=[]
|
||||
nrec=0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size = int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: continue
|
||||
D = len(out); R0 = 4*NS + 12*NT
|
||||
stack=[R0]; seen=set(); objs=[]; recs=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D-23: break
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4)
|
||||
objs.append((o,o+24,"node"))
|
||||
if op in RSZ: objs.append((data,data+RSZ[op],"rec"))
|
||||
if op==4:
|
||||
p=be32(out,data); s=be32(out,data+4); v=be32(out,data+0xC)
|
||||
if p: objs.append((p,None,"gdl"))
|
||||
if s: objs.append((s,None,"gdl"))
|
||||
if v: objs.append((v,None,"vtx"))
|
||||
elif op==24:
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
v=be32(out,data+8); nv=be16(out,data+0xC)
|
||||
nc=be16(out,data+0xE); cv=be32(out,data+0x10)
|
||||
pu=be32(out,data+0x14)
|
||||
if p: objs.append((p,None,"gdl"))
|
||||
if s: objs.append((s,None,"gdl"))
|
||||
if v and nv: objs.append((v,v+16*nv,"vtx"))
|
||||
if cv and nc: objs.append((cv,cv+16*nc,"coll"))
|
||||
if pu: objs.append((pu,None,"pu"))
|
||||
recs.append((base,pu,nv,nc))
|
||||
for q in (be32(out,o+0x14), be32(out,o+0xC)):
|
||||
if q: stack.append(q)
|
||||
known = sorted(set(s for s,e,k in objs if e is not None))
|
||||
ends={}
|
||||
for (s,e,k) in objs:
|
||||
if e is None:
|
||||
nxt=[x for x in known if x>s]
|
||||
e=min(nxt) if nxt else D
|
||||
ends[s]=e
|
||||
for (base,pu,nv,nc) in recs:
|
||||
nrec+=1
|
||||
ext = ends[pu]-pu
|
||||
rel[ext - 2*nv]+=1
|
||||
if ext != 2*nv and len(examples)<10:
|
||||
examples.append((base,hex(pu),ext,nv,nc))
|
||||
print("op24 records:",nrec)
|
||||
print("(PointUsage extent - 2*numVertices) histogram:")
|
||||
for k,c in sorted(rel.items()): print(" %+d x%d" % (k,c))
|
||||
print("mismatch examples (base, pu, extent, nv, nc):")
|
||||
for e in examples: print(" ",e)
|
||||
@@ -0,0 +1,86 @@
|
||||
# D43: for every seg-5 VTX/MOVEMEM/MTX/DL reference in every GDL, check whether
|
||||
# the offset falls inside a known vertex/coll array or elsewhere.
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
RSZ={1:0x10,2:0x1C,4:0x14,8:0x10,9:0x24,10:0x1C,18:0x08,21:0x14,23:0x02,24:0x20}
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
def be16(b, o): return struct.unpack_from(">H", b, o)[0]
|
||||
def be32raw(b, o): return struct.unpack_from(">I", b, o)[0]
|
||||
|
||||
where = Counter(); examples = []
|
||||
nfiles=0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size = int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: continue
|
||||
D = len(out); R0 = 4*NS + 12*NT; nfiles+=1
|
||||
stack=[R0]; seen=set(); gdls=[]; arrays=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D: break
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4); ch=be32(out,o+0x14)
|
||||
if op==4:
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
v=be32(out,data+0xC); nv=be16(out,data+0x10)
|
||||
if v: arrays.append((v, v+16*nv, "vtx(op4)"))
|
||||
elif op==24:
|
||||
p=be32(out,data); s=be32(out,data+4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
v=be32(out,data+8); nv=be16(out,data+0xC)
|
||||
cv=be32(out,data+0x10); nc=be16(out,data+0xE)
|
||||
if v: arrays.append((v, v+16*nv, "vtx(op24)"))
|
||||
if cv: arrays.append((cv, cv+16*nc, "coll(op24)"))
|
||||
for q in (ch, be32(out,o+0xC)):
|
||||
if q: stack.append(q)
|
||||
for g in set(gdls):
|
||||
i=g; n=0
|
||||
while i+8 <= D and n < 100000:
|
||||
w0=be32raw(out,i); w1=be32raw(out,i+4)
|
||||
c=w0>>24
|
||||
if c in (1,3,4,6,7) and ((w1>>24)&0xff)==5:
|
||||
off = w1 & 0xFFFFFF
|
||||
hit=None
|
||||
for a,b,k in arrays:
|
||||
if a <= off < b: hit=k; break
|
||||
where[(c,hit)] += 1
|
||||
if hit is None and len(examples) < 8:
|
||||
examples.append((base, c, hex(off)))
|
||||
if c==0xB8: break
|
||||
i+=8; n+=1
|
||||
print("files:",nfiles)
|
||||
print("(opcode, region) for seg-5 refs:")
|
||||
for (c,hit),cnt in sorted(where.items(), key=lambda x:(x[0][0],str(x[0][1]))):
|
||||
print(" c=%02X %-10s x%d" % (c, str(hit), cnt))
|
||||
print("non-array examples:", examples)
|
||||
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Dump opcode + words for every seg-5 w1 slot in model GDLs, grouped by opcode."""
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
ROM = open('data/ge007.ntsc-final.z64','rb').read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
MASK = 0xFFFFFF
|
||||
def be32o(b,o): return struct.unpack_from(">I",b,o)[0] & MASK
|
||||
def be32r(b,o): return struct.unpack_from(">I",b,o)[0]
|
||||
|
||||
by_op = defaultdict(Counter) # opcode -> Counter of w1 values (seg5 only)
|
||||
inrange = Counter(); outrange = Counter()
|
||||
samples = defaultdict(list)
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
try: src=zlib.decompress(ROM[addr:addr+size][2:],-15)
|
||||
except Exception: continue
|
||||
D=len(src); R0=4*NS+12*NT
|
||||
seen=set(); stack=[R0]; gdls=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D: continue
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",src,o)[0]&0xff
|
||||
data=be32o(src,o+4)
|
||||
child=be32o(src,o+0x14); nxt=be32o(src,o+0xC)
|
||||
if op==8:
|
||||
a=be32o(src,data+8)
|
||||
if a: stack.append(a)
|
||||
elif op==18:
|
||||
c=be32o(src,data)
|
||||
if c: stack.append(c)
|
||||
else:
|
||||
if child: stack.append(child)
|
||||
if nxt: stack.append(nxt)
|
||||
if op in (4,24):
|
||||
p=be32o(src,data); s=be32o(src,data+4)
|
||||
gdls += [x for x in (p,s) if x]
|
||||
elif op==22:
|
||||
p=be32o(src,data+8)
|
||||
if p: gdls.append(p)
|
||||
for g in sorted(set(gdls)):
|
||||
o=g
|
||||
while o+8<=D:
|
||||
w0=be32r(src,o); c=w0>>24
|
||||
if c==0xB8: break
|
||||
w1=be32r(src,o+4)
|
||||
if (w1>>28)==5:
|
||||
off=w1&MASK
|
||||
by_op[c][off]+=1
|
||||
if off < D: inrange[c]+=1
|
||||
else: outrange[c]+=1
|
||||
if len(samples[c])<6 and off>=D: samples[c].append((base,hex(o),hex(w0),hex(w1)))
|
||||
o+=8
|
||||
|
||||
print("opcode -> (in-range count, out-of-range count):")
|
||||
for c in sorted(set(list(inrange)+list(outrange))):
|
||||
print(f" 0x{c:02X}: in={inrange[c]} out={outrange[c]} distinct_vals={len(by_op[c])}")
|
||||
print()
|
||||
for c,v in samples.items():
|
||||
print(f"samples opcode 0x{c:02X} (file, slot_off, w0, w1):")
|
||||
for s in v: print(" ", s)
|
||||
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Analyze G_VTX w1 seg-5 usage: do offsets match vertex-array file offsets?
|
||||
Also: quantify trailing junk after ENDDL within each GDL span (for the converter's
|
||||
packing decision), and per-opcode record counts across all 512 files."""
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
rom = open('data/ge007.ntsc-final.z64','rb').read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b,o): return struct.unpack_from(">I",b,o)[0] & 0xFFFFFF
|
||||
def be32r(b,o): return struct.unpack_from(">I",b,o)[0]
|
||||
|
||||
seg5_offs = Counter()
|
||||
matched = 0; unmatched = []; total_seg5 = 0
|
||||
files_with_seg5 = set()
|
||||
opcounts = Counter()
|
||||
junk_total = 0; junk_files = 0; tight = 0; loose = 0
|
||||
nsz_bad = []
|
||||
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
try: out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
except Exception: continue
|
||||
D=len(out); R0=4*NS+12*NT
|
||||
seen=set(); stack=[R0]; vtoffs=set(); gdls=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D: continue
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4)
|
||||
opcounts[op]+=1
|
||||
if op==4:
|
||||
vtoffs.add(be32(out,data+0xC))
|
||||
gdls += [x for x in (be32(out,data), be32(out,data+4)) if x]
|
||||
elif op==24:
|
||||
vtoffs.add(be32(out,data+8)); vtoffs.add(be32(out,data+0x10))
|
||||
gdls += [x for x in (be32(out,data), be32(out,data+4)) if x]
|
||||
elif op==22:
|
||||
vtoffs.add(be32(out,data+4))
|
||||
p=be32(out,data+8)
|
||||
if p: gdls.append(p)
|
||||
for q in (be32(out,o+0xC), be32(out,o+0x14)):
|
||||
if q: stack.append(q)
|
||||
# GDL spans in visit order = sorted unique offsets
|
||||
gs = sorted(set(gdls))
|
||||
for i,g in enumerate(gs):
|
||||
end = gs[i+1] if i+1 < len(gs) else D
|
||||
n=0; en=None
|
||||
o=g
|
||||
while o+8<=end:
|
||||
w0=be32r(out,o); c=w0>>24
|
||||
if c==0x04 and ((w1:=be32r(out,o+4))>>24)&0xf == 5:
|
||||
off = w1 & 0xFFFFFF
|
||||
seg5_offs[off]+=1; total_seg5+=1
|
||||
files_with_seg5.add(base)
|
||||
if off in vtoffs: matched+=1
|
||||
elif len(unmatched)<10: unmatched.append((base, hex(w1), sorted(vtoffs)[:6]))
|
||||
if c==0xB8: en=o+8; break
|
||||
o+=8; n+=1
|
||||
if en is None:
|
||||
junk_total += (end-g)
|
||||
elif en < end:
|
||||
loose += 1; junk_total += end-en
|
||||
else:
|
||||
tight += 1
|
||||
# N64 record size check for op5/op11 trailing bytes
|
||||
# (sizes already known from RSZ map; skip here)
|
||||
|
||||
print("G_VTX seg5 total:", total_seg5, "in files:", len(files_with_seg5))
|
||||
print("matched to a vertex-array offset in same file:", matched)
|
||||
print("unmatched samples:", unmatched[:6])
|
||||
print("seg5 offset histogram (top 10):", [(hex(k),v) for k,v in seg5_offs.most_common(10)])
|
||||
print()
|
||||
print("GDL span tightness: tight=%d loose=%d total_trailing_bytes=%d" % (tight, loose, junk_total))
|
||||
print()
|
||||
print("opcode record counts across all files:")
|
||||
for op in sorted(opcounts): print(f" op{op}: {opcounts[op]}")
|
||||
@@ -0,0 +1,109 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seg-5 G_VTX analysis WITH LOD/SWITCH rewiring (as in d43_gdlorder.py).
|
||||
Question: does every seg-5 G_VTX offset point INSIDE a vertex array of the same file?
|
||||
(vo + d where d % 16 == 0 and d < 16*nv) — i.e. is it (array_base_file_offset + displacement)?"""
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
rom = open('data/ge007.ntsc-final.z64','rb').read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b,o): return struct.unpack_from(">I",b,o)[0] & 0xFFFFFF
|
||||
def be32r(b,o): return struct.unpack_from(">I",b,o)[0]
|
||||
|
||||
total_seg5 = 0; inside = 0; outside = []
|
||||
files_with = set()
|
||||
vtx_regions = Counter() # (d mod 16) histogram of displacement
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
try: out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
except Exception: continue
|
||||
D=len(out); R0=4*NS+12*NT
|
||||
|
||||
# walk with LOD/SWITCH rewiring; collect vertex regions (vo, nv) and GDL offsets per record
|
||||
seen=set(); stack=[R0]; regions=[]; gdls=[]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D: continue
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4)
|
||||
child = be32(out,o+0x14); nxt = be32(out,o+0xC)
|
||||
if op==8: # LOD: Child := Affects (data+8)
|
||||
a = be32(out,data+8)
|
||||
if a: stack.append(a)
|
||||
elif op==18: # SWITCH: Child := Controls (data+0)
|
||||
c = be32(out,data)
|
||||
if c: stack.append(c)
|
||||
else:
|
||||
if child: stack.append(child)
|
||||
if nxt: stack.append(nxt)
|
||||
if op==4:
|
||||
nv=struct.unpack_from(">H",out,data+0x10)[0]
|
||||
vo=be32(out,data+0xC)
|
||||
regions.append((vo,nv))
|
||||
gdls += [x for x in (be32(out,data), be32(out,data+4)) if x]
|
||||
elif op==24:
|
||||
nv=struct.unpack_from(">h",out,data+0xC)[0]
|
||||
cv=struct.unpack_from(">h",out,data+0xE)[0]
|
||||
vo=be32(out,data+8); cvo=be32(out,data+0x10)
|
||||
regions.append((vo,nv))
|
||||
if cv: regions.append((cvo,cv))
|
||||
gdls += [x for x in (be32(out,data), be32(out,data+4)) if x]
|
||||
elif op==22:
|
||||
nv=struct.unpack_from(">i",out,data)[0]
|
||||
vo=be32(out,data+4)
|
||||
regions.append((vo,nv))
|
||||
p=be32(out,data+8)
|
||||
if p: gdls.append(p)
|
||||
|
||||
gs = sorted(set(gdls))
|
||||
for i,g in enumerate(gs):
|
||||
end = gs[i+1] if i+1 < len(gs) else D
|
||||
o=g
|
||||
while o+8<=end:
|
||||
w0=be32r(out,o); c=w0>>24
|
||||
if c==0x04:
|
||||
w1=be32r(out,o+4)
|
||||
if (w1>>24)&0xf == 5:
|
||||
off = w1 & 0xFFFFFF
|
||||
total_seg5 += 1; files_with.add(base)
|
||||
hit=False
|
||||
for vo,nv in regions:
|
||||
d = off - vo
|
||||
if 0 <= d < 16*nv and nv>0:
|
||||
vtx_regions[d & 0xf]+=1
|
||||
hit=True; break
|
||||
if hit: inside+=1
|
||||
elif len(outside)<12: outside.append((base, hex(off), [(hex(a),b) for a,b in regions[:8]]))
|
||||
o+=8
|
||||
|
||||
print("seg5 G_VTX total:", total_seg5, "files:", len(files_with))
|
||||
print("pointing inside a vertex array (vo+d, d<16nv):", inside)
|
||||
print("displacement mod 16 histogram:", dict(vtx_regions))
|
||||
print("outside samples:")
|
||||
for s in outside: print(" ", s)
|
||||
@@ -0,0 +1,125 @@
|
||||
# D43: PC image size budget (final).
|
||||
#
|
||||
# Converter output layout (PC image, little-endian):
|
||||
# [switches NS*8][textures NT*12][nodes 48B + records pc-size, DFS preorder]
|
||||
# [bulk arrays: vertex/coll/pointusage, verbatim sizes, s16/f32 bswapped]
|
||||
# [GDLs at tail in visit order, 16B Gfx slots, no inter-GDL padding]
|
||||
#
|
||||
# sub_GAME_7F0762E0 mirror+expand contract:
|
||||
# D = pc_remaining (PC image size), g1 = first GDL offset, R = buffer.
|
||||
# texCopyGdls mirrors [g1,D) to [R-D+g1,R); texLoadFromGdl expands each GDL
|
||||
# rightward from g1 reading the mirror. Each N64 8B slot becomes one PC 16B
|
||||
# slot, so the GDL region doubles: input extent of GDL k = 2*E_k (E_k = N64
|
||||
# extent bytes). Fit and no-clobber both reduce to:
|
||||
# R >= P_final = B_pc + sum(2*E_k) + 16 * sum_markers(K_t - 1)
|
||||
# K_t = worst-case output slots a marker of type t emits (tex.c analysis;
|
||||
# only types 2/3/4 occur in ROM):
|
||||
RSZ_N64 = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28,
|
||||
13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:2, 24:0x20}
|
||||
RSZ_PC = {1:0x18, 2:0x28, 4:0x28, 8:0x18, 9:0x30, 10:0x1C, 12:0x30,
|
||||
13:0x30, 15:0x1C, 17:0x28, 18:0x10, 21:0x14, 22:0x20, 23:2, 24:0x40}
|
||||
NODE_N64, NODE_PC = 24, 48
|
||||
# exact worst-case output slots per marker (tex.c helpers, maxlod<=8,
|
||||
# all texTry* state guards emitting, valid=FALSE branch, water check on):
|
||||
# preamble = PipeSync(<=1) + texWriteTextureCmd(<=1) = 2
|
||||
# LoadToTmemAddr/Zero <= 9, TileFromDef <= 4, TileLods <= 3*maxlod,
|
||||
# Tile <= 3, light begin/end = 0, water sub = 1
|
||||
K_BOUND = {0:2+9+4+3*7+1, 1:2+9+1+9+4+3*7+1, 2:2+9+3*8+1, 3:2+9+3+3+1, 4:2+9+3+1}
|
||||
R_BUF = 0x14820 # size_item_buffer (gun.c), shared body/head/props per hand
|
||||
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
|
||||
def gdl_marker_slots(out, g, e):
|
||||
# count texture markers (G_NOOP w1 type field) in [g,e); returns {type:count}
|
||||
c = Counter()
|
||||
j = g
|
||||
while j + 8 <= e:
|
||||
w0 = struct.unpack_from(">I", out, j)[0]
|
||||
if (w0 >> 24) == 0xC0:
|
||||
t = w0 & 7
|
||||
c[t] += 1
|
||||
j += 8
|
||||
return c
|
||||
|
||||
stats = []
|
||||
nfiles = 0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr, size = int(r[0]), int(r[1])
|
||||
except ValueError: continue
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key = cand; break
|
||||
if not key: continue
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: continue
|
||||
D = len(out); NS, NT = nsnt[key]
|
||||
R0n = 4*NS + 12*NT
|
||||
# walk nodes (DFS preorder), collect records + GDLs
|
||||
stack = [R0n]; seen = set(); nodes = []; gdls = []
|
||||
while stack:
|
||||
o = stack.pop()
|
||||
if o in seen or o >= D-23: break
|
||||
seen.add(o)
|
||||
op = struct.unpack_from(">H", out, o)[0] & 0xff
|
||||
data = be32(out, o+4)
|
||||
nodes.append((o, op, data))
|
||||
if op in (4, 24):
|
||||
p, s = be32(out, data), be32(out, data+4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
elif op == 22:
|
||||
p = be32(out, data+8)
|
||||
if p: gdls.append(p)
|
||||
for q in (be32(out, o+0x14), be32(out, o+0xC)):
|
||||
if q: stack.append(q)
|
||||
gsorted = sorted(set(gdls))
|
||||
g1 = gsorted[0] if gsorted else D
|
||||
# bulk arrays (vertex/coll/pointusage + alignment padding) keep N64 byte
|
||||
# counts on PC: 16B/vertex, 16B/coll vertex, 2B/PU entry.
|
||||
bulk = (g1 - R0n) - len(nodes)*NODE_N64 - sum(RSZ_N64[op] for _, op, _ in nodes)
|
||||
B_pc = 8*NS + 12*NT + len(nodes)*NODE_PC + sum(RSZ_PC[op] for _, op, _ in nodes) + bulk
|
||||
G_in = 0; Kslack = 0
|
||||
for i, g in enumerate(gsorted):
|
||||
e = gsorted[i+1] if i+1 < len(gsorted) else D
|
||||
assert (e - g) % 8 == 0, (base, hex(g), e - g)
|
||||
G_in += 2 * (e - g) # each N64 8B slot -> one PC 16B slot
|
||||
for t, n in gdl_marker_slots(out, g, e).items():
|
||||
Kslack += n * (K_BOUND.get(t, 44) - 1)
|
||||
P_final = B_pc + G_in + 16*Kslack
|
||||
stats.append((base, D, B_pc, G_in, 16*Kslack, P_final))
|
||||
nfiles += 1
|
||||
|
||||
print("files:", nfiles)
|
||||
worst = sorted(stats, key=lambda s: -s[5])[:15]
|
||||
for b, D, Bp, Gi, K, Pf in worst:
|
||||
flag = "OVER" if Pf > R_BUF else " "
|
||||
print("%-22s N64=%7X B_pc=%7X G_in=%7X Kslack=%7X P_final=%8X %s (R=%X)"
|
||||
% (b, D, Bp, Gi, K, Pf, flag, R_BUF))
|
||||
over = [s for s in stats if s[5] > R_BUF]
|
||||
print("files over budget:", len(over), "of", nfiles)
|
||||
totD = sum(s[1] for s in stats); totP = sum(s[5] for s in stats)
|
||||
print("sum N64 bytes: %X sum P_final: %X" % (totD, totP))
|
||||
@@ -0,0 +1,125 @@
|
||||
# D43: PC image size budget (final).
|
||||
#
|
||||
# Converter output layout (PC image, little-endian):
|
||||
# [switches NS*8][textures NT*12][nodes 48B + records pc-size, DFS preorder]
|
||||
# [bulk arrays: vertex/coll/pointusage, verbatim sizes, s16/f32 bswapped]
|
||||
# [GDLs at tail in visit order, 16B Gfx slots, no inter-GDL padding]
|
||||
#
|
||||
# sub_GAME_7F0762E0 mirror+expand contract:
|
||||
# D = pc_remaining (PC image size), g1 = first GDL offset, R = buffer.
|
||||
# texCopyGdls mirrors [g1,D) to [R-D+g1,R); texLoadFromGdl expands each GDL
|
||||
# rightward from g1 reading the mirror. Each N64 8B slot becomes one PC 16B
|
||||
# slot, so the GDL region doubles: input extent of GDL k = 2*E_k (E_k = N64
|
||||
# extent bytes). Fit and no-clobber both reduce to:
|
||||
# R >= P_final = B_pc + sum(2*E_k) + 16 * sum_markers(K_t - 1)
|
||||
# K_t = worst-case output slots a marker of type t emits (tex.c analysis;
|
||||
# only types 2/3/4 occur in ROM):
|
||||
RSZ_N64 = {1:0x10, 2:0x1C, 4:0x14, 8:0x10, 9:0x24, 10:0x1C, 12:0x28,
|
||||
13:0x20, 15:0x1C, 17:0x20, 18:0x8, 21:0x14, 22:0x10, 23:2, 24:0x20}
|
||||
RSZ_PC = {1:0x18, 2:0x28, 4:0x28, 8:0x18, 9:0x30, 10:0x1C, 12:0x30,
|
||||
13:0x30, 15:0x1C, 17:0x28, 18:0x10, 21:0x14, 22:0x20, 23:2, 24:0x40}
|
||||
NODE_N64, NODE_PC = 24, 48
|
||||
# exact worst-case output slots per marker (tex.c helpers, maxlod<=8,
|
||||
# all texTry* state guards emitting, valid=FALSE branch, water check on):
|
||||
# preamble = PipeSync(<=1) + texWriteTextureCmd(<=1) = 2
|
||||
# LoadToTmemAddr/Zero <= 9, TileFromDef <= 4, TileLods <= 3*maxlod,
|
||||
# Tile <= 3, light begin/end = 0, water sub = 1
|
||||
K_BOUND = {0:2+9+4+3*7+1, 1:2+9+1+9+4+3*7+1, 2:2+9+3*8+1, 3:2+9+3+3+1, 4:2+9+3+1}
|
||||
R_BUF = 0x14820 # size_item_buffer (gun.c), shared body/head/props per hand
|
||||
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
|
||||
def gdl_marker_slots(out, g, e):
|
||||
# count texture markers (G_NOOP w1 type field) in [g,e); returns {type:count}
|
||||
c = Counter()
|
||||
j = g
|
||||
while j + 8 <= e:
|
||||
w0 = struct.unpack_from(">I", out, j)[0]
|
||||
if (w0 >> 24) == 0xC0:
|
||||
t = w0 & 7
|
||||
c[t] += 1
|
||||
j += 8
|
||||
return c
|
||||
|
||||
stats = []
|
||||
nfiles = 0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/", 1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr, size = int(r[0]), int(r[1])
|
||||
except ValueError: continue
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key = cand; break
|
||||
if not key: continue
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: continue
|
||||
D = len(out); NS, NT = nsnt[key]
|
||||
R0n = 4*NS + 12*NT
|
||||
# walk nodes (DFS preorder), collect records + GDLs
|
||||
stack = [R0n]; seen = set(); nodes = []; gdls = []
|
||||
while stack:
|
||||
o = stack.pop()
|
||||
if o in seen or o >= D-23: break
|
||||
seen.add(o)
|
||||
op = struct.unpack_from(">H", out, o)[0] & 0xff
|
||||
data = be32(out, o+4)
|
||||
nodes.append((o, op, data))
|
||||
if op in (4, 24):
|
||||
p, s = be32(out, data), be32(out, data+4)
|
||||
if p: gdls.append(p)
|
||||
if s: gdls.append(s)
|
||||
elif op == 22:
|
||||
p = be32(out, data+8)
|
||||
if p: gdls.append(p)
|
||||
for q in (be32(out, o+0x14), be32(out, o+0xC)):
|
||||
if q: stack.append(q)
|
||||
gsorted = sorted(set(gdls))
|
||||
g1 = gsorted[0] if gsorted else D
|
||||
# bulk arrays (vertex/coll/pointusage + alignment padding) keep N64 byte
|
||||
# counts on PC: 16B/vertex, 16B/coll vertex, 2B/PU entry.
|
||||
bulk = (g1 - R0n) - len(nodes)*NODE_N64 - sum(RSZ_N64[op] for _, op, _ in nodes)
|
||||
B_pc = 8*NS + 12*NT + len(nodes)*NODE_PC + sum(RSZ_PC[op] for _, op, _ in nodes) + bulk
|
||||
G_in = 0; Kslack = 0
|
||||
for i, g in enumerate(gsorted):
|
||||
e = gsorted[i+1] if i+1 < len(gsorted) else D
|
||||
assert (e - g) % 8 == 0, (base, hex(g), e - g)
|
||||
G_in += 2 * (e - g) # each N64 8B slot -> one PC 16B slot
|
||||
for t, n in gdl_marker_slots(out, g, e).items():
|
||||
Kslack += n * (K_BOUND.get(t, 44) - 1)
|
||||
P_final = B_pc + G_in + 16*Kslack
|
||||
R_need2 = B_pc + G_in + 32*Kslack
|
||||
stats.append((base, D, B_pc, G_in, 16*Kslack, P_final, Kslack, R_need2))
|
||||
nfiles += 1
|
||||
|
||||
print("files:", nfiles)
|
||||
worst = sorted(stats, key=lambda s: -s[5])[:15]
|
||||
for b, D, Bp, Gi, K, Pf, Ks, R2 in worst:
|
||||
print("%-22s N64=%7X B_pc=%7X G_in=%7X Mtot=%5d P_final=%8X R_need2=%8X" % (b, D, Bp, Gi, Ks, Pf, R2))
|
||||
over = [s for s in stats if s[5] > R_BUF]
|
||||
print("max 16*M_total:", max(s[7]-s[5] for s in stats), "hex %X" % max(s[7]-s[5] for s in stats))
|
||||
print("files over budget:", len(over), "of", nfiles)
|
||||
totD = sum(s[1] for s in stats); totP = sum(s[5] for s in stats)
|
||||
print("sum N64 bytes: %X sum P_final: %X" % (totD, totP))
|
||||
@@ -0,0 +1,48 @@
|
||||
import csv, struct, sys, zlib, os, re
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"): hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
def be32(b, o): return struct.unpack_from(">I", b, o)[0] & 0xFFFFFF
|
||||
flt = sys.argv[1]
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if flt.lower() not in base.lower(): continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
addr,size=int(r[0]),int(r[1])
|
||||
out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
D=len(out); R0=4*NS+12*NT
|
||||
print("== %s D=0x%X NS=%d NT=%d R0=0x%X" % (base,D,NS,NT,R0))
|
||||
seen=set()
|
||||
def show(o, depth):
|
||||
if o in seen or o>=D: return
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4); par=be32(out,o+8); nxt=be32(out,o+0xC); prv=be32(out,o+0x10); ch=be32(out,o+0x14)
|
||||
print(" %snode op=%-2d @%05X data=%05X par=%05X next=%05X prev=%05X child=%05X" % (" "*depth, op, o, data, par, nxt, prv, ch))
|
||||
if data < D:
|
||||
# print first 12 words of the record
|
||||
ws=[be32(out,data+4*k) for k in range(min(6,(D-data)//4))]
|
||||
print(" rec: "+ " ".join("%06X"%w for w in ws))
|
||||
if ch: show(ch, depth+1)
|
||||
if nxt: show(nxt, depth+1)
|
||||
show(R0, 0)
|
||||
break
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Verify collision-vertex format: @8 is 0x05xxxxxx-or-0 (LinkedTo node ptr),
|
||||
@C is a small s16 index. Also verify normal-vertex arrays never have 0x05xxxxxx at @8."""
|
||||
import csv, struct, zlib, os, re
|
||||
from collections import Counter
|
||||
|
||||
rom = open('data/ge007.ntsc-final.z64','rb').read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
try: nsnt[a[0]] = (int(a[4], 0), int(a[8], 0))
|
||||
except ValueError: pass
|
||||
|
||||
def be32(b,o): return struct.unpack_from(">I",b,o)[0] & 0xFFFFFF
|
||||
def be32r(b,o): return struct.unpack_from(">I",b,o)[0]
|
||||
|
||||
coll_bad8 = Counter(); coll_idx_range = [1<<30, -1<<30]
|
||||
norm_weird8 = 0; norm_total = 0; coll_total = 0
|
||||
files_checked = 0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
key=None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt: key=cand; break
|
||||
if not key: continue
|
||||
NS,NT = nsnt[key]
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
try: out=zlib.decompress(rom[addr:addr+size][2:],-15)
|
||||
except Exception: continue
|
||||
D=len(out); R0=4*NS+12*NT
|
||||
files_checked += 1
|
||||
seen=set(); stack=[R0]
|
||||
while stack:
|
||||
o=stack.pop()
|
||||
if o in seen or o>=D: continue
|
||||
seen.add(o)
|
||||
op=struct.unpack_from(">H",out,o)[0]&0xff
|
||||
data=be32(out,o+4)
|
||||
child = be32(out,o+0x14); nxt = be32(out,o+0xC)
|
||||
if op==8:
|
||||
a = be32(out,data+8)
|
||||
if a: stack.append(a)
|
||||
elif op==18:
|
||||
c = be32(out,data)
|
||||
if c: stack.append(c)
|
||||
else:
|
||||
if child: stack.append(child)
|
||||
if nxt: stack.append(nxt)
|
||||
if op==4 or op==22:
|
||||
if op==4:
|
||||
nv=struct.unpack_from(">H",out,data+0x10)[0]; vo=be32(out,data+0xC)
|
||||
else:
|
||||
nv=struct.unpack_from(">i",out,data)[0]; vo=be32(out,data+4)
|
||||
for i in range(nv):
|
||||
v = vo + 16*i
|
||||
w8 = be32r(out, v+8)
|
||||
norm_total += 1
|
||||
if w8 != 0 and (w8 >> 24) != 5:
|
||||
norm_weird8 += 1
|
||||
elif op==24:
|
||||
nv=struct.unpack_from(">h",out,data+0xC)[0]
|
||||
ncv=struct.unpack_from(">h",out,data+0xE)[0]
|
||||
vo=be32(out,data+8); cvo=be32(out,data+0x10)
|
||||
for i in range(nv):
|
||||
v = vo + 16*i
|
||||
w8 = be32r(out, v+8)
|
||||
norm_total += 1
|
||||
if w8 != 0 and (w8 >> 24) != 5:
|
||||
norm_weird8 += 1
|
||||
for i in range(ncv):
|
||||
v = cvo + 16*i
|
||||
w8 = be32r(out, v+8)
|
||||
idx = struct.unpack_from(">h", out, v+0xC)[0]
|
||||
coll_total += 1
|
||||
if w8 != 0 and (w8 >> 24) != 5:
|
||||
coll_bad8[w8 & 0xFFFFFF] += 1
|
||||
coll_idx_range[0] = min(coll_idx_range[0], idx)
|
||||
coll_idx_range[1] = max(coll_idx_range[1], idx)
|
||||
|
||||
print("files:", files_checked, " normal vtx:", norm_total, " weird @8 (non-0/non-seg5):", norm_weird8)
|
||||
print("collision vtx:", coll_total, " bad @8 values:", dict(list(coll_bad8.items())[:10]))
|
||||
print("CollisionRelatedIndex range:", coll_idx_range)
|
||||
@@ -0,0 +1,97 @@
|
||||
# D43 helper: walk every model file's node tree (N64 layout) and report
|
||||
# opcode histogram + structural sanity. Usage: python d43_walk.py
|
||||
import csv, struct, zlib, sys
|
||||
from collections import Counter
|
||||
|
||||
ROM = r"data/ge007.ntsc-final.z64"
|
||||
rom = open(ROM, "rb").read()
|
||||
rows = list(csv.reader(open(r"scripts/filelist.u.csv")))
|
||||
|
||||
OP={1:"HEADER",2:"GROUP",3:"OP03",4:"DL",5:"OP05",6:"OP06",7:"OP07",8:"LOD",9:"BSP",10:"BOX",11:"OP11",12:"GUNFIRE",13:"SHADOW",14:"OP14",15:"INTERLINK",16:"OP16",17:"OP17",18:"SWITCH",21:"GROUPSIMPLE",22:"DLPRIMARY",23:"HEADPH",24:"DLCOLLISION"}
|
||||
|
||||
# N64 record sizes (bytes) per opcode, from bondtypes.h
|
||||
RSZ={1:0x10,2:0x1C,3:0x1C,4:0x14,5:0x1AC,6:0x18,7:0x1B0,8:0x10,9:0x24,10:0x1C,
|
||||
11:0x50,12:0x28,13:0x20,14:0x10,15:0x1C,16:0x18,17:0x20,18:0x08,21:0x14,
|
||||
22:0x10,23:0x02,24:0x20}
|
||||
|
||||
# (numSwitches, numtextures) per model from the exe headers would be ideal;
|
||||
# instead parse assets/obseg/**/ModelFileHeader.inc.c + body records.
|
||||
import re, glob
|
||||
nsnt = {}
|
||||
import os
|
||||
hdrfiles = set()
|
||||
for root, dirs, files in os.walk("assets"):
|
||||
for fn in files:
|
||||
if fn.lower().endswith("modelfileheader.inc.c"):
|
||||
hdrfiles.add(os.path.join(root, fn))
|
||||
nsnt = {}
|
||||
for f in sorted(hdrfiles):
|
||||
for line in open(f):
|
||||
m = re.search(r"MODELFILEHEADER\((.*)\)\s*;?\s*$", line)
|
||||
if not m: continue
|
||||
a = [x.strip() for x in m.group(1).split(",")]
|
||||
if len(a) < 9: continue
|
||||
name = a[0]
|
||||
try:
|
||||
NS = int(a[4], 0); NT = int(a[8], 0)
|
||||
except ValueError:
|
||||
continue
|
||||
nsnt[name] = (NS, NT)
|
||||
|
||||
# body/chr records live in other .inc.c files with the same macro? check c_item_entries etc.
|
||||
print("headers parsed:", len(nsnt))
|
||||
|
||||
opcount = Counter()
|
||||
files_ok = 0
|
||||
files_bad = []
|
||||
total_nodes = 0
|
||||
maxP = 0
|
||||
for r in rows:
|
||||
if len(r) < 3 or not r[2]: continue
|
||||
base = r[2].rsplit("/",1)[-1]
|
||||
if base.endswith(".bin"): base = base[:-4]
|
||||
if not (base[0] in "CGP" and base.endswith("Z") and "_stan" not in base): continue
|
||||
try: addr,size=int(r[0]),int(r[1])
|
||||
except ValueError: continue
|
||||
try: out = zlib.decompress(rom[addr:addr+size][2:], -15)
|
||||
except Exception: files_bad.append((base,"decomp")); continue
|
||||
D = len(out)
|
||||
key = None
|
||||
for cand in (base, base[1:], base[:-1], base[1:-1]):
|
||||
if cand in nsnt:
|
||||
key = cand; break
|
||||
NS,NT = nsnt.get(key, (0,0)) if key else (0,0)
|
||||
if NT is None:
|
||||
# try to find NT from a record table elsewhere; fallback scan
|
||||
files_bad.append((base,"no-NT"))
|
||||
continue
|
||||
root_off = 4*NS + 12*NT
|
||||
def be16(o): return struct.unpack_from(">H", out, o)[0]
|
||||
def be32(o): return struct.unpack_from(">I", out, o)[0] & 0xFFFFFF
|
||||
seen = set()
|
||||
ok = True
|
||||
n = 0
|
||||
stack = [root_off] if root_off < D else []
|
||||
while stack:
|
||||
o = stack.pop()
|
||||
if o in seen or o >= D: ok=False; break
|
||||
seen.add(o)
|
||||
op = be16(o) & 0xff
|
||||
opcount[op] += 1
|
||||
n += 1
|
||||
data = be32(o+4); child = be32(o+20)
|
||||
if not (0 < data < D): ok=False; break
|
||||
# validate record fits
|
||||
rsz = RSZ.get(op, 0)
|
||||
if rsz and data + rsz > D: ok=False; files_bad.append((base,"rec-overflow op%d @%X+%X>%X"%(op,data,rsz,D))); break
|
||||
if child: stack.append(child)
|
||||
total_nodes += n
|
||||
if ok: files_ok += 1
|
||||
else: files_bad.append((base,"walk-fail root=%X NS=%d NT=%d D=%X"%(root_off,NS,NT,D)))
|
||||
|
||||
print("files ok:", files_ok, " bad:", len(files_bad))
|
||||
for b in files_bad[:20]: print(" ", b)
|
||||
print("total nodes:", total_nodes)
|
||||
print("opcode histogram:")
|
||||
for op in sorted(opcount):
|
||||
print(" op%-3d %-12s x%d" % (op, OP.get(op,"??"), opcount[op]))
|
||||
Reference in new issue
Block a user