diff --git a/.gitignore b/.gitignore index caa7137b..f8255955 100644 --- a/.gitignore +++ b/.gitignore @@ -136,3 +136,6 @@ ge007.crash.log # Claude Code local state / agent worktrees .claude/ + +# saved reference webpage assets (keep the .htm, not the sidecar cruft) +docs/reference/*_files/ diff --git a/AGENTS.md b/AGENTS.md index e1b14223..92bb0247 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,6 +6,7 @@ - Active work: **PC port** modelled on the Perfect Dark PC port (same Rare "Indy" engine family). - **Read `docs/PCPortResearch.md` first** — architecture, GE-specific RSP deltas, phased plan, and review findings (A1–D33 in §11; current handoff + plan in §H). - **Starting a fresh session?** Read `docs/HANDOFF.md` — the paste-ready brief with the immediate task, standing procedure, and environment gotchas. +- **Dispatching subagents?** `docs/AGENT-WORKFLOW.md` — task budgets/deadlines, file partitioning, pre-flight, the standard brief template. Every investigation subagent reads `docs/PORT-LEARNINGS.md` (recurring bug classes) first and appends to it. ## Non-negotiables diff --git a/docs/AGENT-WORKFLOW.md b/docs/AGENT-WORKFLOW.md new file mode 100644 index 00000000..ae485e9c --- /dev/null +++ b/docs/AGENT-WORKFLOW.md @@ -0,0 +1,131 @@ +# Agent / subagent workflow for this port + +Guidance distilled from Chris Lewis's *"Decompiling a Nintendo 64 Game +in 84 Days"* (`docs/reference/`, Snowboard Kids, 2026) — an AI-assisted +N64 decomp run 7× faster than its predecessor — adapted to this project, +which is a **PC port on top of a decomp**, not a byte-match decomp. The +mechanics differ (we have no permuter / m2c loop; our inner loop is +root-cause → narrow `#ifdef PORT` fix → visual verify) but the +project-structure lessons transfer directly. + +## 1. Every subagent task gets an explicit budget, stated in the brief + +Their key finding: an open-ended tool (the permuter) that "runs until +100% match or manually stopped" wrecks throughput; giving the agent a +**deadline it can see** let it trade tool-time against other +problem-solving and judge when to give up on a hard function. + +Our open-ended tools are the same shape: rebuild-and-run-the-game cycles, +gdb, env-gated probe iteration, staring at PPM dumps. So every subagent +brief must state: + +- a **wall/iteration budget** — e.g. "≤ N build→run→inspect cycles" or + "~M minutes of investigation"; and +- the **fallback on expiry**: *stop, revert probes, write up what you + have with a confidence rating* — a good write-up of a half-solved bug + is a deliverable, not a failure. + +Escalate the budget as the project hardens: early bugs are cheap +(one-line truncations, off-by-ones), so cap tight; once those are gone +the survivors are structural (matrix handedness, format specs) and +deserve more room. Revisit the default each session. + +## 2. A shared, append-only learnings file that briefs point every agent at + +They had agents record generalisable IDO quirks in +`DECOMPILATION_LEARNINGS.md`; later agents read it and got better. "Agents +helped document IDO, and the documentation made subsequent agents better." + +Our equivalent already exists but is diffuse: `docs/PCPortResearch.md` §F +is 200 KB of narrative. The **recurring PC-port quirk classes** should be +distilled into a short cheat-sheet every subagent brief links and every +subagent is told to append to: + +- pointer-width struct growth (32→64) in ROM-serialized / pun-allocated + structs — the dominant bug class (D53.2, D56, D79, D98–D102) +- 16-byte PC `Gfx`/`Vtx` vs 8-byte N64 — reservation & copy sizing + (D50.6, D58, D85, D95) +- big-endian rodata read on LE: `f32` word-pairs (D73, D112), header + offset tables (D68, D87), packed bitfields (D78, D83) +- N64 hardware idioms fast3d doesn't emulate (fill-rect Z clear → D105; + LOD/detail tiles → D107; K0 segment address folds → D58, D84) +- offline sidecar converter preferred over runtime fixup for whole + formats (D43, D69, D88) + +This is `docs/PORT-LEARNINGS.md` — one screen, an index into §F, not a +replacement. Every investigation-subagent brief links it; every subagent +is told to append any new generalisable quirk before reporting. + +## 3. Parallel work: partition by file, consolidate often + +They ran 4 git worktrees sharded by a hash of the candidate list, and +found the cost was **synchronisation** — a match in one worktree stayed +invisible to the others until merged, and merging all four took >1 h. +Their fix: make the similarity search read *all* worktrees so a new +reference propagates immediately, and merge to main periodically anyway. + +For us (parallel investigation subagents, not worktrees yet): + +- **Partition tracks by the files they will touch** so patches don't + collide. This session: Track 1 = `bg.c`, Track 2 = `chr.c`/`model*.c`, + Track 3 = `gunfire.c` + a new audit doc — disjoint, merged clean. + A brief that can't be file-partitioned shouldn't be parallelised. +- Subagents must **revert their own probes** before reporting (leave only + committed, env-gated, capped probes) and must **not** touch another + track's uncommitted edits — call them out in the report instead. +- The integrator (main session) consolidates to `master` per-track with + the §F write-up, then the next round of agents starts from a clean + tree. Don't let investigation branches drift. +- If we move to real worktrees: `data/` must be **copied, never + junctioned** (see the `data-dir-junction-hazard` memory — + `git worktree remove --force` follows the junction and deletes the real + baserom + sidecars). + +## 4. Cheap automated pass before spending agent tokens + +They scripted `m2c` across every unmatched function first (0.93 % hit +rate, but near-free) and told agents to exhaust known SDK +versions / compiler flags / `#if` paths before writing original code. + +Our pre-flight before dispatching an investigation agent: + +- `./build-pc.sh` green + `/linkcheck` clean +- run the existing env-gated probes for the area (`GE_D96`, `GE_D104`, + `GE_D69*`, …) and attach the output to the brief +- `GE_PCDUMP` + `tools_pc/pixcount.py` baseline capture +- grep `docs/PCPortResearch.md` §F for a prior instance of the bug class +- check `PD_PORT_CHECKOUT` for the PD analogue + (already a non-negotiable) — the brief should name the file to look at + +Give the agent the results, not just the task. "Give the agent the +findings, not just the problem" is the through-line of the whole post. + +## 5. What does *not* transfer + +- **Model choice.** Their (unscientific) take was Codex > Claude for + reproducing IDO codegen. Our task is behavioural correctness, not + compiler-output matching — not comparable; ignore. +- **The permuter / match-percentage loop.** We don't bit-match. Our + "done" bar is: no fault + visually correct vs. N64 footage + (`docs/reference/n64-footage-*`), measured with `pixcount.py`. +- **"Understanding the function is the easy part."** True for decomp, + inverted here — our hard part is often *what N64 hardware/ABI behaviour + the C is silently relying on*. + +## Standard investigation-subagent brief template + +``` +Repo: REPO_ROOT (git, master, Windows; PowerShell + Bash). +READ FIRST: docs/HANDOFF.md (top), docs/PORT-LEARNINGS.md, + docs/PCPortResearch.md §F , AGENTS.md non-negotiables. +TASK: . +FILES YOU MAY TOUCH: . +KNOWN-GOOD / RULED OUT: . +PRE-FLIGHT ATTACHED: . +BUDGET: . On expiry: revert probes, write up with confidence. +CONSTRAINTS: no game-logic changes; ABI/layout/format only, #ifdef PORT, documented in §F. + revert your own temp probes; do not touch other agents' uncommitted edits. +REPORT: (a) root cause + file:line evidence (b) fix diff or why-not + (c) probes left in tree (d) confidence. Append any generalisable + quirk to docs/PORT-LEARNINGS.md. +``` diff --git a/docs/PORT-LEARNINGS.md b/docs/PORT-LEARNINGS.md new file mode 100644 index 00000000..a0698209 --- /dev/null +++ b/docs/PORT-LEARNINGS.md @@ -0,0 +1,74 @@ +# PORT-LEARNINGS — recurring N64→PC bug classes + +One-screen index into `docs/PCPortResearch.md` §F. **Every investigation +subagent reads this first and appends any new generalisable quirk.** +Details, evidence, and the fix for each instance live in §F under the +cited Dxx label. + +## A. Pointer-width struct growth (32→64) — the dominant class + +A decomp struct with pointer fields, or one pun-allocated into a fixed +N64-sized hole / hardcoded byte count, is **larger on x86-64**. Reading it +from ROM bytes misaligns; allocating it N64-sized overruns adjacent +state. + +- Symptom: garbage pointer deref, corrupted neighbour field, delayed + fault far from the cause. +- Fix: store embedded ROM addresses as `u32`, cast at use site (PD ground + truth); or give real inline storage / `sizeof()`-based alloc under + `#ifdef PORT`. +- Instances: D53.2 (ModelSlot pun), D56 (watch Model raw offsets), D57 + (rwdata record count), D67 (image_entry), D79 (bg_room_data), D98 + (struct player alloc), D100 (player.model inline Model), D101/D102 + (ModelNode*/weapon Model puns), D115 (gunfire THROW* raw offsets). +- **Open landmine:** raw hardcoded-offset accessors into `struct player` + / `struct hand` — see `docs/AUDIT-M6-player-offsets.md`. + +## B. 16-byte PC `Gfx` / `Vtx` vs 8-byte N64 + +Any buffer reservation, `memcpy` size, slot stride, or pool budget +expressed in N64 `Gfx`/`Vtx` units is **half-size** on PC. + +- Instances: D50.6 (texCopyGdls copied only w0 of each 16-byte slot), + D58 (DL reserve 0x100→0x200), D85 (`bgWidenRoomGdl` 8→16 + bswap), + D95 (2× master-DL buffer + raised mempool ceiling). + +## C. Big-endian rodata / ROM data read on little-endian PC + +ROM assets and compiled-in `.rodata` are big-endian. Anything not run +through a converter or a runtime bswap fixup reads scrambled. + +- `f32` values are BE **word pairs** — a naive byte reversal off-by-one + corrupts every float: D73 (sinf/cosf `du` pairs → `DVAL()` macro), + D112 (`d43_emit.py put_f32` `src[doff:doff+4][::-1]`). +- Header offset tables / pointers: D54 (cseq ALMidiHdr), D68 + (Globalimagetable), D87 (ramromfilestructure), D88 (Usetup* tables). +- Packed bitfields cross byte boundaries differently: D78 / D83 + (StandTile id/room, header mid/tail). +- **Rule:** prefer an offline sidecar converter (D43, D69, D88 pattern, + `tools_pc/d*_emit.py`) over a runtime fixup for a whole format. + +## D. N64 hardware idioms fast3d does not emulate + +- Z buffer cleared by pointing the colour image at it + fill-rect → does + nothing in fast3d; must emit `G_CLEAR_DEPTH_EXT` (D105). +- LOD / detail mip tiles: fast3d fabricates a crop when detail textures + are off → force base tile (D107). +- K0 segment-address folds (`OS_K0_TO_PHYSICAL | 0x80000000`) and + hand-inlined `BG_SEG_TO_PTR` need the `(u32)` 32-bit wrap the macro has + (D58, D84). +- Portal near-plane: z==0 clip points project to ±1e20; x86-64 float + garbage can come back `min>max` / non-finite and slip past a + degenerate-box check that clamps to full-screen on N64 (D106). + +## E. Process / method notes + +- Investigation loop is: reproduce → env-gated capped probe → root-cause + → narrow `#ifdef PORT` fix → visual verify (`GE_PCDUMP` + + `tools_pc/pixcount.py` vs `docs/reference/n64-footage-*`). +- gdb **launch** mode is too slow for timing-dependent faults; gdb + **attach** to an already-running process is fine and fast. +- Check `PD_PORT_CHECKOUT` for the PD analogue before + writing anything new — same Rare engine family. +- Don't re-investigate a closed §F finding or re-derive a format spec + that already has a converter. diff --git a/docs/reference/Decompiling a Nintendo 64 Game in 84 Days _ Chris' Blog.htm b/docs/reference/Decompiling a Nintendo 64 Game in 84 Days _ Chris' Blog.htm new file mode 100644 index 00000000..62601afe --- /dev/null +++ b/docs/reference/Decompiling a Nintendo 64 Game in 84 Days _ Chris' Blog.htm @@ -0,0 +1,510 @@ + + +Decompiling a Nintendo 64 Game in 84 Days | Chris' Blog

Chris' Blog

Decompiling a Nintendo 64 Game in 84 Days

I’m very pleased to announce that the original Snowboard Kids is now 100% decompiled! This means that all functions1 have matching C implementations that, when compiled, produce identical machine code to the original game.

This was obviously not a one-person effort. I am particularly grateful to inspectredc, Bl00D4NGEL, and queueRAM for their significant contributions to the project. No amount of AI would have been able to replace them.2 I would also like to thank iFuzzle, JamesBLewis, and douglasjv for lending their tokens to the cause.

My + hope is that a full decompilation will prove useful to the Snowboard +Kids community. Speedrunners in particular have long focused on the +first game. Working source code can help shed light on externally +observed phenomena such as CPU pathing and the exact factors +contributing to player speed. A full understanding of the source code +will also be useful for static recompilation and more ambitious modding +efforts in the future.

The speed at which the project was finished is also noteworthy. Snowboard Kids took only 84 days to decompile compared with 596 days for Snowboard Kids 2, roughly one-seventh of the elapsed time.3

chart showing weekly Snowboard Kids decompilation progress

Weekly progress of the Snowboard Kids decompilation.

What + explains this difference? Well, it’s 2026, so the answer is at least +partially AI. But it would be a gross oversimplification to attribute +the difference entirely to LLMs.

What Was Different

To + state the obvious, I was not starting from scratch. By this point I had + already spent nearly two years on a similar project and was vastly +faster than when I began. This advantage is difficult to quantify and +was somewhat offset by new challenges such as working with a different +compiler.

Overall, roughly 4.8% of matching commits involved expert intervention.4 + I have already credited these amazing people once, but it’s worth +reiterating. This project would not have been possible without +significant help from the decompilation community, particularly inspectredc, queueRAM, and Bl00D4NGEL <3.

The difficulty generally didn’t come from trying to understand what a function did5 but rather how that logic was expressed in C and how the resulting code was compiled. Snowboard Kids was compiled with IDO 5.3 rather than the GCC 2.7.2 compiler used by Snowboard Kids 2.

Most + programmers will be familiar with GCC. It is a widely used open-source +compiler still in active development today. IDO, on the other hand, was a + proprietary compiler developed by SGI, whose own story is closely +entwined with that of the Nintendo 64.6 + Its source was not available, and its original development environment +was tied to obsolete SGI hardware and software. To use and properly +understand it today, the decompilation community has had to +reverse-engineer and decompile parts of the compiler toolchain as well as statically recompile the IDO 5.3 and 7.1 suites to run on modern hardware. That closed history makes IDO harder to reason about than an open compiler such as GCC.

IDO splits optimisation and code generation across several different passes, transforming code quite aggressively along the way.7 Tiny changes to the C can then ripple through those passes and produce a completely different register allocation.

The + community has made great strides in understanding IDO and its quirks, +but this remains more of an art than a science. LLMs and I are not +particularly good at reproducing its output. The usual workflow, for +both me and the agents, was to figure out what a function did and then +write C that approximated that purpose. From there, small tweaks +assisted by the permuter + could catch any remaining differences. But you can’t permute yourself +into a match in all cases, particularly when the underlying structure is + wrong. IDO’s behaviour made this workflow far less predictable.

A motivated human team with the right expertise and intuition can match or even exceed the pace of the Snowboard Kids decompilation. The Pilotwings 64 decompilation was completed in only 74 days.8 Pilotwings 64 had 16% fewer functions than Snowboard Kids, but more compiled code overall, so this is not a clean comparison either.

screenshot of Pilotwings 64

Pilotwings 64, a charming flight simulator, was a Nintendo 64 launch title and remains a cult classic to this day.

Snowboard Kids was also smaller than its sequel, containing 2,145 functions compared with 2,995 in Snowboard Kids 2. Function count is a crude measure of difficulty, but there were simply fewer functions to decompile.

Where Agents Did Help

I’ve already written elsewhere + about using agents to decompile functions. The same basic process was +used here, so I’ll focus on what changed. Unlike the previous project, +this one began with access to frontier models and a capable agent harness.

Library Code and Other Low-Hanging Fruit

I + was interested to see how agents would fare during the early stages of a + project. One area where they thrived was matching standard-library +code. In theory, this is the most obvious chunk of almost any Nintendo +64 decompilation. The code is not unique to the game, and versions of it + are available online. Snowboard Kids contains more than a hundred source segments from Nintendo’s libultra, alongside functions from the libmus audio library. Tools such as N64Sym can identify probable library functions in the ROM.

This + pass was fairly successful. The main stumbling block was convincing +agents to rely on the existing library source rather than decompile the +same functions again from scratch. This required stronger prompting. +Once a likely library function was identified, agents were instructed to + treat the corresponding source as their starting point and exhaust +plausible SDK versions, compiler options, and conditional compilation +paths before attempting their own implementation.

Another optimisation was to have an agent write a script that ran m2c + against every unmatched function and automatically integrated any exact + matches, rather than relying on agents to attempt those functions +individually.9 + The script matched only 17 of 1,830 functions, an incredibly low +success rate of 0.93%, but anything matched this way was cheaper than +burning agent tokens.

IDO Tooling and Skills

IDO + is weird, but it is often weird in recurring ways. Successfully +matching one function could reveal a compiler quirk that applied to many + others. Codex has become better at carrying lessons between tasks +through features such as local memories. To make those lessons available beyond a single agent, I prompted agents to record observed IDO behaviour in a DECOMPILATION_LEARNINGS.md + file. When an agent discovered a generalisable compiler quirk, it could + record the evidence there for later attempts. This created a useful +feedback loop. Agents helped document IDO, and the resulting +documentation made subsequent agents better at matching IDO code.

But the most useful resource was N64 Decomp Workbench, + a collection of tooling and documentation for debugging late-stage MIPS + decompilation mismatches. It can classify mismatches, account for +relocations, replay individual compiler passes, and help distinguish a +structural problem from a register-allocation problem. Pass replay +requires the relevant compiler binaries and project-specific setup, but +once configured it exposes information that a raw assembly diff cannot. A + raw diff tells you that two functions differ. The Workbench can give +agents a much better idea of why the functions differ and what sort of +change might fix them.

Worktrees and Synchronisation

For + this project, I ran the decompilation harness across four Git +worktrees. Each worktree gave an agent an independent copy of the +repository, allowing several functions to be attempted in parallel.

One small but useful improvement was to give every task an explicit deadline and expose that deadline to the agent. During the Snowboard Kids 2 + project, agents often struggled to use the permuter effectively because + it would continue running until it found a 100% match or was manually +stopped. An explicit deadline allowed agents to set sensible timeouts +and trade off permuting time against other forms of problem-solving. +Anecdotally, it also helped them judge how long to keep working on a +difficult function before giving up. I could then increase the time +allowance as the easy functions disappeared and the remaining work +became more difficult.

Another problem that had existed in Snowboard Kids 2 but became more apparent as I added worktrees was synchronisation. As discussed in my previous post, + each agent was given a function to decompile alongside a set of similar + functions that had already been matched. These provided useful +reference points for reproducing particular IDO instruction patterns. +Work was divided between the worktrees using Nigel’s --shard option, which uses basic hashing to partition candidates between a specified number of workers.

This + allowed more work to happen in parallel without duplication, but +introduced a new problem as the worktrees diverged. One worktree, for +example, might successfully decompile a function that was 99% similar to + a function being attempted elsewhere, but that new reference would +remain invisible to the other agent until the changes were merged. +Periodically merging everything into the main branch and resynchronising + the worktrees fixed this, but synchronising all four could take more +than an hour. Synchronising continuously wasted time, while waiting too +long increased drift.

To make synchronisation less critical, I +updated the similarity search to inspect every worktree. A newly matched + function could immediately become a reference for another agent without + waiting for it to reach the main branch. The harness would produce +messages such as the following.

1✓ Candidate ["func_80094A94", "func_80094FF4 (../sbk-c)",
+2  "func_80094808", "func_8009491C (../sbk-a)",
+3  "func_8009469C (../sbk-c)", "func_80093144"] was fixed!
+

This helped the process remain +efficient as the easy functions disappeared. Synchronisation was still +needed to consolidate and push changes, but it was no longer required +for agents to learn from one another.

Model Choice

I + tried GPT-5.5 and 5.6, Claude 4.5 and Fable, and GLM 5.2. This is +completely unscientific. The models were tested against a changing set +of difficult functions, sometimes after another model had already made +partial progress. Broadly, though, Codex continued to outperform Claude, + as it had towards the end of the previous project. Sol xhigh was +particularly effective once it became available.

GLM 5.2, served +by z.ai, was very disappointing. I had previously been a big fan of GLM. + It was effective, even if it was not quite a frontier model, and its +generous usage limits made up for the gap. That tradeoff became much +less attractive as the limits grew less generous while latency remained +awful. The feedback cycles were so long that I stopped giving it work +and eventually cancelled my subscription.

What Next?

The + immediate priority is to better document the game. A 100% match means +we have C code for every function; it does not mean we understand what +every function does. There are still generated names to replace, unknown + structure fields to identify, awkward matches to clean up, and large +amounts of data to describe.

Work is also underway on a Snowboard Kids recompilation. Fortunately, the first game shares many of the quirks addressed by patches in Snowboard Kids 2: Recompiled.

early screenshot of Snowboard Kids: Recompiled

I’m + looking into porting the levels and other content from the first game +into the second game’s engine, although I have no idea how much work +that will be yet.

Beyond that, I’m interested in decompiling Snowboard Kids Plus on the PlayStation, a Japan-exclusive expanded release of the first game with extra levels and characters.

If you’ve made it this far, you’re probably interested in decompilation and Snowboard Kids. Take a look at the Snowboard Kids decompilation project. There is still plenty of cleanup and documentation work to do, and contributions are very welcome.

  1. Almost. + A small amount of handwritten assembly from system libraries remains +because assembly was the original source rather than a temporary +decompilation placeholder. ↩︎

  2. In the best case, I think the project would have stalled around 89–90% without their IDO expertise. ↩︎

  3. For Snowboard Kids 2, I measured from the first commit on 28 September 2024 to the final matching function on 17 May 2026, a span of 596 days. The 84-day figure for Snowboard Kids is my project-level count; the public repository itself spans from the first commit on 28 May 2026 to the final matching function on 19 August 2026, just over 83 days. ↩︎

  4. Based + on a manual review of the Git history, I identified roughly 41 +expert-assisted matches, corresponding to about 4.8% of the commits I +classified as function-matching work. This retrospective classification +is approximate because commits can contain multiple functions, help was +often incorporated into my own commits, and the figure does not measure +person-hours. Note that I am not counting myself as an “expert”. ↩︎

  5. One notable exception was validateControllerPakSave, + which appears to have been used to debug Controller Pak saves. Most of +its debugging and validation logic was omitted from the final +build. ↩︎

  6. Nintendo has described + adopting an architecture from Silicon Graphics for the Nintendo 64. The + original development system was similarly SGI-centric. Nintendo’s development documentation + describes connecting its emulator board to an SGI Indy workstation, and + IDO was one of the supported compilers. The relationship therefore +extended from the console’s underlying architecture to the machines and +tools used to develop its games. ↩︎

  7. IDO used a multi-stage compilation pipeline. Its uopt documentation + says that eligible loops were unrolled four times by default. +Coincidentally, four is also the number of racers in the game. Loops +with four iterations are therefore candidates for complete unrolling, +and they occur frequently 🫠. ↩︎

  8. Measured from the addition of progress tracking on 26 January 2026 to the final matching functions on 10 April 2026, the decompilation took almost exactly 74 days. Pilotwings 64 had 1,791 functions compared with 2,145 for Snowboard Kids, but approximately 838 KB of code compared with 732 KB for Snowboard Kids. ↩︎

  9. Using a one-shot script like this is not a particularly novel idea. I’m pretty sure I copied it from somewhere, possibly Diddy Kong Racing’s m2c_all.sh. ↩︎

Comments

Loading Bluesky comments...

<< Previous Post

|

Next Post >>

+
+
+
+
+
+ + + + VT4Browsers + Google TI + + + + +
+
+
Fetching file...
+
+
+ +
+
+
+
+
+
+
+
 Click to open IoC report 
with VirusTotal Augment \ No newline at end of file