goldeneye-pc-port/docs/PORT-LEARNINGS.md

PORT-LEARNINGS — recurring N64→PC bug classes

One-screen index into docs/PCPortResearch.md §F. Every investigation subagent reads this first and appends any new generalisable quirk. Details, evidence, and the fix for each instance live in §F under the cited Dxx label.

A. Pointer-width struct growth (32→64) — the dominant class

A decomp struct with pointer fields, or one pun-allocated into a fixed N64-sized hole / hardcoded byte count, is larger on x86-64. Reading it from ROM bytes misaligns; allocating it N64-sized overruns adjacent state.

• Symptom: garbage pointer deref, corrupted neighbour field, delayed fault far from the cause.

• Fix: store embedded ROM addresses as u32, cast at use site (PD ground truth); or give real inline storage / sizeof()-based alloc under #ifdef PORT.

• Instances: D53.2 (ModelSlot pun), D56 (watch Model raw offsets), D57 (rwdata record count), D67 (image_entry), D79 (bg_room_data), D98 (struct player alloc), D100 (player.model inline Model), D101/D102 (ModelNode*/weapon Model puns), D115 (gunfire THROW* raw offsets), D119 (weapons_held[]->chr punned as ChrRecord* to read .act_*.attack_item — aliased WeaponObjRecord.weaponnum at 0x80 on N64 via act-union@0x2C+84; act union moves to ~0x38 on PC).

• Open landmine: raw hardcoded-offset accessors into struct player / struct hand — see docs/AUDIT-M6-player-offsets.md.

• D126 corollary — a trailing runtime list pointer in a ROM-serialized record. If a fixed-size ROM record ends with a T *next (or *child, *parent) that game code writes while walking the setup stream (linked-list build), that field is 4B/end-of-record on N64 but widens to 8B and 8-aligns on PC — the store spills past the N64 record size into the next record and silently corrupts a polymorphic walk downstream (type byte clobbered → wrong strides → wrong command indices → wrong pointer resolution, crashing far away). Offline converter (d*_emit.py) MUST emit these at the real PC sizeof with the pointer slot widened + the matching #ifdef PORT sizepropdef/stride. Instances: d88_propdefs.py types 30/32/33/35 (criteria_* / setup_objective_text ->next), 22 (TagObjectRecord.NextTag, already handled). Check every record type the walk can see for a set_parent_* / *_entry_parent writer.

• D132 corollary — a union { T *ptr; s32 IndexN; } field is 8B/8-aligned on PC even when the ROM image only ever stores the s32. LinkRecord (first/Index1), LockDoorRecord, SafeObjectRecord etc. pack their index words tight (4B) in the N64 setup stream, but the compiler places each IndexN in the LOW 4 bytes of an 8-byte-aligned pointer slot (Index1@8 not @4, Index2@16 not @8, then a trailing *next). A converter that lays them at N64 4-byte offsets makes pdef->Index1 read Index2's value and Index2 read 0 → the record's validity guard fails and switch-doors / dual weapons / locked doors / safes silently never initialise (usually non-crash, because the failed guard also skips the pointer stores). Fix: emit each IndexN at PC offset 8 + 8*(N-1) and size the record with the widened unions + trailing *next (d88_propdefs.py types 14/19/38/44 — proposed in §F D132, not yet applied). Grep every ROM-serialized record for union { with a pointer arm. NB [u16 ID][s16 x] NOT in a union (e.g. TagObjectRecord) stays at offset 4 — only the pointer union forces the 8-align.

• D130 corollary — in-place N64→PC struct-array re-layout aliases when the stride grew less than the read span. A ROM struct array widened on PC (fontchar 24→32B, romdataFixupFont) is often re-laid-out in place: for glyph i, dst = base + PCstride*i, src = base + N64stride*i, so dst − src = (PCstride−N64stride)*i. For small i that delta is less than the per-element field-copy read span, so dst overlaps src, and a forward field-by-field *dst_k = f(*src_k) loop overwrites a not-yet-read src_j (j > k). Result: later fields of the first few elements come out as f(f(earlier field)) — e.g. glyph 1's width became bswap(bswap(index)). Iterating elements backward does NOT help (the overlap is within one element). Fix: read every source field of the element into locals first, then write. Grep every romdataFixup* / in-place relayout for a read-write loop whose dst/src can alias for low indices.

• A negative/OOB index into a widened struct array is fatal on PC, benign on N64. chars[*text - 0x21] with a control byte (*text < 0x21) reads before chars[0]. On N64 that hits the adjacent 4-byte-field kerning table (small values → a garbage-but-TMEM-valid glyph); on PC the struct's 8-byte pixeldata reads a wild pointer → fast3d AV. Latent in textrelated.c (textRender*/textMeasure ASCII paths) — a GLYPH_IDX clamp fixes it if a level actually feeds a control byte to the HUD text (none do yet; D130 was a font-relayout bug, not this).

• D131 corollary — osVirtualToPhysical() truncates a compiled-symbol pointer to 32 bits. The port shim is (u32)(uintptr_t)va (libultra.c). For a runtime DRAM pointer (0x70xxxxxx) the cast is lossless; for a compiled module symbol (.bss/.rodata matrix, e.g. &dword_CODE_bss_8007A100 in explosionRenderPropSmoke) it drops the 0x1_00000000 module high word, so the GBI w1 becomes 0x40xxxxxx and seg_addr() hands fast3d a wild pointer → AV in gfx_sp_matrix / gfx_sp_vertex. Same class as D94 (chraction.c:1243), but in a DL word. ~30 latent sites (grep 'osVirtualToPhysical(' src/game/{explosion,glass,glass2,blood_animation,bondview2}.c), each armed only when that effect first draws. Fixed once in seg_addr(): restore the high word for a fallthrough w1 ∈ [0x40000000, 0x70000000) (module is fixed-based at 0x140000000; DRAM/KSEG0/segmented/phys are all handled in earlier branches). gSPDisplayList(&globalDL_0xNNN) is NOT affected — the port's Gwords.w1 is 64-bit and gDma1p stores the full pointer; only osVirtualToPhysical truncates.

B. 16-byte PC Gfx / Vtx vs 8-byte N64

Any buffer reservation, memcpy size, slot stride, or pool budget expressed in N64 Gfx/Vtx units is half-size on PC.

• Instances: D50.6 (texCopyGdls copied only w0 of each 16-byte slot), D58 (DL reserve 0x100→0x200), D85 (bgWidenRoomGdl 8→16 + bswap), D95 (2× master-DL buffer + raised mempool ceiling).
• D135 corollary — an unported GBI parser (not just a buffer size). Code that walks a DL command stream with raw byte/word indices into an 8-byte N64 Gfx (*(s8*)gdl for the cmd, ((u32*)gdl)[1] for w1, ((u8*)gdl)[5..7] for G_TRI1 vtx indices, ((u32*)gdl)[0/1] nibble reads for G_TRI4) reads the wrong bytes of the PC 16-byte {u64 w0; u64 w1} slot and desyncs on the first command → walks off the DL → wild deref. The N64 32-bit words survive in the LOW dword of each 64-bit field, so the mechanical port is: u32 w0 = (u32)gdl->words.w0; u32 w1 = (u32)gdl->words.w1; then replace every raw access with the equivalent shift/mask on w0/w1 (BE byte i of a word → (word >> (8*(3-i))) & 0xff). gdl++ (advances by sizeof(Gfx)) is already correct. Watch (s32)ptr truncation in vtx-base math and x | 0x80000000 KSEG0 folds (identity on PC — just drop the OR, and guard segmented w1). Instances: bgTestHitOnObj (propobj.c, FIXED); bg.c room-geometry hit-test sub_GAME_… (~3373-3646, still latent — shooting walls). PD ground truth: pd_port uses gdl->dma.cmd + GFX_W0_BYTE(i)/GFX_W1_BYTE(i) macros (3-i / 11-i on 64-bit LE).
• Also bites RAW hardcoded struct-stride writes, not just Gfx/Vtx. D128: sub_GAME_7F0B37EC did ((u8*)g_BgPortals)[(portal<<3)+6] |= 2 — the N64 bg_portal_data_entry is 8B (ptr@0, cr1@4..cb2@7); on PC the widened offset_portal ptr makes it 16B (cr1@8..cb2@11), so the write landed in the middle of another portal's pointer. Fix = use the struct accessor under #ifdef PORT (every other site already does: g_BgPortals[portal].controlbytes1 |= PORTALFLAG_SPECIAL). Grep for << 3 / * 8 / + 6 style raw offsets into any struct that gained a pointer field.

C. Big-endian rodata / ROM data read on little-endian PC

ROM assets and compiled-in .rodata are big-endian. Anything not run through a converter or a runtime bswap fixup reads scrambled.

• f32 values are BE word pairs — a naive byte reversal off-by-one corrupts every float: D73 (sinf/cosf du pairs → DVAL() macro), D112 (d43_emit.py put_f32 src[doff:doff+4][::-1]).
• Header offset tables / pointers: D54 (cseq ALMidiHdr), D68 (Globalimagetable), D87 (ramromfilestructure), D88 (Usetup* tables).
• Negative-terminated index chains (PointUsage[]) in converted model rodata cycle forever if element endianness/stride is wrong: D120 (opcode-0x18 collision record, d43_emit.py — guarded, not fixed).
• Packed bitfields cross byte boundaries differently: D78 / D83 (StandTile id/room, header mid/tail).
• Rule: prefer an offline sidecar converter (D43, D69, D88 pattern, tools_pc/d*_emit.py) over a runtime fixup for a whole format.
• D139 — (u8)word / (s8)word to grab "the first field" of a struct reads the WRONG byte on LE. N64 code that does (u8)obj[0] (obj a u32*) to read a byte-3 field of a big-endian header word — e.g. the propDef type in [u16 extrascale][u8 state][u8 type] — gets the low byte, which is the last BE field. On LE it's extrascale. cleanupObjects did this for its walk-termination + type dispatch → never saw PROPDEF_END → ran off the blob → freed garbage → crash on stage unload. Fix: use the struct member (pdef->type) like every other consumer. Grep for (u8)*/(s8)*/& 0xff on a u32/s32 that's really a serialized multi-field word — especially GBI-word and header-word reads that were never routed through a converter or #ifdef PORT.
• Python slice-assignment width is a silent corruptor (D125): out[a:a+8] = b"\x00\x00\x00\x00" on a bytearray does NOT raise — it deletes (8 - len(rhs)) bytes, shrinking out and shifting everything after a down. A widened pointer/stan slot filled with a 4-byte literal shrank d88_emit.py's output 4 B per pad; once len(out) fell below a later region's write offset, out[off:off+n] = data clamped to an empty range at the current end and inserted there — string blobs drifted, boundpad names truncated (p138d2→8d2), stanPackId rejected → door model=NULL → crash. When emitting into a pre-sized bytearray, every out[a:b] = rhs MUST have len(rhs) == b - a; grep the emitter for width-mismatched slice writes when a converted region lands wrong.
• A one-shot "sync patched values from the ROM copy into a compiled shadow array" pass must key its slot-detection on a field that is still intact at sync time. D124: gimgSyncCompiledGlobalDLs looked for the 0xABCDxxxx IMAGESEG marker in the ROM copy — but texLoad() had already overwritten every one of those with a real pointer, so the sync silently copied nothing and the compiled globalDL_0xNNN explosion DLs kept their link-time markers (latent on all levels; only tripped when an explosion/smoke/particle DL is first drawn). Detect the slot from the destination (compiled) array, which still holds the marker.
• A polymorphic-record converter with a per-type handler table + a "generic" fallback silently corrupts any type the table forgot: the fallback's word-granular bswap is wrong for sub-word fields. D122 — d88_propdefs.py had no arm for 6 inherits ObjectRecord propDef types (TINTED_GLASS/VEHICHLE/AIRCRAFT/TANK/AUTOGUN/AMMO); the generic arm bswap32'd the [s16 obj][s16 pad] word as one u32, putting the model id in the wrong half → OOB table deref. When you add a converter, enumerate every type byte a level can emit and assert the handler set is total; [u16|u16] / [s16|s16] packed words need a half-swap (_hh_word), never a 32-bit swap.
• D123 corollary: a pointer-width tail slot is not always runtime-populated. Some hold a small integer id in the ROM image that game code reads via that same field before overwriting it (VehichleRecord/AircraftRecord.ailist: prop.c:1764/1786 does x->ailist = ailistFindById(x->ailist)). A converter that lumps every pointer-width tail slot into one "widen 4→8B, emit zeroes" bucket destroys the id → ailistFindById(0) silently returns global list 0 (GAILIST_AIM_AT_BOND) → ai() runs a CHR aim list with a NULL chr → NULL deref in chrIsNotDeadOrShot. Before zeroing a widened slot, confirm the setup code writes it unconditionally; if it reads-then-writes, emit the value (_bswap32 into the low 4 bytes, LE). Decide per field, not per type.

C2. Port-layer / SDL shims

• #include <PR/os.h> in a port .c/.h that also sees <errno.h> breaks: OSContStatus/OSContPad have a u8 errno; field vs errno.h's macro. libultra.c wraps the include in #pragma push_macro("errno") / #undef errno; cleaner for a new module is to duplicate the handful of CONT_* bits it needs (D118 input.c).
• GE aims with digital C-buttons, no analog-aim hook outside src/. Mouse-look is bridged by integrating the relative-mouse delta into a clamped per-axis accumulator and emitting a C-button per poll while |accum| ≥ 0.5, draining one unit → proportional press dwell. Non-linear (GE's own accel curve) but playable (D118).
• osContGetReadData(pad) must fill one OSContPad per channel (MAXCONTROLLERS-long array), not just controller 0 — joy.c passes the whole samples[i].pads array (D118).
• Controller state has one source: port/src/input.c. libultra.c's SI section marshals inputComputePad() into g_contPad[]; it is driven by osContStartReadData (per logic tick), no separate video.c frame hook.

D. N64 hardware idioms fast3d does not emulate

• Z buffer cleared by pointing the colour image at it + fill-rect → does nothing in fast3d; must emit G_CLEAR_DEPTH_EXT (D105).
• LOD / detail mip tiles: fast3d fabricates a crop when detail textures are off → force base tile (D107).
• K0 segment-address folds (OS_K0_TO_PHYSICAL | 0x80000000) and hand-inlined BG_SEG_TO_PTR need the (u32) 32-bit wrap the macro has (D58, D84).
• Portal near-plane: z==0 clip points project to ±1e20; x86-64 float garbage can come back min>max / non-finite and slip past a degenerate-box check that clamps to full-screen on N64 (D106).
• D74 wrap-block is DEAD CODE (gfx_pc.cpp:1546/1551): guard is cms & G_TX_WRAP but G_TX_WRAP == 0, so always false (line 1887 does it right with cms == G_TX_WRAP). And if it did run it indexes tex_width2[i] etc. (arrays [2], per-texunit) with the vertex loop index i 0..2 → OOB. Latent; harmless for CLAMP glyphs, would matter for wrapped textures on tris. Fix deferred (M-11) — logged in docs/GRAPHICS-BACKLOG.md.

D2. The HUD/model X-mirror (D114/D116) — DO NOT re-static-trace

Sessions M-6/M-7/M-8/M-11 all reached the same wall: textRenderGlyph → GBI → fast3d gfx_dp_texture_rectangle → buf_vbo → GL are each runtime-verified non-mirrored ([D116/vbo] probe: x-left↔u=0), yet glyphs render X-flipped. M-11 additionally confirmed the ammo digits use the same textrelated.c path (via gunfire.c:5906), no separate renderer. Every prior attempt to "fix" it drifts toward a global S-swap that mirrors the whole screen — a bad trade for a cosmetic bug. Next attempt needs a RenderDoc/apitrace capture of one glyph texrect OR the asymmetric-1-texel-texture experiment — nothing else. Cosmetic, deprioritised below level-progression / crash work. See docs/GRAPHICS-BACKLOG.md.

E. Process / method notes

• Investigation loop is: reproduce → env-gated capped probe → root-cause → narrow #ifdef PORT fix → visual verify (GE_PCDUMP + tools_pc/pixcount.py vs docs/reference/n64-footage-*).
• gdb launch mode is too slow for timing-dependent faults; gdb attach to an already-running process is fine and fast.
• Check PD_PORT_CHECKOUT for the PD analogue before writing anything new — same Rare engine family.
• Don't re-investigate a closed §F finding or re-derive a format spec that already has a converter.
• The port is NOT frame-deterministic (D117): osGetCount() is wall-clock on PC, and GE is a variable-timestep sim (frametiming.c waitForNextFrame → deltaFrames = 60 Hz ticks of real time elapsed per render), so "frame N" differs 15–40 % between runs. Use tools_pc/framediff.py <ppmdir> (structural: 16×12 grid mean-colour + non-clear-% + aHash, --mask X0,Y0,X1,Y1 for the HUD, --update to refresh tools_pc/golden/) — NOT an exact compare. A GE_DETERM=1 fixed-tick mode was assessed not-narrow (redesigns retrace/tick semantics); design is in §F D117 if someone picks it up, after which framediff.py --exact becomes usable.
• D24-implications — host-scheduling nondeterminism / fake priority semantics: the pthread kernel does not enforce N64's 0–31 priorities (osYieldThread = Sleep(0)), so interleavings impossible on console can occur, and frame timing has host jitter. When a flaky timing bug appears (especially Phase 3 audio underruns), reach first for host thread priorities (SetThreadPriority: scheduler thread time-critical, tick normal) and the deferred GE_DETERM mode (D117) — not for kernel changes.
• "Regression vs. steady state" is decided by a build-and-compare, not by reading an old handoff line (D133). Handoff prose like "the entire intro renders" is often aspirational — the author's intent, not a measured coverage number. Before bisecting a suspected render regression: git worktree add the commit whose handoff made the claim (COPY data/, never junction — see the data-dir-junction-hazard memory), build, capture the same GE_PCDUMP window, pixcount.py. A pixel-identical non-clear count on a static 2D screen (e.g. the legal/disclaimer framebuffer, 6677 px here) across two independently-built binaries proves the path is unchanged → not a regression, don't bisect. Clean up the worktree with git worktree remove --force (safe only because data/ was copied).
• D134 — a dropped OS_MESG_NOBLOCK "interrupt" event is a permanent stall, and it looks exactly like D117 flakiness. The port's SP/DP task-done events shared the scheduler's 8-slot interruptQ with the 60 Hz VI pacemaker. A gfx task runs the whole frame synchronously on the sched thread, so a slow frame lets the pacemaker fill the queue and the done-post is silently discarded → curRSPTask never clears → no further frames (frames=2 heartbeat, retrace queue pinned valid=8/8 ret=-1). Rule: classify every posted event as droppable (retrace — N64 drops these too) or must-arrive (task done, DMA done); a must-arrive event needs guaranteed delivery, and OS_MESG_BLOCK is wrong whenever the poster runs on the queue's only consumer thread (self-deadlock) — drop the oldest droppable message instead, and reserve slots in the pacemaker. Before blaming a hang on D117 / machine load, run it N times and count: this reproduced 2 of 3 boots on an idle machine and had been written off as sweep flakiness for ~8 sessions.
• Watch item: osYieldThread = Sleep(0). The one place we fake cooperative behavior. If a level-sweep hot loop misbehaves under host load, this shim is the first thing to inspect. Keep as-is until then.