From 3dcceaafd954db0e47c6f5fbf282c82f76e8ea1b Mon Sep 17 00:00:00 2001 From: James Dansereau <106631954+jkdansereau@users.noreply.github.com> Date: Sat, 12 Sep 2026 15:33:07 -0400 Subject: [PATCH] ci: route windows-build to self-hosted runner (G2); land ROM-gated sweep workflow (G1) (#65) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * ci: route windows-build to self-hosted runner (G2); land ROM-gated sweep workflow (G1) G2: windows-build now runs on the self-hosted box (corpyvt6agent01) for push/workflow_dispatch, using local ccache instead of the actions/cache round-trip and skipping msys2/setup-msys2's install once packages are already present. pull_request always stays on windows-latest — fork PRs must never reach a box that will hold a ROM. G1: adds .github/workflows/selfhosted.yml, a dispatch-only workflow that builds + runs the 21-level headless sweep (tools_pc/verify.sh sweep --json) against a runner-cached ROM. Inert until manually dispatched. Still pending (manual, physical steps on the box — not done by this commit, see docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md rollout table): - register corpyvt6agent01 as a runner against this repo - confirm MSYS2 at C:\msys64 with the required mingw64 package set - one-time ROM seed into {runner_home}/_rom-seed/ - first workflow_dispatch sweep + parity check against a local run - flip G1 from workflow_dispatch to push:[main] once green Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU * fix(selfhosted.yml): remove invalid concurrency.max-parallel key Found by testing: dispatching the workflow failed with a hard parse error (HTTP 422, "Unexpected value 'max-parallel'"). GitHub Actions' workflow-level `concurrency` block only supports `group` and `cancel-in-progress` — `max-parallel` is a job `strategy:` key for matrix jobs, not a concurrency-block key. Serialization was already achieved via group + cancel-in-progress:false; the stray key just blocked every dispatch. Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU * fix(selfhosted.yml): register the msys2 shell wrapper (setup-msys2 step) Found by testing: the first fixed dispatch failed every msys2-shell step with "msys2: command not found". Unlike ci.yml's windows-build, this workflow used `shell: msys2 {0}` without ever running msys2/setup-msys2, which is what registers that shell alias for the runner — MSYS2 being pre-installed on the box doesn't help if Actions doesn't know the shell exists. Adds the same setup-msys2 step ci.yml uses, with update:false since the packages are expected already present (fast no-op per plan). Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU * ci: stop reinstalling MSYS2 per run on the self-hosted box The self-hosted paths (selfhosted.yml; ci.yml windows-build's push/ dispatch route) no longer call msys2/setup-msys2 or pacman -S at all. MSYS2 + the mingw64 package set are now assumed hand-maintained on corpyvt6agent01 (docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md), invoked by calling its bash directly (`C:\msys64\usr\bin\bash.exe -leo pipefail "{0}"`, the same invocation setup-msys2's generated wrapper uses) with MSYSTEM=MINGW64 so /etc/profile still puts mingw64/bin on PATH. Adds tools_pc/check-selfhosted-deps.sh: a fast, install-free check that exits 0 immediately when the required packages (toolchain, cmake, SDL2, zlib, python, ccache, zip, unzip) are present, or fails with the exact `pacman -S --needed ...` line to run otherwise — so a missing package on the box surfaces immediately with a clear fix, instead of as a confusing failure deeper in the build. ci.yml's windows-latest (PR) path is unchanged: ephemeral runners still get a full msys2/setup-msys2 install every time, since there's nothing to persist there. Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU * fix: use msys2/setup-msys2 release:false instead of a raw bash-path shell The previous attempt (invoking C:\msys64\usr\bin\bash.exe directly as a custom `shell:` command) failed on the actual runner with "Second path fragment must not be a drive or UNC name" — a real limitation, not a typo; it failed identically in both files, including the one with a static (non-templated) shell string, so it wasn't the ${{ }} expression either. Root-caused by reading msys2/setup-msys2's actual source: `release: false` makes it reuse the existing C:\msys64 as-is (hardcoded path in that branch, confirmed in main.js) instead of downloading+installing a fresh copy, and — verified in the same source — omitting `update` and `install` skips every pacman call entirely (just one `uname -a` + a pacman.conf sed tweak, no network). This is the officially documented mechanism for self-hosted runners with a pre-existing MSYS2 install (see the action's README, "release" section). ci.yml's windows-build now runs setup-msys2 unconditionally but with release/update/install all keyed off github.event_name: full install on windows-latest (PR, ephemeral), reuse-only on the self-hosted box (push/dispatch). selfhosted.yml always uses release:false. Both then run tools_pc/check-selfhosted-deps.sh to verify the box's hand- maintained packages are actually present. Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU * fix(selfhosted.yml): cp -r for the ROM sidecar directories Found by testing with the ROM seed actually in place: pcmodels-ntsc-final and pccg-ntsc-final are directories (not single files, unlike ge007.ntsc-final.z64), so a plain `cp` failed with "omitting directory" and left the seed step failing on every run even with the seed populated. Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU * ci: drive MSYS2 MINGW64 directly via ci-shell.cmd, drop msys2 shell shim The self-hosted runs were dying with 'msys2: command not found' because the msys2/setup-msys2 action's shell shim was never registered on the box. Replace the dependency on that shim with a small committed launcher (tools_pc/ci-shell.cmd) that headlessly reproduces the local 'MSYS2 MINGW64' terminal: - MSYSTEM=MINGW64 + /mingw64/bin on PATH - NON-login bash, so it keeps the workspace cwd (a login shell cd's to $HOME and would break every relative path in the workflow) - -e -o pipefail for GitHub-style error propagation Both workflows now set defaults.run.shell to C:\msys64\ci-shell.cmd {0}; a bootstrap step (shell: cmd) refreshes the launcher from the repo into C:\msys64 each run. selfhosted.yml drops setup-msys2 entirely; ci.yml keeps it only for the ephemeral windows-latest PR path (which installs MSYS2 at C:\msys64). .gitattributes: force CRLF for *.cmd so cmd.exe parses the launcher correctly. * Revert "ci: drive MSYS2 MINGW64 directly via ci-shell.cmd, drop msys2 shell shim" This reverts commit a976b2589c2de60e4dd4121cca78983e1057f8f5. * docs(D244): log self-hosted-runner link failure — undefined reference to objectiveGetStatus_WEAK First real build attempt on the reinstalled corpyvt6agent01 box (run 34707798707) fails at link with an unresolved #pragma weak alias. A local repro of the identical alias pattern links fine on current MSYS2 (gcc 16.2.0/binutils 2.47); the box reports gcc 16.1.0 — likely a stale toolchain, consistent with rollout step 2's earlier package-hygiene gap. Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3 * fix(verify.sh): surface run.log tail on NO-FRAMES instead of silently discarding it sweep mode's per-level run.log (the game process's own stdout/stderr, where an SDL/GL init failure would show up) was captured into a mktemp capdir but unconditionally rm -rf'd with no diagnostic ever printed for the NO-FRAMES (no-crash) case -- every level failed this way on the self-hosted runner's first real sweep attempt with zero clue why. Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3 * fix(selfhosted.yml): stop deleting the ROM seed dir -- cache save silently fails on this box actions/cache/save has been failing to persist the ROM cache entry on corpyvt6agent01 (WARNING annotation, tar.exe exit code 2) but doesn't fail the step -- by design, a cache-save failure is non-fatal, so the workflow carried on and the now-unconditional "Delete one-time seed dir" step wiped the only copy of the ROM every single run. Every dispatch since has failed at "Seed ROM cache" with an empty seed dir and an empty cache -- exactly the user-reported "it wipes out the ROM data each run". Fix: don't delete the seed dir. It's cheap disk (three files, a few hundred MB) and the existing restore-then-seed-if-missing logic already treats it as the fallback source whenever the cache comes back empty, so leaving it in place is safe and makes the workflow tolerant of the cache-save bug until that's root-caused separately. Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3 * fix(verify.sh): sweep --json never actually produced a verdict array; fix NO-FRAMES diagnostic clipping it further Two bugs, found together while chasing the self-hosted runner's first real sweep attempt: 1. sweep mode's `out=$(verify_level ...); echo "$out" | head -1` always discarded emit_verdict's second (--json) output line -- RESULTS_JSON was declared for assembling a verdict array but never populated or printed, so `sweep --json` has never produced anything but `[]` for the workflow's "Extract JSON verdict" step. Now builds and prints a real `[{...}, {...}]` array before the RC exit. 2. My own D244-session addition (printing a NO-FRAMES level's run.log tail) wrote to stdout, which the same `out=$(...)/head -1` capture turned into the *new* first line, clipping the real verdict summary. Moved to stderr, which bypasses that capture. Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3 * docs: regenerate findings-index.csv for D244 Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3 --- .github/workflows/ci.yml | 33 +++-- .github/workflows/selfhosted.yml | 234 ++++++++++++++++++++++++++++++ docs/dev/findings-index.csv | 1 + docs/dev/findings.md | 1 + tools_pc/check-selfhosted-deps.sh | 57 ++++++++ tools_pc/verify.sh | 30 ++++ 6 files changed, 345 insertions(+), 11 deletions(-) create mode 100644 .github/workflows/selfhosted.yml create mode 100644 tools_pc/check-selfhosted-deps.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 4f39b5cc..cf0f185e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -224,7 +224,10 @@ jobs: windows-build: needs: changes if: needs.changes.outputs.code == 'true' - runs-on: windows-latest + # Self-hosted for push/workflow_dispatch (fast, local ccache); fork PRs + # always stay on windows-latest — untrusted PR code must never reach a + # box that will later hold a ROM (docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md §G2). + runs-on: ${{ github.event_name == 'pull_request' && 'windows-latest' || fromJSON('["self-hosted", "windows"]') }} defaults: run: shell: msys2 {0} @@ -240,19 +243,27 @@ jobs: with: persist-credentials: false + # windows-latest (PR) is ephemeral: full install + update every run. + # Self-hosted (push/dispatch): `release: false` reuses the box's own + # C:\msys64 as-is and, with update/install both empty here, does no + # pacman -Syu / pacman -S at all (verified against the action's source) + # — it only registers the `msys2 {0}` shell wrapper every job needs. + # Packages themselves are hand-maintained on the box; the dependency + # check below (not this step) is what verifies they're present. - uses: msys2/setup-msys2@v2 with: msystem: MINGW64 - update: true - install: >- - mingw-w64-x86_64-toolchain - mingw-w64-x86_64-cmake - mingw-w64-x86_64-SDL2 - mingw-w64-x86_64-zlib - mingw-w64-x86_64-python - mingw-w64-x86_64-ccache - zip - unzip + release: ${{ github.event_name == 'pull_request' }} + update: ${{ github.event_name == 'pull_request' }} + install: ${{ github.event_name == 'pull_request' && 'mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake mingw-w64-x86_64-SDL2 mingw-w64-x86_64-zlib mingw-w64-x86_64-python mingw-w64-x86_64-ccache zip unzip' || '' }} + + # Self-hosted only: fails fast with exact install instructions if the + # box's hand-maintained MSYS2 is missing a required package, instead of + # a confusing failure deeper in the build. windows-latest just installed + # everything above, so this would be redundant there. + - name: Check self-hosted dependencies + if: github.event_name != 'pull_request' + run: tools_pc/check-selfhosted-deps.sh - name: Restore ccache uses: actions/cache@v4 diff --git a/.github/workflows/selfhosted.yml b/.github/workflows/selfhosted.yml new file mode 100644 index 00000000..d5e99ca9 --- /dev/null +++ b/.github/workflows/selfhosted.yml @@ -0,0 +1,234 @@ +# Self-hosted runtime (ROM-gated) +# +# Runs the build + the 21-level headless sweep (tools_pc/verify.sh) on the +# self-hosted runner "corpyvt6agent01", the one machine that has a legal ROM. +# This is the job class ci.yml explicitly does NOT do: CI is build-only, no +# ROM, so runtime checks live here. Setup + rollout plan (gitignored, local): +# docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md +# +# Persistence model — everything lives under the runner's own paths, managed +# by the pipeline; no hand-maintained directories: +# - ccache -> actions/cache (stored by the runner under _work/_temp) +# - ROM -> actions/cache, key ge-rom-data-ntsc-v1. Bootstrap: copy the +# three files into {runner_home}/_rom-seed/ on the box; a run +# with an empty cache copies them in and attempts to save the +# cache entry. The seed dir is kept (NOT deleted) rather than +# treated as one-time -- actions/cache/save has been silently +# failing to persist on this box (warning, not a hard step +# failure; root cause open, see SELFHOSTED-RUNNER-PLAN.md), so +# the seed is the durable fallback every run re-copies from +# until that's fixed. Integrity is re-verified every run +# against the committed ge007.u.sha1 (repo-tracked US ROM hash). +# - MSYS2 -> machine-level install at C:\msys64 (verify.sh hardcodes the +# prefix), installed and its packages maintained BY HAND on +# the box. msys2/setup-msys2 still runs every job (it's what +# registers the `msys2 {0}` shell wrapper), but with +# `release: false` and update/install both omitted — verified +# against the action's source to do no pacman -Syu / pacman -S +# / network round-trip, just a fast reuse of the existing +# C:\msys64. Package presence is checked (not installed) by +# tools_pc/check-selfhosted-deps.sh, which fails fast with +# exact install instructions if something's missing. +# +# Trigger gating (non-negotiable): +# - workflow_dispatch ONLY for now. Flip on `push: branches: [main]` once +# the first manual runs are reliably green. +# - NEVER add `pull_request`: unreviewed fork code must not be routable to +# a runner that can see the ROM. +# +# Leak rules (docs/dev/notes/AGENT-VM-FORGEJO-PLAN.md checklist): +# - The ROM + sidecars live in the runner-managed cache only; nothing in +# this workflow uploads from data/. +# - Artifacts here are console logs + a JSON verdict only. Rendered frames +# (ppm/) are ROM-derived content and are NOT uploaded. + +name: Self-hosted runtime + +on: + workflow_dispatch: + inputs: + levels: + description: "Sweep subset, SWEEP_LEVELS format (blank = all 21)" + required: false + default: "" + type: string + +permissions: + contents: read + +# One ROM job at a time per ref; never cancel an in-flight sweep out from +# under itself (the sweep is serialized by its own lockdir + watchdog). Group +# + cancel-in-progress:false is what actually serializes runs — GitHub +# Actions' workflow-level `concurrency` has no `max-parallel` key (that only +# exists under a job's `strategy:` for matrix jobs); a stray one here is a +# hard parse error that blocks every dispatch. +concurrency: + group: selfhosted-${{ github.ref }} + cancel-in-progress: false + +env: + ROM_CACHE_KEY: ge-rom-data-ntsc-v1 + +jobs: + build-and-sweep: + runs-on: [ self-hosted, windows ] + defaults: + run: + shell: msys2 {0} + env: + CCACHE_DIR: ${{ github.workspace }}/.ccache + CCACHE_MAXSIZE: 500M + steps: + - uses: actions/checkout@v7 + with: + persist-credentials: false + + # `release: false` reuses the box's own C:\msys64 as-is instead of + # downloading/installing MSYS2 fresh (verified against the action's + # source: with this set, and update/install both omitted, it does + # exactly one `uname -a` + a pacman.conf sed tweak — no pacman -Syu, + # no pacman -S, no network round-trip). This step still has to run + # every time — it's what registers the `msys2 {0}` shell wrapper + # above — but does no install/update work. + - uses: msys2/setup-msys2@v2 + with: + msystem: MINGW64 + release: false + + # Packages themselves are hand-maintained on the box (never installed + # by this workflow) — fails fast with exact install instructions if + # one's missing, instead of a confusing failure deeper in the build. + - name: Check self-hosted dependencies + run: tools_pc/check-selfhosted-deps.sh + + # --- ROM + sidecars: runner-managed cache, one-time seed -------------- + - name: Restore ROM + sidecars (runner cache) + uses: actions/cache/restore@v4 + with: + path: | + data/ge007.ntsc-final.z64 + data/pcmodels-ntsc-final + data/pccg-ntsc-final + key: ${{ env.ROM_CACHE_KEY }} + + - name: Check ROM store presence + id: romcheck + run: | + present=true + for f in ge007.ntsc-final.z64 pcmodels-ntsc-final pccg-ntsc-final; do + if [ ! -f "data/$f" ]; then echo "missing from cache: $f"; present=false; fi + done + echo "present=$present" >> "$GITHUB_OUTPUT" + + # {runner_home} = two levels up from RUNNER_WORKSPACE ({home}/_work/{repo}). + # NOT deleted after the cache save below (see that step's comment) -- + # kept as a durable fallback source until the cache save is proven to + # actually round-trip on this box. + - name: Seed ROM cache (from durable seed dir) + if: steps.romcheck.outputs.present == 'false' + run: | + seed="$(dirname "$(dirname "$RUNNER_WORKSPACE")")/_rom-seed" + [ -d "$seed" ] || { + echo "ROM cache is empty and no seed found at $seed." >&2 + echo "One-time bootstrap: copy ge007.ntsc-final.z64, pcmodels-ntsc-final," >&2 + echo "and pccg-ntsc-final into that directory on the runner, then re-run." >&2 + exit 2 + } + mkdir -p data + # -r: pcmodels-*/pccg-* are directories, not single files (found by + # testing — a plain cp failed with "omitting directory"). + cp -r "$seed/ge007.ntsc-final.z64" "$seed/pcmodels-ntsc-final" \ + "$seed/pccg-ntsc-final" data/ + echo "seeded ROM store from $seed" + + # Explicit save (restore alone never saves): persist the seeded store + # immediately, before the build/sweep can fail out from under it. + # KNOWN ISSUE (2026-09-12): this reliably logs a WARNING annotation + # ("Failed to save: ... tar.exe failed with exit code 2") on this box + # and never actually persists anything to the cache -- but + # actions/cache/save treats a save failure as non-fatal by design, so + # the step still reports success and the job carries on. Root cause not + # yet found (see docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md). Consequence: + # do NOT delete the seed dir after this step (previously done, and it + # meant a *silent* cache-save failure combined with the delete wiped + # the only copy of the ROM every run -- exactly what broke rollout + # step 5's repeated dispatches). The seed dir now stays in place + # indefinitely as a durable fallback; "Check ROM store presence" / + # "Seed ROM cache" above already re-copy from it whenever the cache + # comes back empty, so this is safe to leave as a no-op once the cache + # save is fixed for real. + - name: Save ROM cache + if: steps.romcheck.outputs.present == 'false' + uses: actions/cache/save@v4 + with: + path: | + data/ge007.ntsc-final.z64 + data/pcmodels-ntsc-final + data/pccg-ntsc-final + key: ${{ env.ROM_CACHE_KEY }} + + - name: Verify ROM integrity (committed US sha1) + run: | + cd data + want="$(cut -d' ' -f1 ../ge007.u.sha1)" + got="$(sha1sum ge007.ntsc-final.z64 | cut -d' ' -f1)" + [ "$want" = "$got" ] || { echo "ROM hash mismatch: want $want got $got" >&2; exit 2; } + echo "ROM sha1 ok: $got" + + # --- build ------------------------------------------------------------- + # Same opt-in ccache CMakeLists.txt wires up (speed-ups plan Step 5); + # persisted via actions/cache like ci.yml's windows-build. + - name: Restore ccache + uses: actions/cache/restore@v4 + with: + path: ${{ env.CCACHE_DIR }} + key: ccache-selfhosted-${{ hashFiles('CMakeLists.txt') }}-${{ github.sha }} + restore-keys: | + ccache-selfhosted-${{ hashFiles('CMakeLists.txt') }}- + ccache-selfhosted- + + - name: Regenerate ROM-asset symbol file (US) + run: python scripts/gen_romassets.py u + + - name: Build (ntsc-final) + run: ./build-pc.sh ntsc-final + + # verify.sh: build-if-stale -> headless GE_PCDUMP per level -> crash + # symbolication -> pixcount -> framediff vs tools_pc/golden/. Serialized + # + watchdog + one retry by design (D117: not frame-deterministic). + - name: Level sweep (all 21, or dispatch subset) + run: | + [ -n "${{ inputs.levels }}" ] && export SWEEP_LEVELS='${{ inputs.levels }}' + tools_pc/verify.sh sweep --json > sweep.log 2>&1 + rc=$? + cat sweep.log + exit $rc + + # Pull the JSON verdict array out of the mixed stdout for easy review. + - name: Extract JSON verdict + if: always() + run: | + python -c " + import json, sys + s = open('sweep.log', encoding='utf-8', errors='replace').read() + i = s.rfind('[{') + out = '[]' + if i >= 0: + d, _ = json.JSONDecoder().raw_decode(s[i:]) + out = json.dumps(d, indent=1) + open('sweep-verdict.json', 'w').write(out + '\n') + print(out) + " + + # Console logs + verdict only — no frames, no ROM bytes (leak rules). + - name: Upload sweep report + if: always() + uses: actions/upload-artifact@v7 + with: + name: selfhosted-sweep-${{ github.run_number }} + path: | + sweep.log + sweep-verdict.json + ge007.crash.log + if-no-files-found: ignore + retention-days: 14 diff --git a/docs/dev/findings-index.csv b/docs/dev/findings-index.csv index 3cdc9f90..6e33e90c 100644 --- a/docs/dev/findings-index.csv +++ b/docs/dev/findings-index.csv @@ -39,6 +39,7 @@ D229,OPEN,"Water on IsWater levels renders green and pulses blue<->green on an i D230,OPEN,In-level music quality:,OPEN — not investigated. D233,FIXED,All levels (intermittent):,"FIXED (port/fast3d/gfx_pc.cpp, gfx_sp_tri1) —" D234,OPEN,"Dam: guard towers along the dam structure invisible except the first one (M-106, user QA report).",OPEN — not root-caused. +D244,OPEN,Self-hosted CI runner (corpyvt6agent01) link failure:,"OPEN, not yet fixed. Next:" D228,OPEN,AK47 (and other) weapon models still show WHITE missing-texture patches after D217's texpool-alignment fix —,"OPEN — newly split out, not investigated." D204,FIXED,PC audio ran ~2 % below real time permanently:,"FIXED (port/src/audio.c, port/src/libultra.c, port/include/audio.h) —" D203,OPEN,"Steam Deck (SteamOS, x86_64 Linux) v0.1.0 release bundle:",OPEN — negative idle repros only; not yet reproduced with input. diff --git a/docs/dev/findings.md b/docs/dev/findings.md index 0556921c..3866e5dc 100644 --- a/docs/dev/findings.md +++ b/docs/dev/findings.md @@ -412,6 +412,7 @@ covers D24–D69; the log continues in §H (D32 procedure, D70–D121). | D230 | **In-level music quality: a bass instrument sounds corrupted on PC, correct on N64 reference (M-105, user reports; repro refined M-106).** Filed to give the post-D77 residual its own label — D77 (in-level music *absence*) is RESOLVED (PR #42, missing `return` in `sub_GAME_7F0C0BF0` zeroed the seqp volume on the only in-level trigger path), so "music doesn't play" claims in tracked docs are stale; what actually remains is this quality gap. **M-106 repro, deterministic (not intermittent):** Control's elevator track and its regular in-level track both carry the bad instrument throughout; Cavern's elevator track has it too; Runway's track is otherwise fine but the same instrument enters ~5–7s after the track starts and sounds bad from then on. User confirmed against original N64 reference audio that a bass instrument belongs there on Runway — this is **not** a wrong-instrument/wrong-bank selection (rules out the D77 trigger-path family and the seqp-volume fix's neighborhood); the instrument choice is correct, only its *sound* is wrong on PC. Determinism (same track, same spot, every time) points at a static decode/synthesis bug for this instrument's wavetable/ADPCM data — most likely low-frequency-specific (pitch-shift/downsample artifacting, a wrap/loop-point error, or a clipping/gain issue that only bites bass-range samples) — rather than a live mixing/voice-stealing issue (D147/D152/D202 family), which would be intermittent. The mixer layer and the trigger layer are both proven working by every track that sounds right, so the suspect neighborhood is the seq/instrument-bank decode path (`csplayer.c`/`synthesizer.c`, bank loading), not `port/` audio plumbing. | **OPEN — not investigated.** Repro: Control (elevator + regular track, throughout), Cavern (elevator, throughout), Runway (~5–7s in). Next: isolate the instrument/patch id these three tracks share and diff its wavetable/ADPCM decode against a known-good (non-bass) instrument on the same tracks. | | D233 | **All levels (intermittent): indoor level geometry invisible near doors — you can see through walls/floors into rooms you can't see into, while world objects (terminals, furniture) in that space still draw (M-106, user QA report; screenshots `docs/img/bugs/20260912-cullingbug-bugged.png` vs `-notbugged.png`).** Root-caused M-107 (session-016TaKy6, port/fast3d only): **fast3d's CPU-side "trivial reject" AND-test in `gfx_sp_tri1` (`port/fast3d/gfx_pc.cpp:1455-1460`) has a latent w-sign bug independent of D222/D106.** The per-vertex outcode bits computed in `gfx_sp_vertex` (`x<-w`/`x>w`/`y<-w`/`y>w`/`z>w`, `gfx_pc.cpp:1372-1386`) are only valid half-space tests when `w>0`; for a vertex that has crossed behind the camera plane (`w<0` — common when the camera is close to large room-structure polygons, e.g. walls/ceiling trusses near a doorway) the comparison sense is flipped, so its outcode can come out wrong. If that spurious bit happens to match the other two (genuinely correct) vertices' bits, `v1->clip_rej & v2->clip_rej & v3->clip_rej` fires and the whole triangle is dropped before it ever reaches OpenGL — even though GL's own homogeneous clipper (which *does* handle `w<0` correctly) would have rendered the visible part of it. Confirmed against the two screenshots: same room/camera/ammo-HUD, one frame is missing the room's ceiling trusses/walls (replaced by flat sky) while the console/terminal props in the same room still draw fully — exactly the "big background polys vanish, small prop models don't" split you'd expect, since large near-camera polygons are far more likely to have a vertex cross behind the eye than small self-contained prop meshes. The backface-cull code 6 lines below (`gfx_pc.cpp:1462-1495`) already special-cases this exact `w<0` hazard for its own cross-product sign (`(v1->w<0)^(v2->w<0)^(v3->w<0)`) — the trivial-reject block above it never got the same treatment. **Independent of D222** (D222 is a game-logic frustum-cull bug in `src/game/fr.c`/`bondview2.c` that only manifests at non-default `Video.FovScale`; this is a GPU-pipeline clip-rejection bug in `port/fast3d/` that fires at any FovScale, including the default 100, whenever the camera gets close to a large polygon — matching the report's "all levels, near doors" framing with no FovScale mentioned). Also independent of D106 (D106 fixed the *game-side* portal-visibility bbox computation for a different z==0 edge case in `bg.c`; this is the *renderer-side* per-triangle clip decision, a different stage of the pipeline entirely) — same neighbourhood/symptom vocabulary ("things vanish near a portal-adjacent boundary"), different bug. | **FIXED (port/fast3d/gfx_pc.cpp, `gfx_sp_tri1`) — high confidence on mechanism, UNVERIFIED visually.** Fix: skip the trivial-reject AND-test entirely whenever any of the triangle's 3 vertices has `w<0`, deferring to GL's clipper instead (mirrors the cull code's existing w-sign handling one block below). Change only ever *adds* triangles that were previously dropped — cannot make anything newly invisible, so it's a safe, narrowly-targeted fix with no game-logic touch (fast3d is PC-only code, no N64 build parity to preserve, no `#ifdef PORT` needed). **Build verified green** (`./build-pc.sh` equivalent — MSYS2 mingw64 cmake+ninja, full `ge007.x86_64.exe` link succeeds, only one pre-existing unrelated `-Wformat` warning in the same file). **Could not visually verify** — this worktree has no ROM in `./data/` and no interactive display, so `GE_PCDUMP`/`framediff` against a live Dam/doorway scene was not possible here. **A human must run:** build normally, then `-level_02` (Frigate, matches the screenshot's ship-corridor geometry) or any doorway-heavy interior with `GE_PCDUMP=` before/after this commit, walk up close to a room boundary/wall, and confirm the flicker is gone. | | D234 | **Dam: guard towers along the dam structure invisible except the first one (M-106, user QA report).** Investigated M-107 (session-016TaKy6, port/fast3d scope): **NOT root-caused — no fast3d-side mechanism found that plausibly explains "only the first of N repeated instances renders."** Ruled out: fast3d has no matrix/model instance cache that could explain a first-instance-only bug (the D217-family "keyed by address, not content" cache-bug class doesn't recur here — the *only* content-addressed cache in fast3d is the texture cache, already excluded from this investigation per the M-106 triage note); the modelview matrix stack (`gfx_pc.cpp:105`, depth 11, push properly gated `< 11` at `gfx_pc.cpp:1220`) is correctly bounded and would affect every level's nested-DL geometry, not just Dam's towers specifically; the D233 w<0 clip-rejection bug (above) doesn't fit either — towers are viewed from a distance along the dam, not close enough for a camera-behind-vertex condition, and D233's mechanism is per-triangle/per-frame flicker, not a stable "always the same instances missing" pattern. **Leading hypothesis, NOT verified:** GE's road/structure-instancing for repeated architecture typically goes through the room-graph's bytecode "vis program" (`VISOP_*` opcodes interpreted in `bg.c` ~L4450-4620, e.g. `VISOP_PUSH_IF_ROOM_IN_RANGE`, `VISOP_ADD_VISIBLE_ROOM`) or through per-instance room/portal definitions in the level's compiled BG data — both are level *data*, not code reachable via static analysis without the ROM (this worktree has none extracted). Could equally be a per-instance object/prop transform or cull-volume bug in `src/game/propobj.c`/`chrprop.c` (out of this task's `port/fast3d/` scope, and out of the `src/game/*` model/anim exclusion in the brief regardless). **No fix attempted — do not have enough evidence to touch code.** | **OPEN — not root-caused.** Next step needs either (a) a `GE_PCDUMP`/BG-data dump on `-level_10` (Dam) with the ROM present, comparing tower 1 vs tower 2+'s room/portal/object records, or (b) a targeted `port/src/preprocess/` or level-data probe distinguishing "renders once, never again" (matrix/cache bug) from "never queued for rendering" (visibility-program/data bug) for towers 2-N. Likely a separate bug from D233/D222, not the same root cause — see D233's entry for why. | +| D244 | **Self-hosted CI runner (`corpyvt6agent01`) link failure: `undefined reference to 'objectiveGetStatus_WEAK'` (2026-09-12, first-ever build attempt on the reinstalled box, run `34707798707`).** `src/game/objective_status.c:162` uses `#pragma weak objectiveGetStatus_WEAK = get_status_of_objective` (a decomp ground-truth alias, referenced 2-arg from `chrai.c:2538` — comment calls it "horrible hack to get ai matching"). The box's `g++`/`ld` reported as `x86_64-w64-mingw32/16.1.0`; a minimal local repro of the identical `#pragma weak` cross-TU function alias **links cleanly** on this session's own machine (`mingw-w64-x86_64-gcc 16.2.0-3`, `binutils 2.47-3`). Not a game-logic bug (the alias is unmodified decomp code, ground truth) and not a CMakeLists/link-flag issue (no `--gc-sections`/LTO in play) — the leading, unverified explanation is the self-hosted box's MSYS2 packages predate a PE/COFF weak-external fix or behavior change versus current MSYS2, consistent with rollout step 2's earlier finding that the box's mingw64 package set was stale/incomplete at initial setup. | **OPEN, not yet fixed.** Next: on the box, `pacman -Syu` (or at minimum `pacman -S mingw-w64-x86_64-gcc mingw-w64-x86_64-binutils`) to bring the toolchain current, then re-dispatch `selfhosted.yml`. If updating doesn't clear it, the fallback is a narrow `#ifdef PORT` replacement of the alias with an explicit thin wrapper function in `objective_status.c` (ABI-preserving, no logic change) — but try the toolchain update first since the local repro suggests this is a version/environment gap, not a fundamental PE/COFF limitation. | | D228 | **AK47 (and other) weapon models still show WHITE missing-texture patches after D217's texpool-alignment fix — a separate failure mode from the green wrong-palette defect (M-97, user playtest of PR #47, `-level_34` FACILITY).** Split out of D217 defect ①, exactly as the M-90 write-up anticipated ("do not close D217 defect ① on this merge — reopen/retitle as the AK47 + NPC-world-model white-texture residual"). **Playtest result that motivates the split:** with PR #47's `texInitPool` 8-byte arena alignment in, the *green* symptom is GONE — PP7/PPK grip and hammer render correct, Bond's watch face correct, no green cast on hands or the 3rd-person world model. But **NPC-held AK47s still render with white patches**, i.e. tiles sampling a missing/undecoded texture rather than a wrong-but-present palette. White ≠ wrong colour: a wrong palette binds a real TLUT with wrong contents (D217's mechanism, now fixed), whereas white is the signature of a texture that never got decoded/uploaded at all, or a tile descriptor pointing at an empty/cleared texpool slot. The M-87 matrix described these as distinct from the start ("AK47 viewmodel has white (missing-texture) patches on the magazine + barrel" vs. the green/blue colour corruption) — this playtest confirms they have distinct root causes, since fixing one left the other untouched. | **OPEN — newly split out, not investigated.** Next: `-level_34` with `GE_D217TEX=1`/`GE_TEXDUMP` while an AK47-carrying guard is on screen; check whether the AK47's affected tiles ever reach `import_texture` at all (a decode that never ran → white) versus reaching it with empty/zeroed source bytes (a decode that ran on a cleared arena slot). The texpool two-ended allocator (`image.c` `texLoad`, `leftpos`/`rightpos`) and the per-level weapon-model texture load path are the obvious suspects — an AK47 is not a level-start weapon on Facility, so its textures load later than the PP7's and may be hitting an exhausted or differently-based pool. Compare against a level where the AK47 IS a starting/early weapon. D217 · D161 lineage. | | D204 | **PC audio ran ~2 % below real time permanently: GE's AI feedback loop (`src/audi.c:531`) has a ~3 ms setpoint that PC scheduling jitter clears, so the SDL queue starves and the device pads playback with silence (FIXED, measured).** `amMain` wakes at 30 Hz (`sched.c:334` forwards every 2nd retrace to the audio client) and asks for `g_MinFrameSize`=720 samples per block = 21600/s against a 22050 Hz device — structurally 2 % short, relying on 784-sample top-up blocks that `audi.c` only requests once the reported AI length falls under ~69 frames (3 ms). Fine on N64 (double-buffered AI, exact VI interrupt); on PC the queue empties before the loop reacts, and the padded silence is unrecoverable time. **Fix (F5, port-only):** `audioGetAiLengthBytes()` subtracts a 1024-frame (~46 ms) cushion before reporting, so the loop tops up while slack remains; `audi.c`'s control law untouched (rule #2 clean). Measured A/B in one binary via `GE_D204_OLD`: **rt 0.980 → 1.000, q min 0 → 368, drop 0.** Also: F1 correct AI_LEN_REG single-buffer semantics (robustness, measures as a no-op), F3 `Audio.QueueLimit` 8192→2880 + non-silent drops (NB: an existing `ge007.ini` pins it), F4 oversize invariant guard, and `GE_D204=1` cheap audio-health monitor (`rt`/`q`/`drop`/`max`, safe to leave on for a full playtest). **Falsified in-session, do not re-open:** the u32 wrap at `audi.c:531` is real (high-water 2064 frames > the 789 threshold) but harmless — `frameSamples` is `s16` (`audi.c:145`), so it lands negative and audi.c's own lower clamp catches it; max block stays 3136 B vs the 3156 B allocation, no heap overrun. **Corrects two M-63 claims:** `soundIndex=109` is guards' return fire (count was double-logged; the "32.6 ms cadence" is just the 2880-byte block quantum every `dumppos` is rounded to) — closed negative; and M-63's "13 % of real time" was `GE_MIXERTRACE`'s own 25 MB unbuffered log starving the audio thread (same repro without it: 58.0 s / 60 s). Does NOT explain D202. | FIXED (`port/src/audio.c`, `port/src/libultra.c`, `port/include/audio.h`) — measured; a by-ear pass on a real playthrough still owed | | D203 | **Steam Deck (SteamOS, x86_64 Linux) v0.1.0 release bundle: Facility (`-level_34`) crashes "after loading in as James Bond" (user bug report).** Idle repro attempts on WSL Ubuntu with the actual `dist/goldeneye-pc-port-0.1.0-linux-x86_64.tar.gz` are **negative**: 120 s idle → frame 3300+ clean exit (the intro auto-advances, so "sitting as Bond" is covered), and a ~4-min stick-forward-only run → frame 6900+ also survived. So the trigger needs real gameplay input (fire/action) or is Deck-environment-specific. **Leading suspect: the D191 family** — both known v0.1.0 field crashes are SIGSEGVs in `modelGetNodeRwData` (`model.c:478`) over a truncated/NULL `root->Parent` node link, and both were triggered by *gameplay state* (first guard kill on Bunker ii; post-cutscene on Statue), never by idling; Facility opens into the Ourumov + soldier-squad scripted beat (D193a instance), which fits the trigger profile. Alternatives: Deck GL driver path (RADV/zink vs llvmpipe) or 40/50/60 Hz timing (D193 family). **Next:** (1) WSL run under gdb with a walk+fire `GE_INPUTSCRIPT` — generate the script into a file first (earlier attempts were blocked by a `bash -lc` quoting quirk that emptied the loop var, so no fire buttons ever went out); on SIGSEGV compare the fault PC against `modelGetNodeRwData`; (2) if not reproducible on WSL, get `ge007.crash.log` from the Deck (the Linux build writes it — proven by a stale one in the smoke tree) + ask what Bond was doing at crash time; optionally a gdb run from the SteamOS terminal. | OPEN — negative idle repros only; not yet reproduced with input. | diff --git a/tools_pc/check-selfhosted-deps.sh b/tools_pc/check-selfhosted-deps.sh new file mode 100644 index 00000000..88aec3d3 --- /dev/null +++ b/tools_pc/check-selfhosted-deps.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# +# Verify the self-hosted runner's MSYS2 MINGW64 toolchain is present. Does +# NOT install anything — the self-hosted workflows (ci.yml windows-build's +# push/dispatch path, selfhosted.yml) assume MSYS2 + these packages are +# maintained by hand on the box (docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md), +# not installed per run. Exits fast (0) when everything's already there; +# fails loudly with exactly what to install otherwise, instead of letting a +# missing package surface later as a confusing build/link error. +# +# Run from an MSYS2 MINGW64 shell: +# tools_pc/check-selfhosted-deps.sh +# +set -euo pipefail + +need_pkgs=( + mingw-w64-x86_64-toolchain + mingw-w64-x86_64-cmake + mingw-w64-x86_64-SDL2 + mingw-w64-x86_64-zlib + mingw-w64-x86_64-python + mingw-w64-x86_64-ccache + zip + unzip +) + +missing=() +for p in "${need_pkgs[@]}"; do + pacman -Qi "$p" >/dev/null 2>&1 || missing+=("$p") +done + +# mingw-w64-x86_64-toolchain is a meta-package; pacman -Qi on the group name +# can report "not found" even when its constituents (gcc, etc.) are present. +# Fall back to checking gcc directly for that one entry before failing on it. +if command -v gcc >/dev/null 2>&1; then + filtered=() + for p in "${missing[@]}"; do + [ "$p" = mingw-w64-x86_64-toolchain ] || filtered+=("$p") + done + missing=("${filtered[@]}") +fi + +if [ ${#missing[@]} -eq 0 ]; then + echo "self-hosted deps OK: MSYS2 MINGW64 toolchain, cmake, SDL2, zlib, python, ccache, zip, unzip all present" + exit 0 +fi + +{ + echo "Missing MSYS2/mingw64 packages on this runner:" + for p in "${missing[@]}"; do echo " - $p"; done + echo + echo "This box is expected to have these installed by hand — self-hosted" + echo "workflows do not auto-install (docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md)." + echo "From an MSYS2 MINGW64 shell:" + echo " pacman -S --needed ${missing[*]}" +} >&2 +exit 2 diff --git a/tools_pc/verify.sh b/tools_pc/verify.sh index 4e701908..89e2edf3 100644 --- a/tools_pc/verify.sh +++ b/tools_pc/verify.sh @@ -297,6 +297,16 @@ except Exception: fi local brief="$R_BRIEF" + if [ "$status" = NO-FRAMES ] && [ -f "$CAPDIR/run.log" ]; then + # stderr, not stdout: sweep mode captures verify_level's stdout via + # `out=$(...)` and only keeps its first line (`head -1`) as the verdict + # summary -- anything printed here on stdout would become that first + # line and clip the real verdict (and, worse, the --json line further + # down). stderr bypasses the capture and reaches the terminal/log file + # directly. + echo " -- $name run.log tail (NO-FRAMES, no crash -- likely startup/init failure) --" >&2 + tail -n 30 "$CAPDIR/run.log" | sed 's/^/ | /' >&2 + fi rm -rf "$CAPDIR" emit_verdict "$name" "$status" "$frames" "$worst" "$sym" "$note" if [ "$status" = CRASH ] && [ -n "$brief" ]; then @@ -330,9 +340,29 @@ case "$MODE" in out=$(verify_level "$n" "$num" "${DUMP:-80-400:40}" 45) rc=$? [ "$rc" -ne 0 ] && RC=1 + # Pre-existing bug (found alongside D244's diagnostics): `head -1` here + # silently discarded emit_verdict's --json line (its 2nd line of + # output, starting with `{`) on every sweep run -- RESULTS_JSON above + # was declared for exactly this and never actually populated, so + # `--json` sweeps have never produced a usable verdict array; the + # workflow's "Extract JSON verdict" step (scans for the last `[{` in + # sweep.log) always got an empty `[]`. echo "$out" | head -1 + if [ "$JSON" = 1 ]; then + j=$(echo "$out" | grep -m1 '^{') + [ -n "$j" ] && RESULTS_JSON+=("$j") + fi done echo "SWEEP DONE" + if [ "$JSON" = 1 ] && [ "${#RESULTS_JSON[@]}" -gt 0 ]; then + { printf '[' + for i in "${!RESULTS_JSON[@]}"; do + [ "$i" -gt 0 ] && printf ',' + printf '%s' "${RESULTS_JSON[$i]}" + done + printf ']\n' + } + fi exit $RC ;;