forked from Ports/goldeneye-pc-port
James Dansereau
3dcceaafd9
* ci: route windows-build to self-hosted runner (G2); land ROM-gated sweep workflow (G1)
G2: windows-build now runs on the self-hosted box (corpyvt6agent01) for
push/workflow_dispatch, using local ccache instead of the actions/cache
round-trip and skipping msys2/setup-msys2's install once packages are
already present. pull_request always stays on windows-latest — fork PRs
must never reach a box that will hold a ROM.
G1: adds .github/workflows/selfhosted.yml, a dispatch-only workflow that
builds + runs the 21-level headless sweep (tools_pc/verify.sh sweep --json)
against a runner-cached ROM. Inert until manually dispatched.
Still pending (manual, physical steps on the box — not done by this
commit, see docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md rollout table):
- register corpyvt6agent01 as a runner against this repo
- confirm MSYS2 at C:\msys64 with the required mingw64 package set
- one-time ROM seed into {runner_home}/_rom-seed/
- first workflow_dispatch sweep + parity check against a local run
- flip G1 from workflow_dispatch to push:[main] once green
Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU
* fix(selfhosted.yml): remove invalid concurrency.max-parallel key
Found by testing: dispatching the workflow failed with a hard parse
error (HTTP 422, "Unexpected value 'max-parallel'"). GitHub Actions'
workflow-level `concurrency` block only supports `group` and
`cancel-in-progress` — `max-parallel` is a job `strategy:` key for
matrix jobs, not a concurrency-block key. Serialization was already
achieved via group + cancel-in-progress:false; the stray key just
blocked every dispatch.
Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU
* fix(selfhosted.yml): register the msys2 shell wrapper (setup-msys2 step)
Found by testing: the first fixed dispatch failed every msys2-shell step
with "msys2: command not found". Unlike ci.yml's windows-build, this
workflow used `shell: msys2 {0}` without ever running msys2/setup-msys2,
which is what registers that shell alias for the runner — MSYS2 being
pre-installed on the box doesn't help if Actions doesn't know the shell
exists. Adds the same setup-msys2 step ci.yml uses, with update:false
since the packages are expected already present (fast no-op per plan).
Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU
* ci: stop reinstalling MSYS2 per run on the self-hosted box
The self-hosted paths (selfhosted.yml; ci.yml windows-build's push/
dispatch route) no longer call msys2/setup-msys2 or pacman -S at all.
MSYS2 + the mingw64 package set are now assumed hand-maintained on
corpyvt6agent01 (docs/dev/notes/SELFHOSTED-RUNNER-PLAN.md), invoked by
calling its bash directly (`C:\msys64\usr\bin\bash.exe -leo pipefail
"{0}"`, the same invocation setup-msys2's generated wrapper uses) with
MSYSTEM=MINGW64 so /etc/profile still puts mingw64/bin on PATH.
Adds tools_pc/check-selfhosted-deps.sh: a fast, install-free check that
exits 0 immediately when the required packages (toolchain, cmake,
SDL2, zlib, python, ccache, zip, unzip) are present, or fails with the
exact `pacman -S --needed ...` line to run otherwise — so a missing
package on the box surfaces immediately with a clear fix, instead of
as a confusing failure deeper in the build.
ci.yml's windows-latest (PR) path is unchanged: ephemeral runners still
get a full msys2/setup-msys2 install every time, since there's nothing
to persist there.
Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU
* fix: use msys2/setup-msys2 release:false instead of a raw bash-path shell
The previous attempt (invoking C:\msys64\usr\bin\bash.exe directly as a
custom `shell:` command) failed on the actual runner with "Second path
fragment must not be a drive or UNC name" — a real limitation, not a
typo; it failed identically in both files, including the one with a
static (non-templated) shell string, so it wasn't the ${{ }} expression
either.
Root-caused by reading msys2/setup-msys2's actual source: `release:
false` makes it reuse the existing C:\msys64 as-is (hardcoded path in
that branch, confirmed in main.js) instead of downloading+installing a
fresh copy, and — verified in the same source — omitting `update` and
`install` skips every pacman call entirely (just one `uname -a` + a
pacman.conf sed tweak, no network). This is the officially documented
mechanism for self-hosted runners with a pre-existing MSYS2 install
(see the action's README, "release" section).
ci.yml's windows-build now runs setup-msys2 unconditionally but with
release/update/install all keyed off github.event_name: full install on
windows-latest (PR, ephemeral), reuse-only on the self-hosted box
(push/dispatch). selfhosted.yml always uses release:false. Both then
run tools_pc/check-selfhosted-deps.sh to verify the box's hand-
maintained packages are actually present.
Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU
* fix(selfhosted.yml): cp -r for the ROM sidecar directories
Found by testing with the ROM seed actually in place: pcmodels-ntsc-final
and pccg-ntsc-final are directories (not single files, unlike
ge007.ntsc-final.z64), so a plain `cp` failed with "omitting directory"
and left the seed step failing on every run even with the seed populated.
Claude-Session: https://claude.ai/code/session_01J5PZ3sTHiuW6MdQnepasSU
* ci: drive MSYS2 MINGW64 directly via ci-shell.cmd, drop msys2 shell shim
The self-hosted runs were dying with 'msys2: command not found' because the
msys2/setup-msys2 action's shell shim was never registered on the box. Replace
the dependency on that shim with a small committed launcher (tools_pc/ci-shell.cmd)
that headlessly reproduces the local 'MSYS2 MINGW64' terminal:
- MSYSTEM=MINGW64 + /mingw64/bin on PATH
- NON-login bash, so it keeps the workspace cwd (a login shell cd's to $HOME
and would break every relative path in the workflow)
- -e -o pipefail for GitHub-style error propagation
Both workflows now set defaults.run.shell to C:\msys64\ci-shell.cmd {0}; a
bootstrap step (shell: cmd) refreshes the launcher from the repo into C:\msys64
each run. selfhosted.yml drops setup-msys2 entirely; ci.yml keeps it only for
the ephemeral windows-latest PR path (which installs MSYS2 at C:\msys64).
.gitattributes: force CRLF for *.cmd so cmd.exe parses the launcher correctly.
* Revert "ci: drive MSYS2 MINGW64 directly via ci-shell.cmd, drop msys2 shell shim"
This reverts commit a976b2589c.
* docs(D244): log self-hosted-runner link failure — undefined reference to objectiveGetStatus_WEAK
First real build attempt on the reinstalled corpyvt6agent01 box (run
34707798707) fails at link with an unresolved #pragma weak alias. A local
repro of the identical alias pattern links fine on current MSYS2
(gcc 16.2.0/binutils 2.47); the box reports gcc 16.1.0 — likely a stale
toolchain, consistent with rollout step 2's earlier package-hygiene gap.
Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3
* fix(verify.sh): surface run.log tail on NO-FRAMES instead of silently discarding it
sweep mode's per-level run.log (the game process's own stdout/stderr, where
an SDL/GL init failure would show up) was captured into a mktemp capdir but
unconditionally rm -rf'd with no diagnostic ever printed for the NO-FRAMES
(no-crash) case -- every level failed this way on the self-hosted runner's
first real sweep attempt with zero clue why.
Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3
* fix(selfhosted.yml): stop deleting the ROM seed dir -- cache save silently fails on this box
actions/cache/save has been failing to persist the ROM cache entry on
corpyvt6agent01 (WARNING annotation, tar.exe exit code 2) but doesn't fail
the step -- by design, a cache-save failure is non-fatal, so the workflow
carried on and the now-unconditional "Delete one-time seed dir" step wiped
the only copy of the ROM every single run. Every dispatch since has failed
at "Seed ROM cache" with an empty seed dir and an empty cache -- exactly
the user-reported "it wipes out the ROM data each run".
Fix: don't delete the seed dir. It's cheap disk (three files, a few hundred
MB) and the existing restore-then-seed-if-missing logic already treats it as
the fallback source whenever the cache comes back empty, so leaving it in
place is safe and makes the workflow tolerant of the cache-save bug until
that's root-caused separately.
Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3
* fix(verify.sh): sweep --json never actually produced a verdict array; fix NO-FRAMES diagnostic clipping it further
Two bugs, found together while chasing the self-hosted runner's first real
sweep attempt:
1. sweep mode's `out=$(verify_level ...); echo "$out" | head -1` always
discarded emit_verdict's second (--json) output line -- RESULTS_JSON was
declared for assembling a verdict array but never populated or printed,
so `sweep --json` has never produced anything but `[]` for the
workflow's "Extract JSON verdict" step. Now builds and prints a real
`[{...}, {...}]` array before the RC exit.
2. My own D244-session addition (printing a NO-FRAMES level's run.log tail)
wrote to stdout, which the same `out=$(...)/head -1` capture turned into
the *new* first line, clipping the real verdict summary. Moved to stderr,
which bypasses that capture.
Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3
* docs: regenerate findings-index.csv for D244
Claude-Session: https://claude.ai/code/session_01CUFSnpgrAyQSDVHoQQ6Sx3
tools_pc/ — PC-port dev & test scripts
Index for the ~60 scripts here (speed-ups plan Step 4 / N2). None of these are part of the build; they are converters, verification tooling, and investigation artifacts.
Marker:
living— kept and re-run; part of the verification ritual, the release flow, or a converter the build depends on (via the sidecars it emits).artifact— added to chase one finding. Safe to move totools_pc/archive/<dxx>/once that finding is closed and the technique is captured indocs/dev/findings.md/docs/porting-notes.md.
Living — converters the sidecars depend on
| Script | Role |
|---|---|
d43_emit.py |
Offline N64→PC model-file converter → RZ sidecar + manifest.csv (D50 / Plan B). Regen after any model-format change. |
d43_convert.py |
Reference single-file model converter + full-512 layout/pointer validator (D43). |
d69_emit.py |
Offline converter for stage bg/*.seg + Tbg_*_stanZ → concatenated sidecar (D69/D78–D82). |
d88_emit.py |
Offline converter for per-level Usetup*Z stage-setup files → appended to the pccg.bin sidecar (D88). |
d88_propdefs.py |
The propDefs polymorphic-record stream N64→PC converter used by d88_emit.py (D88.4). |
d69_emit.py / d88_emit.py / d43_emit.py |
Run all three (+ d88_propdefs) to rebuild data/ sidecars from the ROM. |
Living — verification & release
| Script | Role |
|---|---|
verify.sh |
Speed-ups plan Step 2. One command → one verdict: build-if-stale, headless GE_PCDUMP/GE_INPUTSCRIPT capture, crash-detect + symbolication (via crash_brief.py), pixcount, framediff vs golden, --json. verify.sh <level>, verify.sh sweep [subset], verify.sh parity <level> --against <dir>. |
crash_brief.py |
Speed-ups plan Step 9 / R5. ge007.crash.log → symbolicated frames + a dev-process.md-shaped dispatch-ready brief: matches porting-notes.md sections / findings-index.csv entries by resolved function/file, flags known parked-crash signatures (Cuba credits, D188 va_list, D189 stack overrun) so agents stop re-investigating them. Called automatically by verify.sh on CRASH; also runnable standalone. |
framediff.py |
Visual regression: candidate GE_PCDUMP frames vs tools_pc/golden/, per-region divergence. Structural/tolerant by default; --exact after GE_DETERM. |
pixcount.py |
Count non-black pixels in a PPM dump — "did the scene render anything" as a number. |
level_sweep.sh |
Bare -level_XX boot of all 21 solo levels → PASS / NO-FRAMES / CRASH. Predates verify.sh sweep; kept as the battle-tested full-21-level runner until verify.sh sweep has done an equivalent soak. |
playtest.sh |
Launch a level for docs/dev/LEVEL-PLAYTEST.md human validation (WS6). |
audiodebug.ps1 |
Audio counterpart to debug.ps1 (that one is for crashes; this one is for audio behaviour). Drives the GE_D204 health monitor + GE_AUDIOTRACE/GE_AUDIODUMP and prints a pass/fail verdict: real-time ratio, queue starvation, dropped blocks, oversized blocks, plus a soundIndex histogram. -AB runs the same binary twice (with and without GE_D204_OLD) and prints a before/after table — how D204 was measured. -Play for an instrumented interactive playtest (the D202 workflow), -Soak for a 5-min stability run, -SyncData to mirror ./data first. |
debug.ps1 / repro_gdb.sh / attach_animgen.sh |
Launch (or attach to) the game under gdb so a crash always leaves a backtrace. |
bundle-win.sh / bundle-linux.sh |
Package a built tree as a distributable archive (exe + licenses + prepare-assets/). |
romverify.c |
One-shot .z64 integrity check against the repo's ground truths. |
dump_objectives.py |
Dump per-level objectives + win/fail criteria from the ROM (playtest validation; Step 10 / N8 acceptance oracle). |
disasm.py |
Minimal MIPS disassembler for the BE ROM (RAM-addr → file offset). |
ppm2bmp.py |
PPM → 24-bit BMP, no deps — eyeball GE_PCDUMP frames without PIL. |
gen_findings_index.py |
Regenerate docs/dev/findings-index.csv (grep-before-you-read aid for the 200 KB finding log). --check in CI-style use. |
gen_env_probes.py |
Drift check for docs/dev/GE-ENV-PROBES.md — re-greps live getenv("GE_*") sites, reports NEW/GONE. |
Living — compiler-verified layout probes (kept: re-run when structs change)
| Script | Role |
|---|---|
d43_layoutprobe.c |
Prints sizeof/offsetof for every struct d43_emit.py must reproduce, built with the port toolchain. |
d88_layoutprobe.c |
Same, for the PROPDEF_* record structs d88_emit.py converts. |
mtxtest.c |
Standalone numerical test of the fast3d matrix pipeline conventions (GE swapped-perspective, packing, MUL order). |
Investigation artifacts — D43 model-file format (finding closed; archive candidates)
d43_chainbound.py, d43_cover.py, d43_decode.py, d43_fullwalk.py,
d43_gdldump.py, d43_gdlhist.py, d43_gdlorder.py, d43_gdlseq.py,
d43_invariants.py, d43_layout.py, d43_layout1.py, d43_lutscan.py,
d43_pointusage.py, d43_seg5.py, d43_seg5ops.py, d43_seg5vtx.py,
d43_seg5vtx2.py, d43_sizes.py, d43_sizes2.py, d43_tree_dump.py,
d43_vtxfmt.py, d43_walk.py — layout/pointer/GDL-order analysis passes that
derived the D43 converter spec. d43_sizes.py is still exec()'d by a couple
of the others.
Investigation artifacts — other closed/parked findings
| Script | Finding |
|---|---|
d125_check.py |
D125 — is the emitted propDefs blob byte-identical to d88_propdefs.convert_stream()? |
d88_propdef_scan.py |
D88.4 — histogram of PROPDEF_* record types across shipped levels. |
d51_gdb.py / dump_animgen.cmd |
D51 texture-tile crash capture / Facility-outro-hang anim-state dump (gdb scripts). |
Other
dist/— staged bundle README template (README.md.in) + tokens.golden/— per-level golden frames forframediff.py(Step 3 fills this out).__pycache__/— ignored.