chore(agentic): speed-ups plan Steps 1, 4 (N2/R8/R9), 6

Static/tooling slice of docs/dev/notes/AGENTIC-SPEEDUPS-PLAN.md — no build or
runtime environment needed to verify any of it.

- Step 6 / N7: add a VERIFY: line to the investigation-brief template
  (docs/dev-process.md) and the CLAUDE.md dispatch-preflight checklist — the
  brief must now name the exact probe/verify invocation that proves the task
  done, or state that runtime verification is impossible here.
- Step 4 / N2: tools_pc/README.md — indexes the ~60 scripts with a
  living vs artifact marker so closed-finding artifacts can be archived.
- Step 4 / R8: tools_pc/gen_findings_index.py -> docs/dev/findings-index.csv
  (label, bucket, one_liner, status_raw) — grep-before-you-read aid for the
  ~200 KB finding log. Heuristic parser of the §F/§H tables.
- Step 4 / R9: tools_pc/gen_env_probes.py — re-greps live getenv("GE_*")
  sites and reports NEW/GONE drift vs docs/dev/GE-ENV-PROBES.md (keeps the
  curated prose, only checks the site map). Doc reconciled: 7 new probes
  added, GE_D54 kept as a tombstone.
- CI: validate job now runs both generators with --check.

Step 1 / N1 (HANDOFF preflight) applied in the local-only docs/HANDOFF.md.
Step 2 (verify.sh) and the HANDOFF/HANDOFF-ARCHIVE trim are left for a
session with a build environment.
This commit is contained in:
James Dansereau committed 2026-09-04 19:21:35 -04:00
1 parent 5c3944582b
commit dbea5a30fe
8 files changed
+427 -3

No files matched your search

+73
View File
@@ -0,0 +1,73 @@
# `tools_pc/` — PC-port dev & test scripts
Index for the ~60 scripts here (speed-ups plan Step 4 / N2). None of these are
part of the build; they are converters, verification tooling, and
investigation artifacts.
**Marker:**
- **`living`** — kept and re-run; part of the verification ritual, the release
flow, or a converter the build depends on (via the sidecars it emits).
- **`artifact`** — added to chase one finding. Safe to move to
`tools_pc/archive/<dxx>/` once that finding is closed and the technique is
captured in `docs/dev/findings.md` / `docs/porting-notes.md`.
## Living — converters the sidecars depend on
| Script | Role |
|---|---|
| `d43_emit.py` | Offline N64→PC model-file converter → RZ sidecar + `manifest.csv` (D50 / Plan B). Regen after any model-format change. |
| `d43_convert.py` | Reference single-file model converter + full-512 layout/pointer validator (D43). |
| `d69_emit.py` | Offline converter for stage `bg/*.seg` + `Tbg_*_stanZ` → concatenated sidecar (D69/D78–D82). |
| `d88_emit.py` | Offline converter for per-level `Usetup*Z` stage-setup files → appended to the `pccg.bin` sidecar (D88). |
| `d88_propdefs.py` | The `propDefs` polymorphic-record stream N64→PC converter used by `d88_emit.py` (D88.4). |
| `d69_emit.py` / `d88_emit.py` / `d43_emit.py` | Run all three (+ `d88_propdefs`) to rebuild `data/` sidecars from the ROM. |
## Living — verification & release
| Script | Role |
|---|---|
| `framediff.py` | Visual regression: candidate `GE_PCDUMP` frames vs `tools_pc/golden/`, per-region divergence. Structural/tolerant by default; `--exact` after `GE_DETERM`. |
| `pixcount.py` | Count non-black pixels in a PPM dump — "did the scene render anything" as a number. |
| `level_sweep.sh` | Bare `-level_XX` boot of all 21 solo levels → PASS / NO-FRAMES / CRASH. (Step 2 folds this into `verify.sh sweep`.) |
| `playtest.sh` | Launch a level for `docs/dev/LEVEL-PLAYTEST.md` human validation (WS6). |
| `debug.ps1` / `repro_gdb.sh` / `attach_animgen.sh` | Launch (or attach to) the game under gdb so a crash always leaves a backtrace. |
| `bundle-win.sh` / `bundle-linux.sh` | Package a built tree as a distributable archive (exe + licenses + `prepare-assets/`). |
| `romverify.c` | One-shot `.z64` integrity check against the repo's ground truths. |
| `dump_objectives.py` | Dump per-level objectives + win/fail criteria from the ROM (playtest validation; Step 10 / N8 acceptance oracle). |
| `disasm.py` | Minimal MIPS disassembler for the BE ROM (RAM-addr → file offset). |
| `ppm2bmp.py` | PPM → 24-bit BMP, no deps — eyeball `GE_PCDUMP` frames without PIL. |
| `gen_findings_index.py` | Regenerate `docs/dev/findings-index.csv` (grep-before-you-read aid for the 200 KB finding log). `--check` in CI-style use. |
| `gen_env_probes.py` | Drift check for `docs/dev/GE-ENV-PROBES.md` — re-greps live `getenv("GE_*")` sites, reports NEW/GONE. |
## Living — compiler-verified layout probes (kept: re-run when structs change)
| Script | Role |
|---|---|
| `d43_layoutprobe.c` | Prints `sizeof`/`offsetof` for every struct `d43_emit.py` must reproduce, built with the port toolchain. |
| `d88_layoutprobe.c` | Same, for the `PROPDEF_*` record structs `d88_emit.py` converts. |
| `mtxtest.c` | Standalone numerical test of the fast3d matrix pipeline conventions (GE swapped-perspective, packing, MUL order). |
## Investigation artifacts — D43 model-file format (finding closed; archive candidates)
`d43_chainbound.py`, `d43_cover.py`, `d43_decode.py`, `d43_fullwalk.py`,
`d43_gdldump.py`, `d43_gdlhist.py`, `d43_gdlorder.py`, `d43_gdlseq.py`,
`d43_invariants.py`, `d43_layout.py`, `d43_layout1.py`, `d43_lutscan.py`,
`d43_pointusage.py`, `d43_seg5.py`, `d43_seg5ops.py`, `d43_seg5vtx.py`,
`d43_seg5vtx2.py`, `d43_sizes.py`, `d43_sizes2.py`, `d43_tree_dump.py`,
`d43_vtxfmt.py`, `d43_walk.py` — layout/pointer/GDL-order analysis passes that
derived the D43 converter spec. `d43_sizes.py` is still `exec()`'d by a couple
of the others.
## Investigation artifacts — other closed/parked findings
| Script | Finding |
|---|---|
| `d125_check.py` | D125 — is the emitted `propDefs` blob byte-identical to `d88_propdefs.convert_stream()`? |
| `d88_propdef_scan.py` | D88.4 — histogram of `PROPDEF_*` record types across shipped levels. |
| `d51_gdb.py` / `dump_animgen.cmd` | D51 texture-tile crash capture / Facility-outro-hang anim-state dump (gdb scripts). |
## Other
- `dist/` — staged bundle README template (`README.md.in`) + tokens.
- `golden/` — per-level golden frames for `framediff.py` (Step 3 fills this out).
- `__pycache__/` — ignored.
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Drift check for docs/dev/GE-ENV-PROBES.md (speed-ups plan Step 4 / R9).
That doc is a hand-curated table of every `GE_*` env probe -- var, file:line,
what it does, live/dead. It was last snapshotted by a manual grep and drifts
as probes are added/removed. This script does NOT regenerate the prose (the
curation is the value); it re-greps the live `getenv("GE_...")` sites and
reports:
* NEW -- a var with live sites that the doc never names
* GONE -- a var the doc names that has no live site any more
* a fresh var -> file:line site map (paste into the doc's File:line cells)
Usage:
python tools_pc/gen_env_probes.py # print the drift report
python tools_pc/gen_env_probes.py --check # exit 1 if NEW or GONE
"""
from __future__ import annotations
import re
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
DOC = ROOT / "docs" / "dev" / "GE-ENV-PROBES.md"
SCAN_DIRS = ["src", "port"]
GETENV_RE = re.compile(r'getenv\(\s*"(GE_[A-Z0-9_]+)"\s*\)')
def live_sites() -> dict[str, list[str]]:
sites: dict[str, list[str]] = {}
for d in SCAN_DIRS:
for path in sorted((ROOT / d).rglob("*")):
if path.suffix not in {".c", ".cpp", ".h", ".def", ".inc"}:
continue
try:
lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
except OSError:
continue
for n, line in enumerate(lines, 1):
for m in GETENV_RE.finditer(line):
rel = path.relative_to(ROOT).as_posix()
sites.setdefault(m.group(1), []).append(f"{rel}:{n}")
return sites
def doc_vars() -> tuple[set[str], set[str]]:
"""Return (all vars named in the doc, vars whose row is an intentional tombstone)."""
if not DOC.exists():
return set(), set()
named, tombstone = set(), set()
for line in DOC.read_text(encoding="utf-8").splitlines():
vs = re.findall(r"GE_[A-Z0-9_]+", line)
named.update(vs)
if "tombstone" in line.lower():
tombstone.update(vs)
return named, tombstone
def main() -> int:
check = "--check" in sys.argv[1:]
sites = live_sites()
documented, tombstoned = doc_vars()
live = set(sites)
new = sorted(live - documented)
gone = sorted(v for v in documented - live if v not in ({"GE_DETERM"} | tombstoned))
out = []
out.append(f"# GE_* probe drift vs {DOC.relative_to(ROOT).as_posix()}")
out.append(f"# {len(live)} vars with live sites, {len(documented)} named in the doc\n")
if new:
out.append("## NEW -- live sites, not in the doc (add a row):")
for v in new:
out.append(f" {v}")
for s in sites[v]:
out.append(f" {s}")
if gone:
out.append("\n## GONE -- named in the doc, no live site (mark removed):")
out += [f" {v}" for v in gone]
if not new and not gone:
out.append("doc is in sync with live getenv sites (site line numbers still drift -- see map below)")
out.append("\n## full live site map:")
for v in sorted(sites):
out.append(f" {v}: " + ", ".join(sites[v]))
print("\n".join(out))
if check and (new or gone):
sys.stderr.write(f"\nGE-ENV-PROBES.md drift: {len(new)} new, {len(gone)} gone\n")
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
+157
View File
@@ -0,0 +1,157 @@
#!/usr/bin/env python3
"""Generate docs/dev/findings-index.csv from the §F/§H tables in findings.md.
Agentic speed-ups plan Step 4 / R8: findings.md is ~200 KB. Agents should grep
this CSV (label, status, one-liner) to decide *which* Dxx entries to actually
read, instead of linear-reading the log.
Heuristic parser: every markdown table row whose first cell matches a Dxx
label (`D12`, `D88.4`, `D75 · D77`, `D24-D30`, ...). The topic cell is
truncated to a one-liner; the status cell is normalised to a short bucket
(RESOLVED / OPEN / FIXED / PARTIAL / PROPOSED / ?) with the raw text kept.
Usage:
python tools_pc/gen_findings_index.py # write the CSV
python tools_pc/gen_findings_index.py --check # exit 1 if stale
python tools_pc/gen_findings_index.py --stdout # print, don't write
"""
from __future__ import annotations
import csv
import io
import re
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
SRC = ROOT / "docs" / "dev" / "findings.md"
OUT = ROOT / "docs" / "dev" / "findings-index.csv"
LABEL_RE = re.compile(r"^D\d+(?:\.\d+)?(?:[-–]D\d+)?(?:\s*[·,]\s*D\d+(?:\.\d+)?)*\+?$")
STATUS_BUCKETS = [
("FIXED", re.compile(r"\bfixed\b", re.I)),
("RESOLVED", re.compile(r"\bresolved\b", re.I)),
("PARTIAL", re.compile(r"\bpartial|clamp landed|landed\b|safety-net", re.I)),
("PROPOSED", re.compile(r"\bproposed|not applied|design\b", re.I)),
("OPEN", re.compile(r"\bopen\b|not started|awaiting|owed|pending", re.I)),
]
def one_liner(cell: str, limit: int = 160) -> str:
"""Strip markdown emphasis/code and collapse to a single short line."""
txt = cell.replace("**", "").replace("`", "")
txt = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", txt) # [t](u) -> t
txt = re.sub(r"\s+", " ", txt).strip()
# first sentence-ish
m = re.match(r"(.{20,}?[.—:])\s", txt)
if m and len(m.group(1)) <= limit:
txt = m.group(1)
if len(txt) > limit:
txt = txt[: limit - 1].rstrip() + "…"
return txt
LEAD_RE = re.compile(r"^\**\s*(OPEN|RESOLVED|FIXED|PROPOSED|PARTIAL|LANDED|CLAMP|DISPROVEN)", re.I)
def bucket(status_cell: str) -> str:
# An explicit leading verdict token wins over prose keywords deeper in the
# cell (e.g. "OPEN (parked). ... D168 fixed ..." is OPEN, not FIXED).
lead = LEAD_RE.match(status_cell.strip())
if lead:
tok = lead.group(1).upper()
return {"LANDED": "PARTIAL", "CLAMP": "PARTIAL", "DISPROVEN": "RESOLVED"}.get(tok, tok)
for name, rx in STATUS_BUCKETS:
if rx.search(status_cell):
return name
return "?"
def split_row(line: str) -> list[str] | None:
line = line.strip()
if not line.startswith("|"):
return None
# split on '|' that is not inside backticks
cells, buf, tick = [], [], False
for ch in line:
if ch == "`":
tick = not tick
buf.append(ch)
elif ch == "|" and not tick:
cells.append("".join(buf))
buf = []
else:
buf.append(ch)
cells.append("".join(buf))
cells = [c.strip() for c in cells]
# leading/trailing empty from outer pipes
if cells and cells[0] == "":
cells = cells[1:]
if cells and cells[-1] == "":
cells = cells[:-1]
return cells or None
def rows():
section = None
for raw in SRC.read_text(encoding="utf-8").splitlines():
h = re.match(r"^#+\s+([A-Z])\.\s", raw) or re.match(r"^#+\s.*§?([FH])\b", raw)
if raw.startswith("### ") and re.search(r"\b([FH])\.", raw):
section = re.search(r"\b([FH])\.", raw).group(1)
cells = split_row(raw)
if not cells or len(cells) < 2:
continue
label = cells[0].replace("–", "-")
if not LABEL_RE.match(cells[0]):
continue
topic = cells[1]
status = cells[2] if len(cells) > 2 else ""
yield {
"label": label,
"bucket": bucket(status),
"one_liner": one_liner(topic),
"status_raw": one_liner(status, 200) if status else "",
}
def build_csv() -> str:
buf = io.StringIO()
w = csv.DictWriter(
buf, fieldnames=["label", "bucket", "one_liner", "status_raw"], lineterminator="\n"
)
w.writeheader()
seen = set()
for r in rows():
key = r["label"]
if key in seen:
continue
seen.add(key)
w.writerow(r)
return buf.getvalue()
def main() -> int:
args = set(sys.argv[1:])
text = (
"# GENERATED by tools_pc/gen_findings_index.py -- do not hand-edit.\n"
"# Grep this to pick which findings.md entries to read; it is not authoritative.\n"
+ build_csv()
)
if "--stdout" in args:
sys.stdout.write(text)
return 0
want = text.encode("utf-8")
if "--check" in args:
cur = OUT.read_bytes() if OUT.exists() else b""
if cur != want:
sys.stderr.write("findings-index.csv is stale: re-run gen_findings_index.py\n")
return 1
return 0
OUT.write_bytes(want)
print(f"wrote {OUT.relative_to(ROOT)} ({text.count(chr(10))} lines)")
return 0
if __name__ == "__main__":
raise SystemExit(main())