level_sweep.sh: early-exit watchdog; M-22 re-sweep 20/21 PASS

The 45s watchdog is now a ceiling, not a flat per-level cost: a level is
killed as soon as its whole GE_PCDUMP frame set is on disk (PASS) or the
ppm count has been frozen for STALL_SECS (default 18). EXPECT_FRAMES is
derived from SWEEP_DUMP (start-end:stride). Full 21-level sweep ~7min
(was ~16).

M-22 full re-sweep after D134: 20/21 PASS. Facility(34) + Jungle(37) --
the M-20 re-verify failures -- both PASS clean, confirming D134 was their
cause. Only Cuba(54) crashes: known D129 residual (bare -level_54 hits the
cast/credits scroll referencing an unloaded text bank), parked with D76.
This commit is contained in:
James Dansereau committed 2026-08-29 23:57:03 -04:00
1 parent 46d4f03e5a
commit 78c0f72157
2 files changed
+46 -4

No files matched your search

+17 -2
View File
@@ -7,8 +7,23 @@ auto-injected, D121), `GE_PCDUMP="80-260:40"`, ~24 s watchdog then
0x140000000). "render %" = `tools_pc/pixcount.py` non-clear on the last
captured frame. Binary at `f2beae4b` + this session's build.
Last full re-sweep: 2026-08-29, after D123 (C1) + D124-Jungle (C2) landed.
Build = `f2beae4b` + `tools_pc/d88_propdefs.py` (D123) + `port/src/gimgfixup.c` (D124).
Last full re-sweep: 2026-08-29 (M-22), after D134 landed (SP/DP task-done
event no longer dropped into a full sched interruptQ — the "frame-2 hang"
that M-13..M-21 wrote off as D117/machine-load flakiness). Build = HEAD
`a8f27c16`. Optimized `level_sweep.sh` (early-exit once the GE_PCDUMP set
is on disk / STALL_SECS freeze) — full 21-level sweep now ~7 min.
**M-22 result: 20 / 21 PASS.** Every level loads + renders + survives the
capture window EXCEPT Cuba. Facility (34) and Jungle (37) — the M-20
re-verify failures — both PASS cleanly now, confirming D134 was their
cause. Only remaining crash:
- **Cuba (54)** — `textMeasure` (`textrelated.c:770`) ← `bondviewRenderCredits`
(`bondview2.c:8912`), AV @ `0x1400c3b77`, fault addr NULL, after ~frame
900. This is the **known D129 residual**: the bare `-level_54` boot
reaches the cast/credits scroll referencing a text bank the real
front-end flow would have loaded. Not a real-flow blocker; Cuba loads +
renders 300+ frames fine. Parked with D76 (`docs/GRAPHICS-BACKLOG.md`).
| Level | # | Status | Crash site | Class |
|---|---|---|---|---|
+29 -2
View File
@@ -8,6 +8,11 @@
# uses a 45 s watchdog, and it RETRIES a NO-FRAMES row once before reporting it.
# Do not run it next to a build.
#
# Speed (M-22): the 45 s watchdog is a CEILING, not a fixed cost. A level that
# has produced its whole GE_PCDUMP frame set is killed immediately (PASS), and a
# level that renders a few frames then stops moving is cut after ~STALL_SECS. So
# a healthy 21-level sweep now finishes in a few minutes instead of ~16.
#
# Env overrides:
# SWEEP_SECS=45 per-level watchdog seconds
# SWEEP_DUMP=80-400:40 GE_PCDUMP window
@@ -16,6 +21,17 @@ cd "$(dirname "$0")/.."
SWEEP_SECS=${SWEEP_SECS:-45}
SWEEP_DUMP=${SWEEP_DUMP:-80-400:40}
STALL_SECS=${STALL_SECS:-18} # frames stopped growing this long -> stop early
# how many .ppm files a complete GE_PCDUMP window produces (start-end:stride)
_d=${SWEEP_DUMP%%:*}; _stride=${SWEEP_DUMP##*:}
_start=${_d%%-*}; _end=${_d##*-}
if [ "$_start" -ge 0 ] 2>/dev/null && [ "$_stride" -gt 0 ] 2>/dev/null; then
EXPECT_FRAMES=$(( (_end - _start) / _stride + 1 ))
else
EXPECT_FRAMES=4
fi
[ "$EXPECT_FRAMES" -lt 1 ] && EXPECT_FRAMES=1
LV=${SWEEP_LEVELS:-"Dam:33 Facility:34 Runway:35 Surface1:36 Bunker1:09 Silo:20 Frigate:26 Surface2:43 Bunker2:27 Statue:22 Archives:24 Streets:29 Depot:30 Train:25 Jungle:37 Control:23 Caverns:39 Cradle:41 Aztec:28 Egypt:32 Cuba:54"}
OUT=/tmp/sweep_results
@@ -46,10 +62,21 @@ run_level() {
GE_PCDUMP="$SWEEP_DUMP" ./build-pc/ge007.x86_64.exe -level_$num > $LOGS/$name.log 2>&1 &
local pid=$!
# watchdog: poll so a clean early exit does not cost the full window
local waited=0
# watchdog: poll so we do not pay the full window when the outcome is already
# known -- exit as soon as the whole dump set is on disk (PASS), or the frame
# count has been frozen for STALL_SECS (level rendered a bit then wedged), or
# the process exits on its own (clean exit / crash).
local waited=0 fcnt=0 prev=0 stall=0
while [ $waited -lt $SWEEP_SECS ] && kill -0 $pid 2>/dev/null; do
sleep 1; waited=$((waited + 1))
fcnt=$(ls ppm/*.ppm 2>/dev/null | wc -l)
[ "$fcnt" -ge "$EXPECT_FRAMES" ] && break
if [ "$fcnt" -gt 0 ] && [ "$fcnt" -eq "$prev" ]; then
stall=$((stall + 1)); [ $stall -ge $STALL_SECS ] && break
else
stall=0
fi
prev=$fcnt
done
taskkill //F //IM ge007.x86_64.exe > /dev/null 2>&1
wait $pid 2>/dev/null