From cb20bb508bec1083f48073519ff5f5e835f40a6a Mon Sep 17 00:00:00 2001 From: Ludovic Marie <136255825+ludovic111@users.noreply.github.com> Date: Tue, 27 Oct 2026 04:31:40 +0100 Subject: [PATCH] remote.sh: runs get CPUs 0-3 by default; benchmarks hold them with --exclusive playhorizon's CI runner has CPUs 4-7, and remote.sh pinned agents' runs there too: CPUs 4-7 sat at 100% while 0-3 idled, and window harnesses in pull requests' parity jobs lost ticks. Runs share a lock per CPU set; an --exclusive run (a benchmark) holds it alone. world_reopen also waits for the world to load before its first count, as after each reopen. Co-Authored-By: Claude Opus 5.5 --- docs/parity-workflow.md | 2 +- docs/release-flow.md | 4 ++-- tools/remote.sh | 37 ++++++++++++++++++++++--------- tools/test_world_reopen_parity.py | 1 + 4 files changed, 30 insertions(+), 14 deletions(-) diff --git a/docs/parity-workflow.md b/docs/parity-workflow.md index 6cb235c3c..a18f84de9 100644 --- a/docs/parity-workflow.md +++ b/docs/parity-workflow.md @@ -46,7 +46,7 @@ Harnesses a host can't run are listed in the file: `local_only` (`save_roundtrip ### Any command on playhorizon: `tools/remote.sh` -For Linux work that doesn't need a whole parity run (a crate's tests, a few harnesses, a server build), `tools/remote.sh ` runs the command on playhorizon at the worktree's commit and streams its output back; its exit status is the command's. Commit and push first: the host fetches the commit from GitHub, and nothing else leaves this machine (no files, no credentials, no environment). Each worktree keeps one checkout there, `~/remote-work/` with its own `target/`, so the next run only rebuilds what changed, and one run per name goes at a time. The command runs niced in the host's CI slice, pinned to CPUs 4-7 (`--cpus`; 0-3 stay free for benchmarks), with scratch game and launcher homes and the vanilla data linked as CI links it; a command naming `parity`, `lan_e2e` or `horizon-prepare` also gets `.local/horizon-client`, prepared as CI prepares it. Whatever the command leaves running (a game, a daemon) is stopped when it ends, and stopping `remote.sh` (or losing the connection) stops the command on the host. Logs stay in `~/remote-work/.logs/`; `--list` shows the checkouts, `--gc [DAYS]` removes old ones. +For Linux work that doesn't need a whole parity run (a crate's tests, a few harnesses, a server build), `tools/remote.sh ` runs the command on playhorizon at the worktree's commit and streams its output back; its exit status is the command's. Commit and push first: the host fetches the commit from GitHub, and nothing else leaves this machine (no files, no credentials, no environment). Each worktree keeps one checkout there, `~/remote-work/` with its own `target/`, so the next run only rebuilds what changed, and one run per name goes at a time. The command runs niced in the host's CI slice, pinned to CPUs 0-3 (`--cpus`): the CI runner has 4-7, so these runs never slow a pull request's parity job. A benchmark asks for its CPUs with `--cpus 0-3 --exclusive` (and says so in its pull request): it starts only when no other run uses them, and later runs wait for it to end. The command gets scratch game and launcher homes and the vanilla data linked as CI links it; a command naming `parity`, `lan_e2e` or `horizon-prepare` also gets `.local/horizon-client`, prepared as CI prepares it. Whatever the command leaves running (a game, a daemon) is stopped when it ends, and stopping `remote.sh` (or losing the connection) stops the command on the host. Logs stay in `~/remote-work/.logs/`; `--list` shows the checkouts, `--gc [DAYS]` removes old ones. ```sh tools/remote.sh .claude/worktrees/my-fix cargo test -p horizon-world --lib # 33 s, warm diff --git a/docs/release-flow.md b/docs/release-flow.md index e6a0a3856..1917fddd4 100644 --- a/docs/release-flow.md +++ b/docs/release-flow.md @@ -108,8 +108,8 @@ and its failures bisect and comment on merges. AMD EPYC threads, 30 GB, in Detroit; reachable only through the maintainer's tailnet). Nobody sits at it, so the watcher never pauses it except by hand. It gets half the machine: `RUNNER_MEMORY=24`, `CARGO_BUILD_JOBS` and the harness slots at 4, and a drop-in (`~/.config/systemd/user/horizon-runner.service.d/cpus.conf`) that starts it under -`taskset -c 4-7`, so CPUs 0-3 stay free for benchmarks pinned there (the user slice can't set `AllowedCPUs`: systemd -doesn't delegate the cpuset controller to it). +`taskset -c 4-7` (the user slice can't set `AllowedCPUs`: systemd doesn't delegate the cpuset controller to it). +Agents' `tools/remote.sh` runs and benchmarks share CPUs 0-3, so they never slow a pull request's job. Without Nix, `linux-runner.sh` takes the build libraries, the JDK, sccache, Xvfb and Mesa's software Vulkan (lavapipe; there is no GPU) from apt, and Python 3.13 from uv. The partner can reach the machine, so gh is not logged in there: the registration token comes from another machine (`RUNNER_TOKEN`), and no job with secrets may diff --git a/tools/remote.sh b/tools/remote.sh index 0955666cb..fa913e7cc 100755 --- a/tools/remote.sh +++ b/tools/remote.sh @@ -1,9 +1,10 @@ #!/usr/bin/env bash # Run a command on a Linux build host (default playhorizon) at a worktree's commit, and get its output back. # -# tools/remote.sh [--host H] [--name N] [--cpus LIST] [--jobs J] +# tools/remote.sh [--host H] [--name N] [--cpus LIST] [--exclusive] [--jobs J] # tools/remote.sh .claude/worktrees/my-fix cargo test -p horizon-world --lib # tools/remote.sh . python3 tools/parity.py food light +# tools/remote.sh --cpus 0-3 --exclusive . python3 tools/perf_bench.py # a benchmark: CPUs 0-3 to itself # tools/remote.sh --list # checkouts on the host, their size and last use # tools/remote.sh --gc [DAYS] # remove checkouts unused for DAYS (default 2) # @@ -18,19 +19,24 @@ # What the command leaves running is stopped when it ends; stopping this script stops the command on the host. # The exit status is the command's. # +# CPUs: playhorizon's CI runner has CPUs 4-7 (a systemd drop-in pins it, docs/release-flow.md), so runs here get +# 0-3 by default and never slow a pull request's parity job. A run holds its CPU set's lock shared; --exclusive (a +# benchmark: name it with --cpus and say so in its pull request) holds it alone, so it starts only when no other run +# uses those CPUs, and the others wait for it to end. --cpus 4-7 or all compete with CI: only when it is idle. +# # The host is lent by a partner who keeps hypervisor access: never send secrets, tokens or credentials there, # and never treat a result from it as a CI result. set -euo pipefail -# CPUs 0-3 stay free for benchmarks (perf-world-entry measures on playhorizon pinned to them); --cpus all for every one. -host=playhorizon name="" jobs="" cpus=4-7 -usage() { sed -n '2,8s/^# \{0,1\}//p' "$0" >&2; exit 2; } +host=playhorizon name="" jobs="" cpus=0-3 exclusive=0 +usage() { sed -n '2,9s/^# \{0,1\}//p' "$0" >&2; exit 2; } while [ $# -gt 0 ]; do case $1 in --host) host=$2; shift 2;; --name) name=$2; shift 2;; --jobs) jobs=$2; shift 2;; --cpus) cpus=$2; shift 2;; + --exclusive) exclusive=1; shift;; --list) exec ssh "ludovic@$host" 'cd ~/remote-work 2>/dev/null && for d in */; do d=${d%/}; printf "%-32s %8s last used %s\n" "$d" "$(du -sh "$d" 2>/dev/null | cut -f1)" "$(stat -c %y "$d.logs" 2>/dev/null | cut -c1-16)"; done';; --gc) days=${2:-2}; exec ssh "ludovic@$host" "cd ~/remote-work 2>/dev/null || exit 0 @@ -58,7 +64,8 @@ command=$(printf '%q ' "$@") # The host side reads its script and these four values on stdin (nothing is re-split by a remote shell); its # own environment comes from the runner's files there. { -printf 'name=%q sha=%q jobs=%q cpus=%q command=%q\n' "$name" "$sha" "$jobs" "$cpus" "$command" +[[ $cpus =~ ^[0-9,-]+$|^all$ ]] || { echo "remote.sh: --cpus takes a list such as 0-3, or all" >&2; exit 2; } +printf 'name=%q sha=%q jobs=%q cpus=%q exclusive=%q command=%q\n' "$name" "$sha" "$jobs" "$cpus" "$exclusive" "$command" cat <<'REMOTE' set -euo pipefail if [ "$cpus" = all ]; then pin=(); jobs=${jobs:-$(nproc)}; else pin=(taskset -c "$cpus"); jobs=${jobs:-$(taskset -c "$cpus" nproc)}; fi @@ -71,19 +78,27 @@ unset WAYLAND_DISPLAY mkdir -p ~/remote-work && cd ~/remote-work exec 9> "$name.lock" flock -n 9 || { echo "remote.sh: $name is busy with another run" >&2; exit 75; } +# The CPU set's lock: shared by ordinary runs, held alone by an --exclusive one (a benchmark). +exec 8> ".cpus-$cpus.lock" +if [ "$exclusive" = 1 ]; then + flock -x -n 8 || { echo "remote.sh: other runs use CPUs $cpus; try again when they are done (--list)" >&2; exit 75; } +elif ! flock -s -n 8; then + echo "remote.sh: an exclusive run (a benchmark) has CPUs $cpus; waiting for it to end" >&2 + flock -s 8 +fi if [ ! -d "$name/.git" ]; then - git clone -q --filter=blob:none --no-checkout https://github.com/ludovic111/horizon "$name" 9>&- + git clone -q --filter=blob:none --no-checkout https://github.com/ludovic111/horizon "$name" 8>&- 9>&- fi cd "$name" -# The lock's descriptor stays out of everything started here (9>&-): a daemon one of them starts (sccache's server, +# The lock's descriptor stays out of everything started here (8>&- 9>&-): a daemon one of them starts (sccache's server, # from the preparer's build or the command's) would hold the lock after the run. -git fetch -q --no-tags origin "$sha" 9>&- +git fetch -q --no-tags origin "$sha" 8>&- 9>&- git checkout -q --force --detach "$sha" git clean -qfd -e /target -e /.local # The host's vanilla data, linked as CI's runner links it; for harnesses also .local/horizon-client, prepared by # this commit's own preparer (once per preparer version, kept beside the data and shared by every name). if [[ $command =~ parity|lan_e2e|horizon-prepare ]]; then - RUNNER_WORKSPACE=$(dirname "$HORIZON_SHARED_SOURCE") RUNNER_TEMP=$(mktemp -d) bash tools/ci/vanilla-data.sh >&2 9>&- + RUNNER_WORKSPACE=$(dirname "$HORIZON_SHARED_SOURCE") RUNNER_TEMP=$(mktemp -d) bash tools/ci/vanilla-data.sh >&2 8>&- 9>&- else mkdir -p .local for d in minecraft tools datagen; do @@ -103,11 +118,11 @@ unit=remote-$name.scope systemctl --user stop "$unit" 2>/dev/null || true session=$PPID ( while kill -0 "$session" 2>/dev/null; do sleep 2; done; systemctl --user stop "$unit" 2>/dev/null ) \ - < /dev/null > /dev/null 2>&1 9>&- & + < /dev/null > /dev/null 2>&1 8>&- 9>&- & watcher=$! status=0 systemd-run --user --scope --quiet --slice=horizon-ci.slice --unit="$unit" \ - "${pin[@]}" nice -n 10 bash -c "$command" 2>&1 < /dev/null 9>&- | tee "$log" || status=${PIPESTATUS[0]} + "${pin[@]}" nice -n 10 bash -c "$command" 2>&1 < /dev/null 8>&- 9>&- | tee "$log" || status=${PIPESTATUS[0]} kill "$watcher" 2>/dev/null || true systemctl --user stop "$unit" 2>/dev/null || true exit "$status" diff --git a/tools/test_world_reopen_parity.py b/tools/test_world_reopen_parity.py index d3e9c1da6..9487fd1b7 100644 --- a/tools/test_world_reopen_parity.py +++ b/tools/test_world_reopen_parity.py @@ -84,6 +84,7 @@ def main(): failures = 0 try: ctl('wait-ready', '300') + ctl('wait-for', 'loaded', '--timeout', '300') # as after each reopen below ctl('command', 'summon minecraft:minecart ~2 ~ ~2') ctl('wait', '20') expected = carts()