remote.sh: runs get CPUs 0-3 by default; benchmarks hold them with --exclusive

playhorizon's CI runner has CPUs 4-7, and remote.sh pinned agents' runs there
too: CPUs 4-7 sat at 100% while 0-3 idled, and window harnesses in pull
requests' parity jobs lost ticks. Runs share a lock per CPU set; an
--exclusive run (a benchmark) holds it alone. world_reopen also waits for
the world to load before its first count, as after each reopen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ludovic Marie
2026-10-27 04:31:40 +01:00
parent 4168590221
commit cb20bb508b
4 changed files with 30 additions and 14 deletions
+1 -1
View File
@@ -46,7 +46,7 @@ Harnesses a host can't run are listed in the file: `local_only` (`save_roundtrip
### Any command on playhorizon: `tools/remote.sh`
For Linux work that doesn't need a whole parity run (a crate's tests, a few harnesses, a server build), `tools/remote.sh <worktree> <command...>` runs the command on playhorizon at the worktree's commit and streams its output back; its exit status is the command's. Commit and push first: the host fetches the commit from GitHub, and nothing else leaves this machine (no files, no credentials, no environment). Each worktree keeps one checkout there, `~/remote-work/<name>` with its own `target/`, so the next run only rebuilds what changed, and one run per name goes at a time. The command runs niced in the host's CI slice, pinned to CPUs 4-7 (`--cpus`; 0-3 stay free for benchmarks), with scratch game and launcher homes and the vanilla data linked as CI links it; a command naming `parity`, `lan_e2e` or `horizon-prepare` also gets `.local/horizon-client`, prepared as CI prepares it. Whatever the command leaves running (a game, a daemon) is stopped when it ends, and stopping `remote.sh` (or losing the connection) stops the command on the host. Logs stay in `~/remote-work/<name>.logs/`; `--list` shows the checkouts, `--gc [DAYS]` removes old ones.
For Linux work that doesn't need a whole parity run (a crate's tests, a few harnesses, a server build), `tools/remote.sh <worktree> <command...>` runs the command on playhorizon at the worktree's commit and streams its output back; its exit status is the command's. Commit and push first: the host fetches the commit from GitHub, and nothing else leaves this machine (no files, no credentials, no environment). Each worktree keeps one checkout there, `~/remote-work/<name>` with its own `target/`, so the next run only rebuilds what changed, and one run per name goes at a time. The command runs niced in the host's CI slice, pinned to CPUs 0-3 (`--cpus`): the CI runner has 4-7, so these runs never slow a pull request's parity job. A benchmark asks for its CPUs with `--cpus 0-3 --exclusive` (and says so in its pull request): it starts only when no other run uses them, and later runs wait for it to end. The command gets scratch game and launcher homes and the vanilla data linked as CI links it; a command naming `parity`, `lan_e2e` or `horizon-prepare` also gets `.local/horizon-client`, prepared as CI prepares it. Whatever the command leaves running (a game, a daemon) is stopped when it ends, and stopping `remote.sh` (or losing the connection) stops the command on the host. Logs stay in `~/remote-work/<name>.logs/`; `--list` shows the checkouts, `--gc [DAYS]` removes old ones.
```sh
tools/remote.sh .claude/worktrees/my-fix cargo test -p horizon-world --lib # 33 s, warm
+2 -2
View File
@@ -108,8 +108,8 @@ and its failures bisect and comment on merges.
AMD EPYC threads, 30 GB, in Detroit; reachable only through the maintainer's tailnet). Nobody sits at it, so the
watcher never pauses it except by hand. It gets half the machine: `RUNNER_MEMORY=24`, `CARGO_BUILD_JOBS` and the
harness slots at 4, and a drop-in (`~/.config/systemd/user/horizon-runner.service.d/cpus.conf`) that starts it under
`taskset -c 4-7`, so CPUs 0-3 stay free for benchmarks pinned there (the user slice can't set `AllowedCPUs`: systemd
doesn't delegate the cpuset controller to it).
`taskset -c 4-7` (the user slice can't set `AllowedCPUs`: systemd doesn't delegate the cpuset controller to it).
Agents' `tools/remote.sh` runs and benchmarks share CPUs 0-3, so they never slow a pull request's job.
Without Nix, `linux-runner.sh` takes the build libraries, the JDK, sccache, Xvfb and Mesa's software Vulkan
(lavapipe; there is no GPU) from apt, and Python 3.13 from uv. The partner can reach the machine, so gh is not
logged in there: the registration token comes from another machine (`RUNNER_TOKEN`), and no job with secrets may
+26 -11
View File
@@ -1,9 +1,10 @@
#!/usr/bin/env bash
# Run a command on a Linux build host (default playhorizon) at a worktree's commit, and get its output back.
#
# tools/remote.sh [--host H] [--name N] [--cpus LIST] [--jobs J] <worktree> <command...>
# tools/remote.sh [--host H] [--name N] [--cpus LIST] [--exclusive] [--jobs J] <worktree> <command...>
# tools/remote.sh .claude/worktrees/my-fix cargo test -p horizon-world --lib
# tools/remote.sh . python3 tools/parity.py food light
# tools/remote.sh --cpus 0-3 --exclusive . python3 tools/perf_bench.py # a benchmark: CPUs 0-3 to itself
# tools/remote.sh --list # checkouts on the host, their size and last use
# tools/remote.sh --gc [DAYS] # remove checkouts unused for DAYS (default 2)
#
@@ -18,19 +19,24 @@
# What the command leaves running is stopped when it ends; stopping this script stops the command on the host.
# The exit status is the command's.
#
# CPUs: playhorizon's CI runner has CPUs 4-7 (a systemd drop-in pins it, docs/release-flow.md), so runs here get
# 0-3 by default and never slow a pull request's parity job. A run holds its CPU set's lock shared; --exclusive (a
# benchmark: name it with --cpus and say so in its pull request) holds it alone, so it starts only when no other run
# uses those CPUs, and the others wait for it to end. --cpus 4-7 or all compete with CI: only when it is idle.
#
# The host is lent by a partner who keeps hypervisor access: never send secrets, tokens or credentials there,
# and never treat a result from it as a CI result.
set -euo pipefail
# CPUs 0-3 stay free for benchmarks (perf-world-entry measures on playhorizon pinned to them); --cpus all for every one.
host=playhorizon name="" jobs="" cpus=4-7
usage() { sed -n '2,8s/^# \{0,1\}//p' "$0" >&2; exit 2; }
host=playhorizon name="" jobs="" cpus=0-3 exclusive=0
usage() { sed -n '2,9s/^# \{0,1\}//p' "$0" >&2; exit 2; }
while [ $# -gt 0 ]; do
case $1 in
--host) host=$2; shift 2;;
--name) name=$2; shift 2;;
--jobs) jobs=$2; shift 2;;
--cpus) cpus=$2; shift 2;;
--exclusive) exclusive=1; shift;;
--list) exec ssh "ludovic@$host" 'cd ~/remote-work 2>/dev/null && for d in */; do d=${d%/};
printf "%-32s %8s last used %s\n" "$d" "$(du -sh "$d" 2>/dev/null | cut -f1)" "$(stat -c %y "$d.logs" 2>/dev/null | cut -c1-16)"; done';;
--gc) days=${2:-2}; exec ssh "ludovic@$host" "cd ~/remote-work 2>/dev/null || exit 0
@@ -58,7 +64,8 @@ command=$(printf '%q ' "$@")
# The host side reads its script and these four values on stdin (nothing is re-split by a remote shell); its
# own environment comes from the runner's files there.
{
printf 'name=%q sha=%q jobs=%q cpus=%q command=%q\n' "$name" "$sha" "$jobs" "$cpus" "$command"
[[ $cpus =~ ^[0-9,-]+$|^all$ ]] || { echo "remote.sh: --cpus takes a list such as 0-3, or all" >&2; exit 2; }
printf 'name=%q sha=%q jobs=%q cpus=%q exclusive=%q command=%q\n' "$name" "$sha" "$jobs" "$cpus" "$exclusive" "$command"
cat <<'REMOTE'
set -euo pipefail
if [ "$cpus" = all ]; then pin=(); jobs=${jobs:-$(nproc)}; else pin=(taskset -c "$cpus"); jobs=${jobs:-$(taskset -c "$cpus" nproc)}; fi
@@ -71,19 +78,27 @@ unset WAYLAND_DISPLAY
mkdir -p ~/remote-work && cd ~/remote-work
exec 9> "$name.lock"
flock -n 9 || { echo "remote.sh: $name is busy with another run" >&2; exit 75; }
# The CPU set's lock: shared by ordinary runs, held alone by an --exclusive one (a benchmark).
exec 8> ".cpus-$cpus.lock"
if [ "$exclusive" = 1 ]; then
flock -x -n 8 || { echo "remote.sh: other runs use CPUs $cpus; try again when they are done (--list)" >&2; exit 75; }
elif ! flock -s -n 8; then
echo "remote.sh: an exclusive run (a benchmark) has CPUs $cpus; waiting for it to end" >&2
flock -s 8
fi
if [ ! -d "$name/.git" ]; then
git clone -q --filter=blob:none --no-checkout https://github.com/ludovic111/horizon "$name" 9>&-
git clone -q --filter=blob:none --no-checkout https://github.com/ludovic111/horizon "$name" 8>&- 9>&-
fi
cd "$name"
# The lock's descriptor stays out of everything started here (9>&-): a daemon one of them starts (sccache's server,
# The lock's descriptor stays out of everything started here (8>&- 9>&-): a daemon one of them starts (sccache's server,
# from the preparer's build or the command's) would hold the lock after the run.
git fetch -q --no-tags origin "$sha" 9>&-
git fetch -q --no-tags origin "$sha" 8>&- 9>&-
git checkout -q --force --detach "$sha"
git clean -qfd -e /target -e /.local
# The host's vanilla data, linked as CI's runner links it; for harnesses also .local/horizon-client, prepared by
# this commit's own preparer (once per preparer version, kept beside the data and shared by every name).
if [[ $command =~ parity|lan_e2e|horizon-prepare ]]; then
RUNNER_WORKSPACE=$(dirname "$HORIZON_SHARED_SOURCE") RUNNER_TEMP=$(mktemp -d) bash tools/ci/vanilla-data.sh >&2 9>&-
RUNNER_WORKSPACE=$(dirname "$HORIZON_SHARED_SOURCE") RUNNER_TEMP=$(mktemp -d) bash tools/ci/vanilla-data.sh >&2 8>&- 9>&-
else
mkdir -p .local
for d in minecraft tools datagen; do
@@ -103,11 +118,11 @@ unit=remote-$name.scope
systemctl --user stop "$unit" 2>/dev/null || true
session=$PPID
( while kill -0 "$session" 2>/dev/null; do sleep 2; done; systemctl --user stop "$unit" 2>/dev/null ) \
< /dev/null > /dev/null 2>&1 9>&- &
< /dev/null > /dev/null 2>&1 8>&- 9>&- &
watcher=$!
status=0
systemd-run --user --scope --quiet --slice=horizon-ci.slice --unit="$unit" \
"${pin[@]}" nice -n 10 bash -c "$command" 2>&1 < /dev/null 9>&- | tee "$log" || status=${PIPESTATUS[0]}
"${pin[@]}" nice -n 10 bash -c "$command" 2>&1 < /dev/null 8>&- 9>&- | tee "$log" || status=${PIPESTATUS[0]}
kill "$watcher" 2>/dev/null || true
systemctl --user stop "$unit" 2>/dev/null || true
exit "$status"
+1
View File
@@ -84,6 +84,7 @@ def main():
failures = 0
try:
ctl('wait-ready', '300')
ctl('wait-for', 'loaded', '--timeout', '300') # as after each reopen below
ctl('command', 'summon minecraft:minecart ~2 ~ ~2')
ctl('wait', '20')
expected = carts()