remote.sh: runs get CPUs 0-3 by default; benchmarks hold them with --exclusive
playhorizon's CI runner has CPUs 4-7, and remote.sh pinned agents' runs there too: CPUs 4-7 sat at 100% while 0-3 idled, and window harnesses in pull requests' parity jobs lost ticks. Runs share a lock per CPU set; an --exclusive run (a benchmark) holds it alone. world_reopen also waits for the world to load before its first count, as after each reopen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -46,7 +46,7 @@ Harnesses a host can't run are listed in the file: `local_only` (`save_roundtrip
|
||||
|
||||
### Any command on playhorizon: `tools/remote.sh`
|
||||
|
||||
For Linux work that doesn't need a whole parity run (a crate's tests, a few harnesses, a server build), `tools/remote.sh <worktree> <command...>` runs the command on playhorizon at the worktree's commit and streams its output back; its exit status is the command's. Commit and push first: the host fetches the commit from GitHub, and nothing else leaves this machine (no files, no credentials, no environment). Each worktree keeps one checkout there, `~/remote-work/<name>` with its own `target/`, so the next run only rebuilds what changed, and one run per name goes at a time. The command runs niced in the host's CI slice, pinned to CPUs 4-7 (`--cpus`; 0-3 stay free for benchmarks), with scratch game and launcher homes and the vanilla data linked as CI links it; a command naming `parity`, `lan_e2e` or `horizon-prepare` also gets `.local/horizon-client`, prepared as CI prepares it. Whatever the command leaves running (a game, a daemon) is stopped when it ends, and stopping `remote.sh` (or losing the connection) stops the command on the host. Logs stay in `~/remote-work/<name>.logs/`; `--list` shows the checkouts, `--gc [DAYS]` removes old ones.
|
||||
For Linux work that doesn't need a whole parity run (a crate's tests, a few harnesses, a server build), `tools/remote.sh <worktree> <command...>` runs the command on playhorizon at the worktree's commit and streams its output back; its exit status is the command's. Commit and push first: the host fetches the commit from GitHub, and nothing else leaves this machine (no files, no credentials, no environment). Each worktree keeps one checkout there, `~/remote-work/<name>` with its own `target/`, so the next run only rebuilds what changed, and one run per name goes at a time. The command runs niced in the host's CI slice, pinned to CPUs 0-3 (`--cpus`): the CI runner has 4-7, so these runs never slow a pull request's parity job. A benchmark asks for its CPUs with `--cpus 0-3 --exclusive` (and says so in its pull request): it starts only when no other run uses them, and later runs wait for it to end. The command gets scratch game and launcher homes and the vanilla data linked as CI links it; a command naming `parity`, `lan_e2e` or `horizon-prepare` also gets `.local/horizon-client`, prepared as CI prepares it. Whatever the command leaves running (a game, a daemon) is stopped when it ends, and stopping `remote.sh` (or losing the connection) stops the command on the host. Logs stay in `~/remote-work/<name>.logs/`; `--list` shows the checkouts, `--gc [DAYS]` removes old ones.
|
||||
|
||||
```sh
|
||||
tools/remote.sh .claude/worktrees/my-fix cargo test -p horizon-world --lib # 33 s, warm
|
||||
|
||||
@@ -108,8 +108,8 @@ and its failures bisect and comment on merges.
|
||||
AMD EPYC threads, 30 GB, in Detroit; reachable only through the maintainer's tailnet). Nobody sits at it, so the
|
||||
watcher never pauses it except by hand. It gets half the machine: `RUNNER_MEMORY=24`, `CARGO_BUILD_JOBS` and the
|
||||
harness slots at 4, and a drop-in (`~/.config/systemd/user/horizon-runner.service.d/cpus.conf`) that starts it under
|
||||
`taskset -c 4-7`, so CPUs 0-3 stay free for benchmarks pinned there (the user slice can't set `AllowedCPUs`: systemd
|
||||
doesn't delegate the cpuset controller to it).
|
||||
`taskset -c 4-7` (the user slice can't set `AllowedCPUs`: systemd doesn't delegate the cpuset controller to it).
|
||||
Agents' `tools/remote.sh` runs and benchmarks share CPUs 0-3, so they never slow a pull request's job.
|
||||
Without Nix, `linux-runner.sh` takes the build libraries, the JDK, sccache, Xvfb and Mesa's software Vulkan
|
||||
(lavapipe; there is no GPU) from apt, and Python 3.13 from uv. The partner can reach the machine, so gh is not
|
||||
logged in there: the registration token comes from another machine (`RUNNER_TOKEN`), and no job with secrets may
|
||||
|
||||
+26
-11
@@ -1,9 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run a command on a Linux build host (default playhorizon) at a worktree's commit, and get its output back.
|
||||
#
|
||||
# tools/remote.sh [--host H] [--name N] [--cpus LIST] [--jobs J] <worktree> <command...>
|
||||
# tools/remote.sh [--host H] [--name N] [--cpus LIST] [--exclusive] [--jobs J] <worktree> <command...>
|
||||
# tools/remote.sh .claude/worktrees/my-fix cargo test -p horizon-world --lib
|
||||
# tools/remote.sh . python3 tools/parity.py food light
|
||||
# tools/remote.sh --cpus 0-3 --exclusive . python3 tools/perf_bench.py # a benchmark: CPUs 0-3 to itself
|
||||
# tools/remote.sh --list # checkouts on the host, their size and last use
|
||||
# tools/remote.sh --gc [DAYS] # remove checkouts unused for DAYS (default 2)
|
||||
#
|
||||
@@ -18,19 +19,24 @@
|
||||
# What the command leaves running is stopped when it ends; stopping this script stops the command on the host.
|
||||
# The exit status is the command's.
|
||||
#
|
||||
# CPUs: playhorizon's CI runner has CPUs 4-7 (a systemd drop-in pins it, docs/release-flow.md), so runs here get
|
||||
# 0-3 by default and never slow a pull request's parity job. A run holds its CPU set's lock shared; --exclusive (a
|
||||
# benchmark: name it with --cpus and say so in its pull request) holds it alone, so it starts only when no other run
|
||||
# uses those CPUs, and the others wait for it to end. --cpus 4-7 or all compete with CI: only when it is idle.
|
||||
#
|
||||
# The host is lent by a partner who keeps hypervisor access: never send secrets, tokens or credentials there,
|
||||
# and never treat a result from it as a CI result.
|
||||
set -euo pipefail
|
||||
|
||||
# CPUs 0-3 stay free for benchmarks (perf-world-entry measures on playhorizon pinned to them); --cpus all for every one.
|
||||
host=playhorizon name="" jobs="" cpus=4-7
|
||||
usage() { sed -n '2,8s/^# \{0,1\}//p' "$0" >&2; exit 2; }
|
||||
host=playhorizon name="" jobs="" cpus=0-3 exclusive=0
|
||||
usage() { sed -n '2,9s/^# \{0,1\}//p' "$0" >&2; exit 2; }
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
--host) host=$2; shift 2;;
|
||||
--name) name=$2; shift 2;;
|
||||
--jobs) jobs=$2; shift 2;;
|
||||
--cpus) cpus=$2; shift 2;;
|
||||
--exclusive) exclusive=1; shift;;
|
||||
--list) exec ssh "ludovic@$host" 'cd ~/remote-work 2>/dev/null && for d in */; do d=${d%/};
|
||||
printf "%-32s %8s last used %s\n" "$d" "$(du -sh "$d" 2>/dev/null | cut -f1)" "$(stat -c %y "$d.logs" 2>/dev/null | cut -c1-16)"; done';;
|
||||
--gc) days=${2:-2}; exec ssh "ludovic@$host" "cd ~/remote-work 2>/dev/null || exit 0
|
||||
@@ -58,7 +64,8 @@ command=$(printf '%q ' "$@")
|
||||
# The host side reads its script and these four values on stdin (nothing is re-split by a remote shell); its
|
||||
# own environment comes from the runner's files there.
|
||||
{
|
||||
printf 'name=%q sha=%q jobs=%q cpus=%q command=%q\n' "$name" "$sha" "$jobs" "$cpus" "$command"
|
||||
[[ $cpus =~ ^[0-9,-]+$|^all$ ]] || { echo "remote.sh: --cpus takes a list such as 0-3, or all" >&2; exit 2; }
|
||||
printf 'name=%q sha=%q jobs=%q cpus=%q exclusive=%q command=%q\n' "$name" "$sha" "$jobs" "$cpus" "$exclusive" "$command"
|
||||
cat <<'REMOTE'
|
||||
set -euo pipefail
|
||||
if [ "$cpus" = all ]; then pin=(); jobs=${jobs:-$(nproc)}; else pin=(taskset -c "$cpus"); jobs=${jobs:-$(taskset -c "$cpus" nproc)}; fi
|
||||
@@ -71,19 +78,27 @@ unset WAYLAND_DISPLAY
|
||||
mkdir -p ~/remote-work && cd ~/remote-work
|
||||
exec 9> "$name.lock"
|
||||
flock -n 9 || { echo "remote.sh: $name is busy with another run" >&2; exit 75; }
|
||||
# The CPU set's lock: shared by ordinary runs, held alone by an --exclusive one (a benchmark).
|
||||
exec 8> ".cpus-$cpus.lock"
|
||||
if [ "$exclusive" = 1 ]; then
|
||||
flock -x -n 8 || { echo "remote.sh: other runs use CPUs $cpus; try again when they are done (--list)" >&2; exit 75; }
|
||||
elif ! flock -s -n 8; then
|
||||
echo "remote.sh: an exclusive run (a benchmark) has CPUs $cpus; waiting for it to end" >&2
|
||||
flock -s 8
|
||||
fi
|
||||
if [ ! -d "$name/.git" ]; then
|
||||
git clone -q --filter=blob:none --no-checkout https://github.com/ludovic111/horizon "$name" 9>&-
|
||||
git clone -q --filter=blob:none --no-checkout https://github.com/ludovic111/horizon "$name" 8>&- 9>&-
|
||||
fi
|
||||
cd "$name"
|
||||
# The lock's descriptor stays out of everything started here (9>&-): a daemon one of them starts (sccache's server,
|
||||
# The lock's descriptor stays out of everything started here (8>&- 9>&-): a daemon one of them starts (sccache's server,
|
||||
# from the preparer's build or the command's) would hold the lock after the run.
|
||||
git fetch -q --no-tags origin "$sha" 9>&-
|
||||
git fetch -q --no-tags origin "$sha" 8>&- 9>&-
|
||||
git checkout -q --force --detach "$sha"
|
||||
git clean -qfd -e /target -e /.local
|
||||
# The host's vanilla data, linked as CI's runner links it; for harnesses also .local/horizon-client, prepared by
|
||||
# this commit's own preparer (once per preparer version, kept beside the data and shared by every name).
|
||||
if [[ $command =~ parity|lan_e2e|horizon-prepare ]]; then
|
||||
RUNNER_WORKSPACE=$(dirname "$HORIZON_SHARED_SOURCE") RUNNER_TEMP=$(mktemp -d) bash tools/ci/vanilla-data.sh >&2 9>&-
|
||||
RUNNER_WORKSPACE=$(dirname "$HORIZON_SHARED_SOURCE") RUNNER_TEMP=$(mktemp -d) bash tools/ci/vanilla-data.sh >&2 8>&- 9>&-
|
||||
else
|
||||
mkdir -p .local
|
||||
for d in minecraft tools datagen; do
|
||||
@@ -103,11 +118,11 @@ unit=remote-$name.scope
|
||||
systemctl --user stop "$unit" 2>/dev/null || true
|
||||
session=$PPID
|
||||
( while kill -0 "$session" 2>/dev/null; do sleep 2; done; systemctl --user stop "$unit" 2>/dev/null ) \
|
||||
< /dev/null > /dev/null 2>&1 9>&- &
|
||||
< /dev/null > /dev/null 2>&1 8>&- 9>&- &
|
||||
watcher=$!
|
||||
status=0
|
||||
systemd-run --user --scope --quiet --slice=horizon-ci.slice --unit="$unit" \
|
||||
"${pin[@]}" nice -n 10 bash -c "$command" 2>&1 < /dev/null 9>&- | tee "$log" || status=${PIPESTATUS[0]}
|
||||
"${pin[@]}" nice -n 10 bash -c "$command" 2>&1 < /dev/null 8>&- 9>&- | tee "$log" || status=${PIPESTATUS[0]}
|
||||
kill "$watcher" 2>/dev/null || true
|
||||
systemctl --user stop "$unit" 2>/dev/null || true
|
||||
exit "$status"
|
||||
|
||||
@@ -84,6 +84,7 @@ def main():
|
||||
failures = 0
|
||||
try:
|
||||
ctl('wait-ready', '300')
|
||||
ctl('wait-for', 'loaded', '--timeout', '300') # as after each reopen below
|
||||
ctl('command', 'summon minecraft:minecart ~2 ~ ~2')
|
||||
ctl('wait', '20')
|
||||
expected = carts()
|
||||
|
||||
Reference in New Issue
Block a user