32a37e8937
Explore bots paced at 0.8 blocks a second and a 300-second warm-up, so both servers serve the same paths; the work table shows it. Horizon ran as horizon-engine (the server build, from its archive). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
138 lines
8.8 KiB
Markdown
138 lines
8.8 KiB
Markdown
# Horizon's server against vanilla 26.3: mixed bots, equal work (2026-10-18)
|
|
|
|
This reruns `../server-mixed-2026-10-18/` with the bots' load made the same on both servers. In that run the
|
|
`explore` bots flew as fast as each server sent them chunks. Horizon kept up and vanilla didn't, so Horizon
|
|
sent 2.9 to 5.9 times as many chunks a second, and vanilla's walking bots got a fifth of the chunks Horizon's
|
|
got. The comparison of tick times wasn't fair.
|
|
|
|
## Summary
|
|
|
|
- **The work was equal** at 1 and 10 bots. At 25 bots it was equal in chunks sent a second (158.6 against 157.6);
|
|
vanilla was still catching up on chunks its bots hadn't had yet, see "Equal work" below. Every bot joined, none
|
|
was dropped, no chunk was sent twice, and no bot ever waited for the chunk it stood in.
|
|
- **Tick time (MSPT), same work:**
|
|
- 1 bot: 1.8 ms on Horizon against 11.3 on vanilla;
|
|
- 10 bots: 4.7 against 34.3 ms;
|
|
- 25 bots: 11.7 against 50.5 ms. Vanilla dropped just under 20 TPS (19.8) and its worst 5-second P99 was
|
|
1388 ms, against Horizon's 130;
|
|
- 50 bots: Horizon ran at 23.8 ms (20 TPS). Vanilla's watchdog stopped its server 30 s after the last bot
|
|
joined ("A single server tick took 60.01 seconds").
|
|
- **Memory:** Horizon's RSS was 4.1 to 4.9 times smaller: 0.69 against 3.40 GB with one bot, 0.95 against
|
|
3.87 GB with 25, and 1.03 GB with 50. Vanilla's heap is fixed at 4 GB, so its RSS is mostly that heap.
|
|
- **CPU** (100 % is one of the machine's 4 threads): 30 against 128 % at 10 bots, 58 against 192 % at 25.
|
|
- **What changed since the first run:** the first run's vanilla held 50 bots at 29.7 ms because its walking
|
|
bots got 640 to 1100 chunks each. With the explore bots paced, vanilla serves the walking bots in full
|
|
(5400 to 6100 chunks each, against Horizon's 6100 to 6600), and that is the load it can't carry at 50.
|
|
|
|
## Results
|
|
|
|
MSPT is milliseconds per tick from each server's own `/tick query` (50 is the limit for 20 TPS). P95 and P99
|
|
are over the 300 measured seconds. "Worst P99" is the highest P99 of a single 5-second query. RSS and CPU
|
|
cover the server's process tree. "First-time chunks/s" counts chunks a bot hadn't received before (summed
|
|
over the bots). "Waited" is ticks a second that bots stood still waiting for their own chunk. "Pulled back"
|
|
counts position packets that moved a bot back.
|
|
|
|
| Bots | Server | MSPT | P95 | P99 | Worst P99 | TPS | RSS (mean / max) | CPU | Start | Chunks/s | First-time chunks/s | Waited | Pulled back |
|
|
|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
|
|
| 1 | Horizon | **1.77** | 4.21 | 13.04 | 47.5 | 20 | **0.69 / 0.69 GB** | 28 % | 19.7 s | 9.94 | 0.01 | 0 | 0 |
|
|
| 1 | vanilla | 11.34 | 17.12 | 33.22 | 119.1 | 20 | 3.40 / 3.42 GB | 36 % | 20.4 s | 9.98 | 0.01 | 0 | 0 |
|
|
| 10 | Horizon | **4.74** | 10.48 | 19.69 | 120.1 | 20 | **0.86 / 0.88 GB** | 30 % | 19.5 s | 61.35 | 1.79 | 0 | 4159 |
|
|
| 10 | vanilla | 34.30 | 53.72 | 89.08 | 375.8 | 20 | 3.71 / 3.75 GB | 128 % | 19.9 s | 61.66 | 1.93 | 0 | 3976 |
|
|
| 25 | Horizon | **11.72** | 24.45 | 43.79 | 129.8 | 20 | **0.95 / 1.09 GB** | 58 % | 9.6 s | 158.57 | 5.78 | 0 | 3496 |
|
|
| 25 | vanilla | 50.45 | 74.09 | 191.21 | 1388.1 | 19.82 | 3.87 / 4.75 GB | 192 % | 37.8 s | 157.63 | 8.25 | 0 | 3629 |
|
|
| 50 | Horizon | **23.82** | 50.98 | 74.64 | 195.9 | 20 | **1.03 / 1.20 GB** | 98 % | 14.9 s | 302.81 | 10.96 | 0 | 7565 |
|
|
| 50 | vanilla | stopped by its watchdog | | | | | | | 11.2 s | | | | |
|
|
|
|
Chunks each bot received over the whole run, by behaviour (mean):
|
|
|
|
| Bots | Server | walk | explore | idle |
|
|
|---:|---|---:|---:|---:|
|
|
| 1 | Horizon | 6092 | | |
|
|
| 1 | vanilla | 6084 | | |
|
|
| 10 | Horizon | 6248 | 1165 | 473 |
|
|
| 10 | vanilla | 5729 | 1124 | 473 |
|
|
| 25 | Horizon | 6629 | 1267 | 473 |
|
|
| 25 | vanilla | 5387 | 1119 | 473 |
|
|
| 50 | Horizon | 6497 | 1256 | 473 |
|
|
|
|
Both servers pulled bots back about as often (`moved too quickly` 20 and 296 times on vanilla at 10 and 25
|
|
bots).
|
|
|
|
## Equal work
|
|
|
|
`tools/server_bench.py` now prints a work table after the runs. At each bot count it compares every server's
|
|
chunks sent a second and first-time chunks a second with the busiest server's. Work counts as equal when
|
|
both are within 10 % (or within a chunk a second when there are few). The report keeps the table as `work`.
|
|
|
|
| Bots | Chunks/s (Horizon, vanilla) | First-time chunks/s (Horizon, vanilla) | Equal |
|
|
|---:|---|---|---|
|
|
| 1 | 9.94, 9.98 | 0.01, 0.01 | yes |
|
|
| 10 | 61.35, 61.66 | 1.79, 1.93 | yes |
|
|
| 25 | 158.57, 157.63 | 5.78, 8.25 | no: vanilla sent 2.5 more first-time chunks a second |
|
|
|
|
Two settings made the load the same:
|
|
|
|
- **Explore bots paced at 0.8 blocks a second** (`--explore-speed 0.8`), the rate the 2026-10-17 run showed
|
|
both servers keep up with. Every bot then follows the same path on both servers. The path is deterministic
|
|
per bot, and a bot only stops while the chunk it stands in hasn't arrived, which never happened here
|
|
(`Waited` 0). So both servers had the same chunks to send.
|
|
- **A 300-second warm-up** before the 300 measured seconds. A first pass with the old 60-second warm-up
|
|
(aborted after 10 bots) sent the same chunks a second on both (63.1 against 61.4). But vanilla was still
|
|
delivering the bots' first views during the measurement (12.9 first-time chunks a second against Horizon's
|
|
4.1), so its window held more work than Horizon's. With 300 seconds both were in steady state at 1 and 10
|
|
bots. At 25, vanilla still sent 2.5 more first-time chunks a second than Horizon: 1.6 % of all it sent, and
|
|
its walking bots ended the run with 19 % fewer chunks. Read the 25-bot tick times with that in mind: the
|
|
extra first-time chunks are work vanilla had still to do because it fell behind.
|
|
|
|
## Method
|
|
|
|
`tools/server_bench.py` ran each configuration on a fresh server:
|
|
|
|
- the same `server.properties` on both: seed 12345, view and simulation distance 10, offline, creative,
|
|
difficulty easy, `sync-chunk-writes=true`, white-list off;
|
|
- `gamerule respawn_radius 0`, and otherwise vanilla's default game rules (the locator bar on for both);
|
|
- the server waits for `Done`, the bots join one a second, warm up for 300 seconds and are measured for 300,
|
|
with `/tick query` every 5 seconds and RSS and CPU every second.
|
|
|
|
The bots are `horizon-bots --behaviour mixed`: 60 % walk, 20 % explore and 20 % idle, each choice deterministic
|
|
per bot. All are creative and fly 24 blocks above spawn. `walk` flies to random points within 48 blocks at
|
|
5.6 blocks a second, `explore` flies straight out at 0.8 blocks a second, and `idle` hovers.
|
|
|
|
**Machine.** The ThinkPad: Intel Core i7-7500U (2 cores, 4 threads), 32 GB, NixOS. The desktop (24 threads)
|
|
was meant to run it, but it didn't wake (`~/fleet/power wake desktop` timed out, as it did for the first run).
|
|
No other session ran on the ThinkPad during the runs (`load.txt`, a sample a minute). The servers ran under
|
|
`nice -n 19`. The bots ran on the Mac mini (M4 Pro, shared with other work at 70 to 90 % CPU) and reached the
|
|
ThinkPad over the tailnet.
|
|
|
|
**What ran.**
|
|
|
|
- Horizon: the server build from #916 (`horizon-client --no-default-features`, no sound or controllers) at
|
|
519ec121, packaged with `xtask package-server` and installed with `install.sh --archive`, as Linux owners
|
|
will get it. `horizon-server` ran it as `horizon-engine`. The game files were prepared from the owner's own
|
|
Minecraft. `horizon-bots` and `server_bench.py` came from this PR.
|
|
- Vanilla: the pinned 26.3 server jar (the owner's own copy) on OpenJDK 25.0.4.1, `-Xms4g -Xmx4g`.
|
|
|
|
`server_bench.py` stopped after vanilla's 50-bot server crashed and before it wrote its report.
|
|
`mixed-equal.json` holds the run lines it printed. The vanilla 50 entry and its error come from that server's
|
|
log, and the `work` table was computed with `server_bench.work`. `load.txt` is the fleet monitor's view of the
|
|
machines during the runs.
|
|
|
|
## Reproduce
|
|
|
|
curl -fsSL https://raw.githubusercontent.com/ludovic111/horizon/main/packaging/server/install.sh | sh
|
|
horizon-prepare --prepare --server-jar /path/to/server.jar # your own copy
|
|
cargo build --release -p horizon-bots # on the machine that runs the bots
|
|
# in a checkout whose .local/minecraft holds the pinned vanilla jars (tools/lab.py prepare):
|
|
HORIZON_BENCH_BIN=~/.local/share/horizon/current \
|
|
python3 tools/server_bench.py --servers horizon,vanilla --bots 1,10,25,50 --behaviour mixed \
|
|
--explore-speed 0.8 --warmup 300 --nice 19 \
|
|
--bots-via 'ssh <bots machine>' --bots-bin /path/to/horizon-bots --connect <this machine's address> \
|
|
--out mixed-equal.json
|
|
|
|
## Next
|
|
|
|
- One pass per configuration ran. Vanilla at 50 bots, in particular, needs a second pass, ideally on the
|
|
desktop (24 threads), where neither server is CPU-bound.
|
|
- 100 bots, and Paper and Fabric + Lithium (docs/server.md section 9).
|
|
- A longer warm-up, or measuring from when the first-time chunk rate settles, so the 25-bot work is equal too.
|