Benchmark Horizon against vanilla 26.3 and vanilla with performance mods

tools/perf_bench.py runs Horizon, the vanilla 26.3 client and vanilla with
Sodium, Lithium, FerriteCore and ImmediatelyFast through the same steps:
fresh world (seed 1234, creative) per render distance, start to title, start
to in-world, then per scene teleport, chunk load, settle and a fixed 192-block
flight. Every frame time is kept and one function computes average FPS, 1% and
0.1% lows, p50/p95/p99/p99.9 and a frame-time histogram for every game; RSS is
sampled from outside the process for every game. One options.txt (vanilla's
keys, Fancy preset values with a Custom preset so the render distance holds)
is written for all of them.

Vanilla is driven by a dev-only Fabric mod (mods/horizon/src/bench) compiled
against the user's own client jar: it creates the world with
WorldOpenFlows.createFreshLevel, teleports with /tp and places the player on
the path every frame as horizon-ctl fly does. Nothing of the game is
redistributed and the performance mods are fetched from Modrinth with the
hashes pinned in packs/mods.lock.json.

Horizon side: `fly ... PITCH raw` also replies with every frame time, and
--offscreen drives frames from the event loop, so a window nobody sees (a
locked macOS session) still renders.

docs/benchmark.md describes what is measured, how fairness is kept, the
caveats and how to reproduce it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ludovic Marie
2026-10-15 00:17:08 +02:00
parent ff5df6aa5e
commit b691b5ebaa
12 changed files with 1056 additions and 4 deletions
@@ -305,7 +305,8 @@ pub const COMMANDS: &[Cmd] = &[
["horizon-ctl mcp"], &[], mcp: false, local: true),
cmd!("bench", Scripting, "bench SECONDS", "reset frame stats, wait, reply with the summary", "", ["horizon-ctl bench 10"], &[], mcp: false, local: false),
cmd!("stats", Scripting, "stats [reset]", "frame-time summary since the last reset", "", ["horizon-ctl stats"], &[], mcp: false, local: false),
cmd!("fly", Scripting, "fly X1 Y1 Z1 X2 Y2 Z2 SECONDS [PITCH]", "fly the camera along a line, then reply with the frame summary", "",
cmd!("fly", Scripting, "fly X1 Y1 Z1 X2 Y2 Z2 SECONDS [PITCH [raw]]", "fly the camera along a line, then reply with the frame summary",
"With raw, the reply also lists every frame time in milliseconds (frame_ms).",
["horizon-ctl fly 0 100 0 500 100 0 20"], &[], mcp: false, local: false),
];
+4
View File
@@ -424,6 +424,10 @@ impl FrameStats {
self.gpu.extend_from_slice(millis);
}
}
/// Every recorded frame time, in milliseconds, in order.
pub fn frame_millis(&self) -> &[f32] {
&self.millis
}
pub fn reset(&mut self) {
self.millis.clear();
self.cpu.clear();
+8 -1
View File
@@ -9088,7 +9088,14 @@ impl ApplicationHandler for Game<'_> {
return;
}
if let Some(w) = &self.window {
w.request_redraw();
if self.options.offscreen {
// Offscreen frames don't wait for the window to be drawn: an
// occluded window (a locked macOS session) gets no redraws.
let id = w.id();
self.window_event(event_loop, id, WindowEvent::RedrawRequested);
} else {
w.request_redraw();
}
}
event_loop.set_control_flow(ControlFlow::Poll);
}
+9 -2
View File
@@ -32,7 +32,7 @@ const HELP: &[&str] = &[
"options [PAGE] open the options (video sound controls keybinds mouse chat accessibility language font packs graphics graphics-advanced)",
"stats [reset] frame-time summary since the last reset",
"bench SECONDS reset stats, wait, then reply with the summary",
"fly X1 Y1 Z1 X2 Y2 Z2 SECONDS [PITCH] fly the camera along a line at constant speed, then reply with the frame summary",
"fly X1 Y1 Z1 X2 Y2 Z2 SECONDS [PITCH [raw]] fly the camera along a line at constant speed, then reply with the frame summary (raw: every frame time too)",
"open WORLD [SEED] [creative] open or create a world from the title screen",
"save save the world",
"save-all save the world, rewriting every resident chunk",
@@ -51,6 +51,8 @@ pub struct Flight {
pitch: f32,
started: Instant,
seconds: f64,
/// Reply with every frame time too (`fly ... PITCH raw`).
raw: bool,
reply: Sender<String>,
}
@@ -76,7 +78,11 @@ impl Game<'_> {
if t >= 1.0
&& let Some(flight) = self.flight.take()
{
let _ = flight.reply.send(self.frame_stats.summary().to_string());
let mut summary = self.frame_stats.summary();
if flight.raw {
summary["frame_ms"] = self.frame_stats.frame_millis().into();
}
let _ = flight.reply.send(summary.to_string());
}
}
pub(super) fn poll_control(&mut self, event_loop: &ActiveEventLoop) {
@@ -464,6 +470,7 @@ impl Game<'_> {
pitch,
started: Instant::now(),
seconds: n(7)?.max(0.1),
raw: words.get(9) == Some(&"raw"),
reply: reply.clone(),
});
Ok(None)
+1
View File
@@ -7,6 +7,7 @@ To play, use [the launcher](../launcher/README.md); the project site is [playhor
- [Build from source](../README.md#build-from-source): your own copy of the game, resource preparation, `cargo run`.
- [Native client](native-client.md): options, controls, the dedicated server, `horizon-ctl` scripting, packaging, architecture.
- [Graphics](graphics.md): the Simple and Advanced graphics settings, the graphics benchmark and what it measured.
- [Benchmark](benchmark.md): Horizon against vanilla 26.3 and vanilla with performance mods, on the same machine, world, path and settings: what is measured, how it is kept fair, how to reproduce it.
- [Extras](extras.md): optional zoom, minimap, FPS and keystroke overlays and auto-reconnect, all off by default.
- [Launcher](../launcher/README.md): installs, updates and starts builds; its CLI and MCP server; releasing it.
- [Opt-in data collection](telemetry.md): the consent, exactly what is sent, how it is stored and sent.
+50
View File
@@ -0,0 +1,50 @@
# Benchmark: Horizon vs vanilla 26.3
`tools/perf_bench.py` measures Horizon, the vanilla 26.3 client and vanilla with performance mods on the same machine, the same world and the same camera path, and computes every number with the same code. This page says what it measures, how it keeps the comparison fair, what it can't control, and how to reproduce it. The results are in [Results](#results).
## What is measured
One game is started per render distance (8, 16 and 32 chunks by default), each from a fresh world, so nothing generated by an earlier run is reused. In each game:
1. **Start to the title screen.** Wall-clock time from starting the process to the first frame of the title screen with no loading overlay.
2. **Create the world.** A creative world named `PerfBench`, seed 1234, default world type and difficulty, created the way the game's own Create World does. **Start to in-world** is the time from starting the process until the player has spawned and every chunk around the spawn is loaded and drawn.
3. **Four scenes**, the paths of `tools/graphics_bench.py`: straight 192-block flights over a village, a forest, the ocean and a mountain. For each scene the player is teleported to the start (time set to noon), held there flying, and the **chunk load time** is the time from the teleport until every chunk in view distance is loaded and drawn. After 2 seconds of settling, the player flies the path at constant speed in 12 seconds (16 blocks per second, faster than sprint-flying), facing along it at the scene's pitch, placed on the line every frame. Chunks are generated and drawn as they come into range during the flight, as in play.
4. **Frame times.** Every frame of every flight is recorded (time between the starts of consecutive frames). From them: average FPS, **1% and 0.1% lows** (the average frame time of the slowest 1% or 0.1% of frames, as FPS: a single long freeze shows), the median, p95, p99 and p99.9 frame times, the slowest frame, and the share of frames in buckets (under 4.17 ms = 240 FPS, 8.33, 16.67, 33.33, 50, 100 ms, and over).
5. **Memory.** The process's resident set size (RSS), sampled 4 times a second with `ps` from outside the process: **peak** over the whole run (startup included) and **steady**, the median during the flights. For vanilla this counts the whole JVM (heap, JIT, native buffers, the integrated server), for Horizon the whole process. GPU memory where it can be read: Horizon's own Vulkan allocator total, and what `nvidia-smi` attributes to the process on NVIDIA. On Apple Silicon the GPU shares RAM and no per-process figure is available.
## How it is kept fair
- **Same machine, same run conditions.** Games run one at a time; the load average before and after each run and the machine (CPU, RAM, OS) are stored with the results. Runs on a machine doing other work are marked as such and not published.
- **Same settings, one file.** Every game gets the same `options.txt`, written by the driver: render distance, simulation distance 12, unlimited frame rate (`maxFps:260`), VSync off, FOV 70, automatic GUI scale, windowed, no pause on lost focus, and `inactivityFpsLimit:minimized` (both games otherwise drop to 30 FPS after a minute without input). Everything else stays at vanilla's defaults: Horizon reads vanilla's `options.txt` keys and its defaults match vanilla's (the options harness checks it), and Horizon's own settings (`horizon-graphics.txt`) are left at their defaults, which render as vanilla does.
- **Same resolution.** Both windows are 1280x720 (Horizon's default window; vanilla gets `--width 1280 --height 720`). The size each game actually renders at is stored with the results.
- **Same world and path.** Same seed and world type; same paths, yaw and pitch, same flight speed; the same steps in the same order. Vanilla is driven by a small dev-only Fabric mod (`mods/horizon/src/bench`), which only calls the game's own code: it creates the world with `WorldOpenFlows.createFreshLevel` as the Create World screen does, teleports with `/tp`, sets the time with `/time set`, and places the player on the path each frame as Horizon's `horizon-ctl fly` does.
- **Warm-up excluded.** Each scene's flight starts only after its chunks are loaded and drawn and 2 seconds have passed; the load itself is reported separately. JIT warm-up of the JVM during the first scene is part of vanilla's real behaviour and is not removed.
- **The same statistics code.** Horizon returns its raw frame times (`horizon-ctl fly ... raw`) and the vanilla mod writes its own; `stats()` in the driver computes every figure for both.
- **Vanilla as players run it.** Your own 26.3 client jar and libraries (as the official launcher installs them for your OS), the official launcher's default JVM arguments (`-Xmx2G` and its G1 options), Java 25. The game's assets index is not used, so neither game plays sound (Horizon runs with `HORIZON_NO_AUDIO=1`, vanilla with OpenAL's null device); textures, models and shaders come from the client jar as usual.
- **Performance mods.** `--game vanilla-mods` adds the Fabric builds for 26.3 of Sodium, Lithium, FerriteCore and ImmediatelyFast, the versions pinned with their hashes in `packs/mods.lock.json` (downloaded from Modrinth, never committed), plus Fabric API if one of them needs it.
## Known caveats
- **Fabric Loader is present in the vanilla runs**, to load the driver mod. It patches only the frame hook of the driver mod; its cost after startup is negligible, but its class scanning adds to the startup time (roughly a second; it is the same loader the modded runs use).
- **Readiness is detected differently.** Horizon knows exactly how many chunks are pending; for vanilla the driver waits until the loaded chunk count is unchanged for a second with nothing left to compile, and takes the time of the last change. Both stop the clock at the last chunk arriving, within a frame or two.
- **Locked or headless screens.** A window nobody can see is not composited. On a Mac whose session is locked, Horizon draws only with `--offscreen` (it renders into a texture and does not wait for the window), and vanilla keeps rendering into its window without it being shown. Such runs measure rendering without presentation and are marked `offscreen`; numbers for players come from runs with the window on a real screen.
- **Different graphics APIs.** Horizon renders with wgpu (Vulkan on Linux and Windows, Metal on macOS); vanilla 26.3 uses OpenGL (or its Vulkan backend when chosen). The backend each game used is stored with the results. This is part of what is being compared.
- **World generation is the same algorithm but not the same threads.** Both generate the same terrain (the parity harnesses check it), with their own threading; the work per chunk differs, which is also part of what is being compared.
- **Variance.** Flights are short (12 seconds); run each configuration more than once and compare medians before drawing conclusions from small differences. Lows are the noisiest figures.
## Reproduce it
You need Horizon built (`cargo build --release -p horizon-client --bins`) and its resources prepared from your own copy of 26.3 (which also provides `.local/minecraft/26.3`, the vanilla client the driver starts), Java 25 with `javac`, and Python 3.10 or newer.
```sh
python3 tools/perf_bench.py --game horizon --label horizon-mine
python3 tools/perf_bench.py --game vanilla --label vanilla-mine
python3 tools/perf_bench.py --game vanilla-mods --label mods-mine
python3 tools/perf_bench.py --table horizon-mine vanilla-mine mods-mine
```
Options: `--distances 8,16`, `--scenes village,ocean`, `--seconds 20`, `--heap 4G` (vanilla), `--offscreen` (Horizon, for locked or headless sessions). Results go to `benchmarks/perf/<label>.json`; each game's log, and the vanilla driver's event log, stay in `.local/perf-bench/`. Keep the machine otherwise idle, plugged in, and the window on a screen.
## Results
See [benchmarks/perf](../benchmarks/perf) for the raw results.
@@ -0,0 +1,300 @@
package dev.horizonlab.bench;
import com.google.gson.JsonArray;
import com.google.gson.JsonObject;
import com.google.gson.JsonParser;
import com.mojang.blaze3d.systems.RenderSystem;
import java.io.IOException;
import java.io.UncheckedIOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.List;
import net.fabricmc.api.ClientModInitializer;
import net.minecraft.client.Minecraft;
import net.minecraft.client.gui.screens.TitleScreen;
import net.minecraft.client.player.LocalPlayer;
import net.minecraft.world.level.GameType;
import net.minecraft.world.level.LevelSettings;
import net.minecraft.world.level.WorldDataConfiguration;
import net.minecraft.world.level.levelgen.WorldOptions;
import net.minecraft.world.level.levelgen.presets.WorldPresets;
import net.minecraft.world.phys.Vec3;
/**
* Dev-only benchmark driver for the vanilla client (tools/perf_bench.py): the
* same steps horizon-ctl drives in Horizon. From the title screen it creates a
* fresh creative world with the script's seed, waits until the spawn area is
* loaded, then for each scene teleports, waits for the chunks, settles and
* flies the player along a straight line at constant speed, facing along it,
* recording every frame time. Events go to the output file as JSON lines as
* they happen; the game quits after the last scene.
*
* Script (system property horizon.bench.script): {"world", "seed", "settle",
* "seconds", "timeout", "scenes": [{"name", "from": [x,y,z], "to": [x,y,z],
* "pitch"}]}. Output: horizon.bench.out.
*/
public final class VanillaBench implements ClientModInitializer {
private static VanillaBench bench;
private enum Phase { TITLE, WORLD, TELEPORT, LOAD, SETTLE, FLY, DONE }
private final JsonObject script;
private final Path out;
private final List<JsonObject> scenes = new ArrayList<>();
private Phase phase = Phase.TITLE;
private int scene = -1;
private long phaseStart;
private long lastFrame;
private long readyCheck;
private int lastLoaded = -1;
private long lastChange;
private final List<Float> frames = new ArrayList<>();
/** Fabric's entrypoint instance; the benchmark itself is the static one. */
public VanillaBench() {
this.script = null;
this.out = null;
}
private VanillaBench(JsonObject script, Path out) {
this.script = script;
this.out = out;
for (var element : script.getAsJsonArray("scenes")) scenes.add(element.getAsJsonObject());
}
@Override public void onInitializeClient() {
String scriptPath = System.getProperty("horizon.bench.script");
String outPath = System.getProperty("horizon.bench.out");
if (scriptPath == null || outPath == null) return;
try {
JsonObject json = JsonParser.parseString(Files.readString(Path.of(scriptPath))).getAsJsonObject();
bench = new VanillaBench(json, Path.of(outPath));
} catch (IOException e) {
throw new UncheckedIOException(e);
}
bench.event("init");
}
/** Called at the start of every frame (Minecraft.runTick). */
public static void frame(Minecraft mc) {
if (bench != null) bench.step(mc);
}
private void step(Minecraft mc) {
long now = System.nanoTime();
float frameMs = lastFrame == 0 ? 0 : (now - lastFrame) / 1e6f;
lastFrame = now;
double elapsed = (now - phaseStart) / 1e9;
if (phase != Phase.TITLE && phase != Phase.DONE && elapsed > script.get("timeout").getAsDouble()
&& phase != Phase.FLY && phase != Phase.SETTLE) {
JsonObject e = new JsonObject();
e.addProperty("phase", phase.name());
event("timeout", e);
finish(mc);
return;
}
switch (phase) {
case TITLE -> {
if (mc.gui.screen() instanceof TitleScreen && mc.gui.overlay() == null) {
event("title", renderer(mc));
LevelSettings settings = new LevelSettings(script.get("world").getAsString(), GameType.CREATIVE,
LevelSettings.DifficultySettings.DEFAULT, true, WorldDataConfiguration.DEFAULT);
WorldOptions options = new WorldOptions(script.get("seed").getAsLong(), true, false);
enter(Phase.WORLD, now);
mc.createWorldOpenFlows().createFreshLevel(script.get("world").getAsString(), settings, options,
WorldPresets::createNormalWorldDimensions, mc.gui.screen());
}
}
case WORLD -> {
if (ready(mc, null, now)) {
JsonObject e = new JsonObject();
e.addProperty("ready_epoch_ms", epochMs(lastChange));
e.addProperty("chunks", lastLoaded);
event("world", e);
mc.player.connection.sendCommand("weather clear");
next(mc, now);
}
}
case TELEPORT -> {
JsonObject s = scenes.get(scene);
Vec3 from = vec(s.getAsJsonArray("from"));
float pitch = s.get("pitch").getAsFloat();
LocalPlayer player = mc.player;
player.connection.sendCommand("time set 6000");
player.connection.sendCommand(String.format(java.util.Locale.ROOT, "tp @s %.3f %.3f %.3f %.3f %.3f",
from.x, from.y, from.z, yaw(s), pitch));
event("teleport", named(s));
enter(Phase.LOAD, now);
}
case LOAD -> {
JsonObject s = scenes.get(scene);
Vec3 from = vec(s.getAsJsonArray("from"));
// Once the teleport has arrived, hold the player there, flying.
if (mc.player != null && mc.player.position().distanceToSqr(from) < 1024.0) place(mc, s, 0);
if (ready(mc, from, now)) {
JsonObject e = named(s);
e.addProperty("load_ms", (lastChange - phaseStart) / 1e6);
e.addProperty("chunks", lastLoaded);
event("loaded", e);
enter(Phase.SETTLE, now);
}
}
case SETTLE -> {
place(mc, scenes.get(scene), 0);
if (elapsed >= script.get("settle").getAsDouble()) {
frames.clear();
event("fly", named(scenes.get(scene)));
enter(Phase.FLY, now);
}
}
case FLY -> {
frames.add(frameMs);
double t = Math.min(1.0, elapsed / script.get("seconds").getAsDouble());
place(mc, scenes.get(scene), t);
if (t >= 1.0) {
JsonObject e = named(scenes.get(scene));
JsonArray times = new JsonArray();
frames.forEach(times::add);
e.add("frame_ms", times);
e.addProperty("chunks", mc.level.getChunkSource().getLoadedChunksCount());
event("flown", e);
next(mc, now);
}
}
case DONE -> { }
}
}
private void next(Minecraft mc, long now) {
scene++;
if (scene >= scenes.size()) {
finish(mc);
} else {
enter(Phase.TELEPORT, now);
}
}
private void finish(Minecraft mc) {
phase = Phase.DONE;
event("done");
mc.stop();
}
private void enter(Phase next, long now) {
phase = next;
phaseStart = now;
lastLoaded = -1;
lastChange = now;
}
/**
* The player has spawned (near {@code at}, when given), no screen is open,
* every chunk around is there, no section waits to be compiled, and the
* client's loaded chunk count has not changed for a second; lastChange is
* when that state was reached.
*/
private boolean ready(Minecraft mc, Vec3 at, long now) {
if (now - readyCheck < 10_000_000L) return false;
readyCheck = now;
if (mc.level == null || mc.player == null || mc.gui.screen() != null || mc.gui.overlay() != null
|| (at != null && mc.player.position().distanceToSqr(at) > 4.0)) {
lastChange = now;
lastLoaded = -1;
return false;
}
int loaded = mc.level.getChunkSource().getLoadedChunksCount();
if (loaded != lastLoaded || !mc.levelRenderer.hasRenderedAllSections() || !chunksAround(mc)) {
lastLoaded = loaded;
lastChange = now;
return false;
}
return now - lastChange >= 1_000_000_000L;
}
/**
* Every chunk within the render distance minus one of the player is on the
* client (the server sends a little more; the last ring arrives with it).
*/
private static boolean chunksAround(Minecraft mc) {
int r = Math.max(1, mc.options.getEffectiveRenderDistance() - 1);
var center = mc.player.chunkPosition();
var chunks = mc.level.getChunkSource();
for (int dx = -r; dx <= r; dx++) {
for (int dz = -r; dz <= r; dz++) {
if (dx * dx + dz * dz <= r * r && !chunks.hasChunk(center.x() + dx, center.z() + dz)) return false;
}
}
return true;
}
/** Puts the player on the scene's line at fraction t, facing along it, as Horizon's flight. */
private static void place(Minecraft mc, JsonObject s, double t) {
LocalPlayer player = mc.player;
if (player == null) return;
Vec3 a = vec(s.getAsJsonArray("from")), b = vec(s.getAsJsonArray("to"));
Vec3 at = a.add(b.subtract(a).scale(t));
player.setPos(at.x, at.y, at.z);
player.setYRot(yaw(s));
player.setXRot(s.get("pitch").getAsFloat());
player.setOldPosAndRot();
player.setDeltaMovement(Vec3.ZERO);
player.fallDistance = 0;
if (!player.getAbilities().flying) {
player.getAbilities().flying = true;
player.onUpdateAbilities();
}
}
/** Minecraft yaw facing from "from" to "to": 0 faces +Z, 90 faces -X. */
private static float yaw(JsonObject s) {
Vec3 a = vec(s.getAsJsonArray("from")), b = vec(s.getAsJsonArray("to"));
return (float) Math.toDegrees(Math.atan2(-(b.x - a.x), b.z - a.z));
}
private static Vec3 vec(JsonArray a) {
return new Vec3(a.get(0).getAsDouble(), a.get(1).getAsDouble(), a.get(2).getAsDouble());
}
private static JsonObject named(JsonObject s) {
JsonObject e = new JsonObject();
e.addProperty("scene", s.get("name").getAsString());
return e;
}
private static JsonObject renderer(Minecraft mc) {
JsonObject e = new JsonObject();
var info = RenderSystem.getDevice().getDeviceInfo();
e.addProperty("gpu", info.name());
e.addProperty("vendor", info.vendorName());
e.addProperty("backend", info.backendName());
e.addProperty("driver", info.driverInfo());
JsonArray size = new JsonArray();
size.add(mc.getWindow().getWidth());
size.add(mc.getWindow().getHeight());
e.add("size", size);
e.addProperty("render_distance", mc.options.renderDistance().get());
e.addProperty("simulation_distance", mc.options.simulationDistance().get());
return e;
}
private static long epochMs(long nanos) {
return System.currentTimeMillis() - (System.nanoTime() - nanos) / 1_000_000L;
}
private void event(String name) {
event(name, new JsonObject());
}
private void event(String name, JsonObject e) {
e.addProperty("event", name);
e.addProperty("epoch_ms", System.currentTimeMillis());
try {
Files.writeString(out, e + "\n", StandardCharsets.UTF_8, StandardOpenOption.CREATE, StandardOpenOption.APPEND);
} catch (IOException error) {
throw new UncheckedIOException(error);
}
}
}
@@ -0,0 +1,17 @@
package dev.horizonlab.bench.mixin;
import dev.horizonlab.bench.VanillaBench;
import net.minecraft.client.Minecraft;
import org.spongepowered.asm.mixin.Mixin;
import org.spongepowered.asm.mixin.injection.At;
import org.spongepowered.asm.mixin.injection.Inject;
import org.spongepowered.asm.mixin.injection.callback.CallbackInfo;
@Mixin(Minecraft.class)
public abstract class FrameHookMixin {
@Inject(method = "runTick", at = @At("HEAD"))
private void horizonBenchFrame(boolean advanceGameTime, CallbackInfo callback) {
VanillaBench.frame((Minecraft) (Object) this);
}
}
@@ -0,0 +1,21 @@
{
"schemaVersion": 1,
"id": "horizon_bench",
"version": "1.0.0",
"name": "Horizon benchmark driver",
"description": "Dev-only: flies the vanilla client along tools/perf_bench.py's paths and records frame times. Never shipped.",
"license": "MIT",
"environment": "client",
"entrypoints": {
"client": [
"dev.horizonlab.bench.VanillaBench"
]
},
"mixins": [
"horizon_bench.mixins.json"
],
"depends": {
"fabricloader": ">=0.19.5",
"minecraft": "26.3"
}
}
@@ -0,0 +1,11 @@
{
"required": true,
"package": "dev.horizonlab.bench.mixin",
"compatibilityLevel": "JAVA_25",
"client": [
"FrameHookMixin"
],
"injectors": {
"defaultRequire": 1
}
}
+35
View File
@@ -0,0 +1,35 @@
import unittest
from tools.perf_bench import OPTIONS, stats
class PerfBenchStatsTests(unittest.TestCase):
def test_lows_average_the_slowest_frames(self):
times = [10.0] * 990 + [100.0] * 9 + [1000.0]
s = stats(times)
self.assertEqual(s["frames"], 1000)
self.assertAlmostEqual(s["avg_fps"], 1000 * 1000 / (9900 + 900 + 1000))
# Slowest 1%: 9 frames of 100 ms and one of 1000 ms.
self.assertAlmostEqual(s["low_1pct_fps"], 1000 * 10 / 1900)
# Slowest 0.1%: the single 1000 ms frame.
self.assertAlmostEqual(s["low_01pct_fps"], 1.0)
self.assertEqual(s["p50_ms"], 10.0)
self.assertEqual(s["p99_ms"], 10.0)
self.assertEqual(s["p999_ms"], 100.0)
self.assertEqual(s["max_ms"], 1000.0)
self.assertAlmostEqual(sum(s["histogram"].values()), 1.0)
self.assertAlmostEqual(s["histogram"][">=100ms"], 0.01)
def test_a_short_flight_still_has_lows(self):
s = stats([5.0, 20.0])
self.assertAlmostEqual(s["low_1pct_fps"], 50.0)
self.assertAlmostEqual(s["low_01pct_fps"], 50.0)
self.assertEqual(stats([]), {"frames": 0})
def test_one_options_file_sets_the_render_distance(self):
self.assertEqual(OPTIONS["renderDistance"].format(distance=16), "16")
self.assertEqual(OPTIONS["graphicsPreset"], '"custom"')
if __name__ == "__main__":
unittest.main()
+598
View File
@@ -0,0 +1,598 @@
#!/usr/bin/env python3
"""Fair Horizon vs vanilla 26.3 benchmark (methodology: docs/benchmark.md).
Runs one game per render distance, each from a fresh world (seed 1234,
creative), through the same steps: start to the title screen, create the
world, wait for the spawn area, then per scene teleport, wait for the chunks,
settle and fly a fixed straight path at constant speed. Every frame time of
every flight is kept and the statistics are computed here, by one function,
for every game. RSS is sampled from outside the process for every game too.
python3 tools/perf_bench.py --game horizon --label horizon-m4
python3 tools/perf_bench.py --game vanilla --label vanilla-m4
python3 tools/perf_bench.py --game vanilla-mods --label mods-m4
python3 tools/perf_bench.py --table horizon-m4 vanilla-m4 mods-m4
Games:
horizon target/release/horizon-client, driven over its control socket
vanilla your own 26.3 client.jar + Fabric Loader + a dev-only driver
mod (mods/horizon/src/bench), built here; nothing of the game
is redistributed
vanilla-mods the same + Sodium, Lithium, FerriteCore and ImmediatelyFast
(pinned in packs/mods.lock.json)
Results: benchmarks/perf/<label>.json. Games run with no audio, from a
scratch directory under .local/perf-bench.
"""
import argparse
import datetime
import json
import math
import os
import platform
import shutil
import socket
import statistics
import subprocess
import sys
import threading
import time
import zipfile
from pathlib import Path
try:
from . import lab, runtime
except ImportError:
import lab
import runtime
ROOT = lab.ROOT
WORK = ROOT / ".local/perf-bench"
VANILLA = ROOT / ".local/minecraft/26.3"
SEED = 1234
WORLD = "PerfBench"
# (name, from xyz, to xyz, pitch): straight 192-block flights over fixed
# scenery of seed 1234 (the same paths as tools/graphics_bench.py).
SCENES = [
("village", (-960.0, 125.0, 80.0), (-960.0, 125.0, 272.0), 35.0),
("forest", (-1100.0, 120.0, -560.0), (-1100.0, 120.0, -368.0), 25.0),
("ocean", (-300.0, 85.0, -420.0), (-300.0, 85.0, -228.0), 10.0),
("mountain", (560.0, 260.0, 860.0), (752.0, 260.0, 860.0), 30.0),
]
# One options.txt for every game: vanilla's own keys, which Horizon reads too.
# The graphics settings are those of vanilla's default preset (Fancy) with the
# preset set to Custom, so the render distance is the one asked for (a preset
# would replace it on load). maxFps 260 is "Unlimited"; inactivityFpsLimit
# minimized turns off the 30 FPS limit both games apply after a minute without
# input. Everything not listed stays at vanilla's default.
OPTIONS = {
"version": "5023",
"graphicsPreset": '"custom"',
"renderDistance": "{distance}",
"simulationDistance": "12",
"biomeBlendRadius": "2",
"prioritizeChunkUpdates": "1",
"ao": "true",
"renderClouds": '"true"',
"cloudRange": "64",
"particles": "0",
"mipmapLevels": "4",
"entityShadows": "true",
"entityDistanceScaling": "1.0",
"menuBackgroundBlurriness": "5",
"cutoutLeaves": "true",
"improvedTransparency": "false",
"weatherRadius": "10",
"maxAnisotropyBit": "1",
"textureFiltering": "1",
"maxFps": "260",
"enableVsync": "false",
"inactivityFpsLimit": '"minimized"',
"fov": "0.0",
"guiScale": "0",
"fullscreen": "false",
"pauseOnLostFocus": "false",
"onboardAccessibility": "false",
"skipMultiplayerWarning": "true",
"tutorialStep": "none",
"narrator": "0",
"soundCategory_master": "0.0",
"realmsNotifications": "false",
}
# The official launcher's default JVM arguments for 26.3: what players run.
LAUNCHER_JVM = ["-XX:+UnlockExperimentalVMOptions", "-XX:+UseG1GC", "-XX:G1NewSizePercent=20",
"-XX:G1ReservePercent=20", "-XX:MaxGCPauseMillis=50", "-XX:G1HeapRegionSize=32M"]
PERF_MODS = ["sodium", "lithium", "ferrite-core", "immediatelyfast"]
def stats(frame_ms):
"""Every number of a flight, from its frame times in milliseconds.
The x% low is the average frame time of the slowest x% of frames, as
frames per second (so a single 2-second freeze shows in the 0.1% low of a
10,000-frame flight)."""
times = sorted(frame_ms)
n = len(times)
if n == 0:
return {"frames": 0}
total = sum(times)
pct = lambda p: times[min(n, max(1, math.ceil(n * p))) - 1]
def low(share):
slowest = times[n - max(1, math.ceil(n * share)):]
return 1000.0 * len(slowest) / sum(slowest)
buckets = [4.17, 8.33, 16.67, 33.33, 50.0, 100.0]
hist = {}
lo = 0.0
for hi in buckets + [float("inf")]:
key = f"<{hi:g}ms" if hi != float("inf") else f">={buckets[-1]:g}ms"
hist[key] = sum(1 for t in times if lo <= t < hi) / n
lo = hi
return {
"frames": n,
"seconds": total / 1000.0,
"avg_fps": n * 1000.0 / total,
"low_1pct_fps": low(0.01),
"low_01pct_fps": low(0.001),
"p50_ms": pct(0.5),
"p95_ms": pct(0.95),
"p99_ms": pct(0.99),
"p999_ms": pct(0.999),
"max_ms": times[-1],
"histogram": hist,
}
class Sampler(threading.Thread):
"""Samples a process's resident memory every quarter second (ps, so the
same reading for every game and OS)."""
def __init__(self, pid):
super().__init__(daemon=True)
self.pid, self.samples, self.stop = pid, [], threading.Event()
def run(self):
while not self.stop.is_set():
out = subprocess.run(["ps", "-o", "rss=", "-p", str(self.pid)], capture_output=True, text=True)
if out.stdout.strip():
self.samples.append((time.time(), int(out.stdout.split()[0]) * 1024))
self.stop.wait(0.25)
def summary(self, windows):
"""Peak over the whole run; steady = median over the flight windows."""
if not self.samples:
return {}
during = [rss for t, rss in self.samples if any(a <= t <= b for a, b in windows)]
return {
"rss_peak_mb": max(rss for _, rss in self.samples) / 2**20,
"rss_steady_mb": statistics.median(during) / 2**20 if during else None,
"rss_samples": len(self.samples),
}
def nvidia_memory_mb(pid):
"""Video memory nvidia-smi attributes to the process, if it can."""
if not shutil.which("nvidia-smi"):
return None
out = subprocess.run(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
capture_output=True, text=True)
for line in out.stdout.splitlines():
p, _, mem = line.partition(",")
if p.strip() == str(pid):
return float(mem)
return None
def write_options(game_dir, distance):
game_dir.mkdir(parents=True, exist_ok=True)
text = "".join(f"{k}:{v.format(distance=distance)}\n" for k, v in OPTIONS.items())
(game_dir / "options.txt").write_text(text)
return text
def scene_list(args):
picked = args.scenes.split(",") if args.scenes else [s[0] for s in SCENES]
return [s for s in SCENES if s[0] in picked]
def yaw(start, end):
return math.degrees(math.atan2(-(end[0] - start[0]), end[2] - start[2]))
# Horizon -------------------------------------------------------------------
class Control:
def __init__(self, path):
self.path = path
def __call__(self, line, timeout=600):
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
s.settimeout(timeout)
s.connect(str(self.path))
s.sendall((line + "\n").encode())
data = b""
while not data.endswith(b"\n"):
chunk = s.recv(1 << 20)
if not chunk:
break
data += chunk
reply = json.loads(data)
if isinstance(reply, dict) and "error" in reply:
raise RuntimeError(f"{line}: {reply['error']}")
return reply
def poll(control, check, limit, interval=0.05):
"""Calls control('status') until check(status); returns (status, time)."""
deadline = time.time() + limit
while time.time() < deadline:
try:
status = control("status", timeout=10)
if check(status):
return status, time.time()
except (OSError, ValueError, RuntimeError):
pass
time.sleep(interval)
raise TimeoutError(f"not ready after {limit} s")
def wait_loaded(control, near, limit):
"""Waits until the player is spawned (near(position)), nothing is queued and
the resident and generated chunk counts have not changed for a second, as
the vanilla driver does; returns the time of the last change.
(pending_chunks alone misses work the threads have already taken; sections
are remeshed all the time as blocks change, so meshes are not counted.)"""
deadline = time.time() + limit
last, changed = None, time.time()
while time.time() < deadline:
try:
s = control("status", timeout=10)
except (OSError, ValueError, RuntimeError):
s = {}
player = s.get("player")
if not player or not near(player["position"]) or s.get("pending_chunks") != 0 or s.get("screen") == "title":
last, changed = None, time.time()
else:
counts = (s["chunks"], s["generated_chunks"])
if counts != last:
last, changed = counts, time.time()
elif time.time() - changed >= 1.0:
return changed
time.sleep(0.05)
raise TimeoutError(f"not loaded after {limit} s")
def run_horizon(args, distance):
game_dir = WORK / "horizon"
shutil.rmtree(game_dir, ignore_errors=True)
options = write_options(game_dir, distance)
sock = Path("/tmp") / f"horizon-bench-{os.getpid()}.sock"
sock.unlink(missing_ok=True)
client = Path(args.build).resolve() / "horizon-client"
command = [str(client), "--resources", str(Path(args.resources).resolve()),
"--control-socket", str(sock), "--no-pause", "--weather", "clear", "--time", "6000"]
if args.offscreen:
command.append("--offscreen")
env = dict(os.environ, HORIZON_NO_AUDIO="1")
log = (WORK / f"horizon-rd{distance}.log").open("w")
started = time.time()
proc = subprocess.Popen(command, cwd=game_dir, env=env, stdout=log, stderr=subprocess.STDOUT)
sampler = Sampler(proc.pid)
sampler.start()
control = Control(sock)
result = {"options_txt": options, "scenes": {}}
windows = []
try:
title, at = poll(control, lambda s: s.get("screen") == "title" and s.get("renderer"), 300)
result["renderer"] = title["renderer"]
result["startup"] = {"title_s": at - started}
control(f"open {WORLD} {SEED} creative")
result["startup"]["world_s"] = wait_loaded(control, lambda p: True, args.timeout) - started
for name, start, end, pitch in scene_list(args):
control("time 6000")
meshed = control("status")["meshed_sections"]
control("tp {} {} {}".format(*start))
tp = time.time()
# A zero-length flight holds the player at the start, flying.
control("fly {0} {1} {2} {0} {1} {2} 0.1 {3}".format(*start, pitch))
control(f"look {yaw(start, end):.3f} {pitch}")
near = lambda p: sum((a - b) ** 2 for a, b in zip(p, start)) < 4.0
scene = {"load_s": wait_loaded(control, near, args.timeout) - tp}
# Section meshes built while loading and while flying (Horizon only).
loaded_meshes = control("status")["meshed_sections"]
scene["meshes_load"] = loaded_meshes - meshed
time.sleep(args.settle)
loaded_meshes = control("status")["meshed_sections"]
begin = time.time()
summary = control("fly {} {} {} {} {} {} {} {} raw".format(*start, *end, args.seconds, pitch))
windows.append((begin, time.time()))
scene.update(stats(summary.pop("frame_ms")))
scene["horizon_summary"] = {k: summary.get(k) for k in ("cpu", "gpu", "phases_avg_ms", "slow_phases_avg_ms")}
status = control("status")
scene["meshes_flight"] = status["meshed_sections"] - loaded_meshes
renderer = status.get("renderer") or {}
scene["gpu_allocated_mb"] = (renderer.get("gpu_allocated_bytes") or 0) / 2**20 or None
scene["nvidia_mb"] = nvidia_memory_mb(proc.pid)
result["scenes"][name] = scene
print_scene(args.game, distance, name, scene)
finally:
try:
control("quit", timeout=60)
except (OSError, ValueError, RuntimeError):
pass
stop(proc)
sampler.stop.set()
sock.unlink(missing_ok=True)
result["memory"] = sampler.summary(windows)
return result
# Vanilla -------------------------------------------------------------------
def vanilla_manifest():
if not (VANILLA / "manifest.json").exists():
sys.exit(f"{VANILLA} missing: prepare Horizon first (tools/prepare_native_client.py)")
return json.loads((VANILLA / "manifest.json").read_text())
def vanilla_classpath(manifest):
"""The official launcher's client libraries for this OS, from your own prepared copy."""
jars = [VANILLA / "jars/client.jar"]
for artifact in lab.client_library_artifacts(manifest):
jars.append(VANILLA / "libraries/client" / artifact["path"])
return jars
def extract_natives(jars, target):
target.mkdir(parents=True, exist_ok=True)
for jar in jars:
if "natives-" not in jar.name:
continue
with zipfile.ZipFile(jar) as archive:
for name in archive.namelist():
if name.endswith((".so", ".dylib", ".jnilib", ".dll")):
out = target / Path(name).name
if not out.exists():
out.write_bytes(archive.read(name))
def build_bench_mod(jars):
"""Compiles the dev-only driver mod against your own client.jar."""
source = ROOT / "mods/horizon/src/bench"
build = WORK / "mod"
classes = build / "classes"
shutil.rmtree(classes, ignore_errors=True)
classes.mkdir(parents=True)
sources = sorted(map(str, (source / "java").rglob("*.java")))
subprocess.run(["javac", "--release", "25", "-proc:none", "-encoding", "UTF-8", "-d", str(classes),
"-classpath", os.pathsep.join(map(str, jars)), *sources], check=True)
output = build / "horizon-bench.jar"
with zipfile.ZipFile(output, "w", compression=zipfile.ZIP_DEFLATED) as archive:
for path in sorted(classes.rglob("*.class")):
archive.write(path, path.relative_to(classes).as_posix())
for path in sorted((source / "resources").rglob("*")):
if path.is_file():
archive.write(path, path.relative_to(source / "resources").as_posix())
return output
def perf_mod_jars():
"""Sodium, Lithium, FerriteCore and ImmediatelyFast for 26.3 (downloaded from
Modrinth, checked against packs/mods.lock.json, never committed), plus
Fabric API when one of them needs it."""
lock = json.loads((ROOT / "packs/mods.lock.json").read_text())
jars, needs_api = [], False
for name in PERF_MODS:
project = lock["projects"][name]
jar = runtime.fetch(project["url"], ROOT / ".local/pack-cache" / project["filename"],
project["hashes"]["sha512"], "sha512")
with zipfile.ZipFile(jar) as archive:
depends = json.loads(archive.read("fabric.mod.json")).get("depends", {})
needs_api |= any(d == "fabric-api" or d.startswith("fabric-") and d != "fabricloader" for d in depends)
jars.append(jar)
if needs_api:
api = lock["projects"]["fabric-api"]
jars.append(runtime.fetch(api["url"], ROOT / ".local/pack-cache" / api["filename"],
api["hashes"]["sha512"], "sha512"))
return jars
def run_vanilla(args, distance):
manifest = vanilla_manifest()
game_jars = vanilla_classpath(manifest)
fabric = runtime.fabric_libraries()
mod = build_bench_mod(game_jars + fabric)
game_dir = WORK / args.game
shutil.rmtree(game_dir, ignore_errors=True)
options = write_options(game_dir, distance)
mods = game_dir / "mods"
mods.mkdir()
shutil.copy(mod, mods)
loaded_mods = ["horizon-bench"]
if args.game == "vanilla-mods":
for jar in perf_mod_jars():
shutil.copy(jar, mods)
loaded_mods.append(jar.name)
natives = WORK / "natives"
extract_natives(game_jars, natives)
script = WORK / "script.json"
script.write_text(json.dumps({
"world": WORLD, "seed": SEED, "settle": args.settle, "seconds": args.seconds, "timeout": args.timeout,
"scenes": [{"name": n, "from": a, "to": b, "pitch": p} for n, a, b, p in scene_list(args)],
}))
events = WORK / f"{args.game}-rd{distance}.events.jsonl"
events.unlink(missing_ok=True)
assets = WORK / "assets"
(assets / "indexes").mkdir(parents=True, exist_ok=True)
java = Path(os.environ["JAVA_HOME"]) / "bin/java" if os.environ.get("JAVA_HOME") else "java"
command = [str(java), f"-Xmx{args.heap}", *LAUNCHER_JVM,
*runtime.conditional_client_jvm_options(manifest),
"--enable-native-access=ALL-UNNAMED",
f"-Djava.library.path={natives}", f"-Dorg.lwjgl.system.SharedLibraryExtractPath={natives}",
f"-Dhorizon.bench.script={script}", f"-Dhorizon.bench.out={events}",
"-Dfabric.log.disableAnsi=true",
"-cp", os.pathsep.join(map(str, game_jars + fabric)),
"net.fabricmc.loader.impl.launch.knot.KnotClient",
"--username", "Bench", "--uuid", "00000000000000000000000000000000", "--accessToken", "0",
"--offlineDeveloperMode", "--version", "26.3", "--gameDir", str(game_dir),
"--assetsDir", str(assets), "--assetIndex", manifest["assetIndex"]["id"],
"--width", str(args.width), "--height", str(args.height)]
env = dict(os.environ, ALSOFT_DRIVERS="null")
if sys.platform.startswith("linux"):
libs = [p for p in (os.environ.get("NIX_LD_LIBRARY_PATH", ""), "/run/opengl-driver/lib") if p]
env["LD_LIBRARY_PATH"] = ":".join(libs + [env.get("LD_LIBRARY_PATH", "")]).strip(":")
log = (WORK / f"{args.game}-rd{distance}.log").open("w")
started = time.time()
proc = subprocess.Popen(command, cwd=game_dir, env=env, stdout=log, stderr=subprocess.STDOUT)
sampler = Sampler(proc.pid)
sampler.start()
nvidia = None
try:
deadline = started + args.timeout * (len(scene_list(args)) + 2)
while proc.poll() is None and time.time() < deadline:
time.sleep(0.5)
if nvidia is None:
nvidia = nvidia_memory_mb(proc.pid)
finally:
stop(proc)
sampler.stop.set()
lines = [json.loads(l) for l in events.read_text().splitlines()] if events.exists() else []
by = lambda name: [e for e in lines if e["event"] == name]
if not by("done") or by("timeout"):
raise RuntimeError(f"vanilla run failed: {lines[-1:] if lines else 'no events'}; see {log.name}")
title, world = by("title")[0], by("world")[0]
result = {
"options_txt": options, "mods": loaded_mods, "heap": args.heap,
"renderer": {k: title[k] for k in ("gpu", "vendor", "backend", "driver", "size",
"render_distance", "simulation_distance")},
"startup": {"title_s": title["epoch_ms"] / 1000 - started,
"world_s": world["ready_epoch_ms"] / 1000 - started},
"scenes": {},
}
windows = []
loaded = {e["scene"]: e for e in by("loaded")}
flights = {e["scene"]: e for e in by("fly")}
for flown in by("flown"):
name = flown["scene"]
windows.append((flights[name]["epoch_ms"] / 1000, flown["epoch_ms"] / 1000))
scene = {"load_s": loaded[name]["load_ms"] / 1000, **stats(flown["frame_ms"]),
"chunks": flown["chunks"], "nvidia_mb": nvidia}
result["scenes"][name] = scene
print_scene(args.game, distance, name, scene)
result["memory"] = sampler.summary(windows)
return result
def stop(proc):
try:
proc.wait(timeout=60)
except subprocess.TimeoutExpired:
proc.terminate()
try:
proc.wait(timeout=20)
except subprocess.TimeoutExpired:
proc.kill()
def print_scene(game, distance, name, s):
print(f" {game:<12} rd {distance:>2} {name:<9} load {s['load_s']:6.1f} s {s['avg_fps']:7.1f} fps"
f" 1% {s['low_1pct_fps']:6.1f} 0.1% {s['low_01pct_fps']:6.1f}"
f" p50 {s['p50_ms']:6.2f} p99 {s['p99_ms']:7.2f} ms", flush=True)
# Report --------------------------------------------------------------------
def machine():
info = {"host": platform.node(), "os": f"{platform.system()} {platform.release()}",
"arch": platform.machine(), "python": platform.python_version(), "cpus": os.cpu_count()}
if sys.platform == "darwin":
q = lambda key: subprocess.run(["sysctl", "-n", key], capture_output=True, text=True).stdout.strip()
info["cpu"], info["ram_gb"] = q("machdep.cpu.brand_string"), int(q("hw.memsize") or 0) / 2**30
else:
for line in Path("/proc/cpuinfo").read_text().splitlines():
if line.startswith("model name"):
info["cpu"] = line.split(":", 1)[1].strip()
break
info["ram_gb"] = os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") / 2**30
return info
def table(labels):
"""Markdown tables comparing saved runs (benchmarks/perf/<label>.json)."""
runs = [json.loads((ROOT / "benchmarks/perf" / f"{l}.json").read_text()) for l in labels]
names = [r["label"] for r in runs]
print("| RD | scene | " + " | ".join(f"{n} avg / 1% / 0.1% low FPS" for n in names) + " |")
print("| --- | --- |" + " --- |" * len(runs))
distances = list(runs[0]["distances"])
for rd in distances:
for scene in runs[0]["distances"][rd]["scenes"]:
cells = []
for r in runs:
s = r["distances"].get(rd, {}).get("scenes", {}).get(scene)
cells.append(f"{s['avg_fps']:.0f} / {s['low_1pct_fps']:.0f} / {s['low_01pct_fps']:.0f}" if s else "–")
print(f"| {rd} | {scene} | " + " | ".join(cells) + " |")
print()
print("| RD | " + " | ".join(f"{n}: title / world s, scene load s, RSS steady / peak MB" for n in names) + " |")
print("| --- |" + " --- |" * len(runs))
for rd in distances:
cells = []
for r in runs:
d = r["distances"].get(rd)
if not d:
cells.append("–")
continue
loads = [s["load_s"] for s in d["scenes"].values()]
m = d.get("memory", {})
cells.append(f"{d['startup']['title_s']:.1f} / {d['startup']['world_s']:.1f}, "
f"{statistics.mean(loads):.1f}, {m.get('rss_steady_mb') or 0:.0f} / {m.get('rss_peak_mb') or 0:.0f}")
print(f"| {rd} | " + " | ".join(cells) + " |")
def main():
p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
p.add_argument("--game", choices=["horizon", "vanilla", "vanilla-mods"], default="horizon")
p.add_argument("--label", help="result name (default <game>-<host>)")
p.add_argument("--distances", default="8,16,32")
p.add_argument("--scenes", help="comma-separated subset of " + ",".join(s[0] for s in SCENES))
p.add_argument("--seconds", type=float, default=12.0, help="flight duration per scene")
p.add_argument("--settle", type=float, default=2.0, help="seconds held at the start before each flight")
p.add_argument("--timeout", type=float, default=300.0, help="seconds allowed for each load")
p.add_argument("--width", type=int, default=1280, help="vanilla window width (Horizon's is 1280x720)")
p.add_argument("--height", type=int, default=720)
p.add_argument("--heap", default="2G", help="vanilla -Xmx (the official launcher's default is 2G)")
p.add_argument("--offscreen", action="store_true",
help="Horizon renders into a texture and does not wait for the window (locked macOS sessions)")
p.add_argument("--resources", default=str(ROOT / ".local/horizon-client/26.3"))
p.add_argument("--build", default=str(ROOT / "target/release"))
p.add_argument("--note", help="free text stored with the results")
p.add_argument("--table", nargs="+", metavar="LABEL", help="print markdown tables of saved runs, and exit")
args = p.parse_args()
if args.table:
table(args.table)
return
label = args.label or f"{args.game}-{platform.node().split('.')[0]}"
WORK.mkdir(parents=True, exist_ok=True)
commit = subprocess.run(["git", "rev-parse", "--short", "HEAD"], cwd=ROOT,
capture_output=True, text=True).stdout.strip() or None
report = {
"label": label, "game": args.game, "date": datetime.date.today().isoformat(), "commit": commit,
"machine": machine(), "offscreen": args.offscreen, "seed": SEED, "flight_seconds": args.seconds,
"settle_seconds": args.settle, "note": args.note,
"scenes": {n: {"from": a, "to": b, "pitch": c} for n, a, b, c in scene_list(args)},
"distances": {},
}
run = run_horizon if args.game == "horizon" else run_vanilla
for d in [int(x) for x in args.distances.split(",")]:
load_before = os.getloadavg()
result = run(args, d)
result["load_before"], result["load_after"] = load_before, os.getloadavg()
report["distances"][str(d)] = result
out = ROOT / "benchmarks/perf" / f"{label}.json"
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(report, indent=1) + "\n")
print(out)
if __name__ == "__main__":
main()