Files
horizon-airdef/docs/benchmark-report.md
T
Ludovic Marie e0ab3f8fcc Benchmark report format v1: docs/benchmark-report.md and crates/horizon-bench
One versioned JSON report for every benchmark Horizon writes (the game's
--benchmark, the launcher's measurements, tools/perf_bench.py): the local log
a player can attach and, with consent, what the telemetry upload sends.
Summaries over repeats (mean, stdev, 95% CI) and fixed-edge frame-time
histograms, never raw frame streams; no host, user or path in uploads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 07:00:54 +02:00

7.0 KiB

Benchmark report format (version 1)

Every benchmark Horizon writes uses this one JSON format: horizon-client --benchmark, the launcher's own measurements, and tools/perf_bench.py (which also writes it for vanilla and vanilla with mods). The same file is the local log a player can attach to an issue and, only with the player's consent, what the telemetry upload sends (minus the local-only fields below). The types, the statistics and the hardware detection are in crates/horizon-bench; nothing else builds a report by hand.

A report holds summaries and histogram counts, never raw frame streams: a 10-minute run is a few kilobytes whatever the frame rate. Measured numbers are rounded to 3 decimals.

Example

{
  "format": "horizon-benchmark-report",
  "version": 1,
  "kind": "game",
  "created": "2026-10-15T14:03:27Z",
  "app": { "name": "horizon-client", "version": "0.21.0", "commit": "e96e1571" },
  "hardware": {
    "os": "macos", "os_version": "27.0", "arch": "aarch64",
    "cpu": "Apple M4 Pro", "cpu_threads": 14, "ram_mb": 65536,
    "gpu": { "name": "Apple M4 Pro", "vendor": "apple", "driver": "", "backend": "metal", "kind": "integrated" },
    "display": { "width": 2560, "height": 1440, "scale": 2.0, "refresh_hz": 60 }
  },
  "conditions": {
    "scenario": "flight", "seed": "1234", "render_distance": 16, "simulation_distance": 12,
    "window": [1280, 720], "vsync": false, "repeats": 3, "warmup_s": 2.0, "power": "ac", "load_avg_1m": 1.2
  },
  "metrics": {
    "startup.title_ms": { "n": 3, "mean": 812.0, "stdev": 21.5, "min": 790.0, "max": 833.0, "ci95": 53.4 },
    "memory.rss_peak_mb": { "n": 3, "mean": 1210.0, "stdev": 4.0, "min": 1206.0, "max": 1214.0, "ci95": 9.9 }
  },
  "frames": [
    {
      "phase": "village", "repeat": 0, "frames": 4920, "seconds": 12.0,
      "avg_fps": 410.0, "low_1pct_fps": 120.3, "low_01pct_fps": 61.0,
      "p50_ms": 2.31, "p95_ms": 3.8, "p99_ms": 6.9, "p999_ms": 14.2, "max_ms": 21.0,
      "cpu_p50_ms": 1.4, "gpu_p50_ms": 1.9,
      "histogram": [0, 0, 3011, 1520, 301, 62, 15, 7, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    }
  ]
}

Fields

format, version. Always "horizon-benchmark-report" and 1. A reader rejects other formats and newer versions. Adding an optional field keeps the version; renaming, removing or changing the meaning of one (histogram edges included) makes it 2.

kind. "game" (a game run: Horizon, or vanilla from tools/perf_bench.py) or "launcher".

created. When the run ended, UTC, YYYY-MM-DDTHH:MM:SSZ.

app. What was measured: name (horizon-client, horizon-launcher, minecraft), version, optionally commit (Horizon builds) and mods ([{ "name", "version" }], vanilla with mods).

hardware. os (macos, linux, windows), os_version, arch, cpu (brand string), cpu_threads, ram_mb; gpu (name, vendor, driver, backend: metal, vulkan, dx12, gl; kind: integrated, discrete, virtual, cpu, other) and display (the screen the window was on: width, height in physical pixels, scale, refresh_hz) when known. Any field can be missing.

conditions. What has to match for two reports to be compared, all optional: scenario, seed, render_distance, simulation_distance, window ([width, height] actually rendered), vsync, fps_cap (absent: unlimited), offscreen (rendered without presenting: not a player number), repeats, warmup_s, power (ac, battery), load_avg_1m (before the run; above about 1 the machine was busy), backend (when forced). Readers compare reports only with equal scenario, distances, window, vsync, fps_cap and offscreen.

metrics. Scalar measurements, by name; each is a summary over the run's repeats: n, mean, stdev (sample standard deviation, null when n is 1), min, max, ci95 (half-width of the 95% confidence interval of the mean, Student's t, null when n is 1). The unit is the name's suffix: _ms, _s, _mb, _bytes, _fps, _pct, _mspt, _w, _c (degrees Celsius), _count. Names in use:

Name What
startup.title_ms process start to the first frame of the title screen
startup.world_ms process start to in-world, every chunk around the spawn loaded and drawn
chunks.<phase>.load_ms teleport to every chunk in view distance loaded and drawn
chunks.generated_per_s, chunks.meshed_per_s throughput while filling the view distance
chunks.meshes_per_section_count meshes built per section (re-meshing; 1 is ideal)
fps.<phase>.avg_fps, fps.<phase>.low_1pct_fps, fps.<phase>.low_01pct_fps per-phase figures across repeats
tick.mspt, tick.tps integrated server tick time and rate
memory.rss_peak_mb, memory.rss_steady_mb resident memory: peak over the run, median while flying
memory.gpu_mb video memory the game allocated, where readable
power.package_w, power.battery_drain_pct power draw, battery used per hour (laptops)
thermal.cpu_c, thermal.throttled_pct temperature, share of the run spent throttled
launcher.cold_start_ms, launcher.warm_start_ms launcher process start to its first frame
launcher.rss_idle_mb, launcher.rss_peak_mb, launcher.cpu_idle_pct launcher memory and CPU
launcher.release_list_ms fetching the release list
launcher.download_ms, launcher.install_ms, launcher.prepare_ms getting a build ready, with conditions.scenario first-install, update or reuse
launcher.play_to_game_frame_ms Play pressed to the game's first frame
launcher.binary_bytes, game.binary_bytes size of the executables

New names can be added without a new version; the table lists the ones the tools write.

frames. One entry per measured phase and repeat (a scene's flight, the launcher's page switching): phase, repeat, frames, seconds, avg_fps, low_1pct_fps and low_01pct_fps (the average frame time of the slowest 1% or 0.1% of frames, as FPS), p50_ms, p95_ms, p99_ms, p999_ms, max_ms, cpu_p50_ms and gpu_p50_ms when measured, and histogram: frame counts per bucket. The bucket edges are fixed by the version, so histograms from any number of reports add up:

0, 1, 2, 3, 4.17, 5, 6, 7, 8.33, 10, 12, 14, 16.67, 20, 25, 33.33, 40, 50, 66.67, 100, 150, 250, 500, 1000 ms

Bucket i counts frames from edge i (included) to edge i + 1; the last bucket is 1000 ms and over (24 buckets). 4.17, 8.33, 16.67 and 33.33 ms are 240, 120, 60 and 30 FPS.

Local only

label (a name the person running it chose) and local (paths of the logs the run wrote next to the report) stay on the machine: the upload removes them (Report::for_upload). Nothing in a report identifies the player, the machine's name or the world: no host name, user name, path or address.

Where reports go

  • The game: benchmarks/<kind>-<created>-<scenario>.json in the game folder.
  • The launcher: benchmarks/ in the launcher's data folder.
  • tools/perf_bench.py: one report per game and render distance, benchmarks/perf/<label>-rd<N>.json in the repository checkout.