horizon-airdef/docs/version-ports.md

# Version ports: from a new Minecraft release to Horizon

Goal: every Minecraft Java release is turned into Rust by Horizon with as little human work as possible. A new
release is detected, downloaded and prepared on the Mac mini, compared with the version Horizon pins, everything
data-driven is regenerated, every parity harness runs against it, and what is left is a ranked list of agent-sized
work items, each a GitHub issue and a Kira card. This page is the design; `tools/version_port.py` implements it
(status at the end).

The downloads are for development only, on the Mac mini: nothing from them is ever committed, published or shipped
to players. Players still prepare their own copy from their official launcher (`horizon-prepare`), as today.

## How Horizon pins and prepares 26.3 today

- **The pin.** `research-lock.json` `minecraft`: the version id, the sha1 of Mojang's version JSON (and of the
  republished ones it accepts), the client and server jar sha1s, the Java major version. Vineflower, REA and JADX
  are pinned beside it.
- **The vanilla data.** `tools/lab.py prepare` imports the client jar, the server jar and the client libraries from
  the user's own launcher install, unpacks the server bundler into `server-game.jar`, and writes the class
  inventories; `tools/lab.py decompile` runs Vineflower on both jars. On the Mac mini all of this lives once in
  `~/code/horizon-parity/shared-local/` (`minecraft/26.3/{jars,libraries,sources,reports}`, 366 MB), and every
  worktree links to it from `.local/`. Release builds never do this: `horizon-prepare` imports the user's copy.
- **The data generator.** `horizon-prepare` runs `java -DbundlerMainClass=net.minecraft.data.Main -jar server.jar
  --reports` into `.local/datagen/out/reports/`: `blocks.json`, `registries.json`, `packets.json`,
  `commands.json`, `datapack.json`, item default components, biome parameters. It also runs Horizon's Java
  exporters (`mods/horizon/src/test/java/dev/horizonlab/parity/Vanilla*Exporter.java`) against the jar for block
  behaviours, worldgen bundles and the like. CI does the same through `tools/ci/vanilla-data.sh`, keyed by the
  version in `research-lock.json`.
- **Version facts baked into the code.** The client jar's own `version.json` states them (26.3: world version 5023,
  protocol 777, resource pack 97.1, data pack 121.0, Java 25); Horizon copies them by hand today:

  | Where | What |
  | --- | --- |
  | `crates/horizon-core/src/lib.rs` | `VANILLA_VERSION` |
  | `crates/horizon-protocol/src/lib.rs` | `PROTOCOL_VERSION`, `GAME_VERSION` |
  | `crates/horizon-protocol/src/packets/configuration.rs` | the `minecraft:core` known pack version |
  | `crates/horizon-world/src/lib.rs`, `crates/horizon-client/src/game/commands/extra.rs` | `DATA_VERSION` (twice) |
  | `crates/horizon-client/src/save.rs`, `crates/horizon-world/src/datafix/files.rs`, `game/commands/extra.rs` | the version name (three times) |
  | `crates/horizon-client/src/crash.rs`, `bin/horizon-ctl/launch.rs`, `launcher/src/backend.rs` | `"26.3"` fallbacks |
  | `tools/perf_bench.py`, `tools/vanilla_client.py` | the vanilla client's `--version` |

- **Generated tables** (each has its generator in `tools/`):

  | Output | Generator | Input |
  | --- | --- | --- |
  | `crates/horizon-protocol/src/ids.rs` | `generate_packet_ids.py` | `packets.json` report |
  | `crates/horizon-client/src/lan/tracking.rs` | `generate_entity_tracking.py` | decompiled `EntityTypes` |
  | `crates/horizon-client/src/lan/command_suggestions.rs` | `generate_command_suggestions.py` | `VanillaCommandSuggestionsExporter` |
  | `crates/horizon-client/src/lan/command_order.rs` | `generate_lan_command_order.py` | vanilla's command tree |
  | `crates/horizon-client/src/command/vanilla_order.rs` | `generate_command_order.py` | command-parse harness export |
  | `crates/horizon-world/src/datafix/block_state_data.txt` | `gen_block_state_data.py` | decompiled `BlockStateData` |
  | `crates/horizon-world/src/datafix/flat_tables.rs` | `gen_flat_tables.py` | 1.13 flattening fixers (never change) |

  Everything else data-driven (block behaviours, items, worldgen, models, sounds, languages) is not committed: the
  game reads what `horizon-prepare` exports from the user's own jar, so it follows a new version by itself as long
  as the exporters still compile and the formats still parse.

- **The proof.** ~300 harnesses (`tools/test_*_parity.py`) each compare a Rust probe with a Java oracle that calls
  vanilla's own classes; `tools/parity.py` runs them and records each one's dependencies.

## The pipeline

```
detect ─► prepare NEW ─► diff OLD..NEW ─► regenerate ─► harnesses on NEW ─► port report ─► issues + cards
 (daily)   (download,     (data, protocol,  (generators   (candidate          (ranked work    (agents port,
           verify,         registries,       per kind)     worktree)           items)          pin bump PR,
           datagen,        assets, classes                                                     release)
           class index)    → Horizon map)
```

`tools/version_port.py` has one subcommand per stage and `port NEW` runs them all. Each stage writes its result
under the version store and can be rerun alone.

### The version store

`~/code/horizon-parity/versions/<id>/` (`HORIZON_VERSIONS` overrides it), next to `shared-local/` and never inside
it:

```
versions/26.4/
  version.json            Mojang's version JSON (sha1 checked against the manifest)
  jars/client.jar         sha1 checked against version.json
  jars/server.jar         the bundler; server-game.jar beside it, unpacked with sha256 checks
  datagen/reports/...     the data generator's --reports output
  classes.json            the class index: every class's methods, each with a normalized bytecode hash
  facts.json              world/protocol/pack/Java versions from the jar's own version.json
  prepared.json           what was made, with which tool versions
versions/diff-26.3-26.4/  the diff, the source diffs of changed classes, the port report
```

About 150 MB a version (jars 120 MB, reports 20 MB, index 10 MB): client libraries are not needed (the data
generator runs from the bundler, which carries its own) and the full Vineflower tree is not kept (only changed
classes are decompiled, below). The store keeps at most three versions, the pinned one (whose files already exist in
`shared-local/`, so it is only indexed, not copied), the previous one and the candidate; `version_port.py gc`
deletes the rest and every scratch directory. Snapshots, if prepared, are deleted when the next one is.

### Detect

`version_port.py detect` reads `https://piston-meta.mojang.com/mc/game/version_manifest_v2.json`, compares the
latest release (and, with `--snapshots`, the latest snapshot) with the pin and with
`versions/detect-state.json`, and prints what is new; exit status 0 means "nothing new", 10 "new release",
11 "new snapshot". It needs only the network, so it can run anywhere:

- **Mac mini, launchd (chosen):** a `~/fleet` launchd agent runs `version_port.py detect --then port` daily at
  14:00 UTC (Mojang publishes on weekdays around 10:00-14:00 UTC), in a dedicated worktree of `origin/main`,
  at `~/fleet/load prio low`. The whole port runs where the vanilla data, Java and the harnesses are.
- **GitHub Actions:** a scheduled job could run `detect` alone and open a "Minecraft X is out" issue; the Mac mini
  would still have to do the rest, so it adds a hop for nothing. Kept as the fallback if the Mac mini is down for
  long.

Snapshots are optional and only ever diffed (`port --snapshot`: prepare, diff, report, no issues), as an early
warning: agents can read what 26.4 changes weeks before it ships.

### Prepare

`version_port.py prepare NEW`:

1. Fetch the manifest, the version JSON (check its sha1 against the manifest), the client and server jars (check
   their sha1s against the version JSON); `lab.download` already refuses partial or mismatched files. Releases
   since 26.1 ship unobfuscated, so no mappings are fetched; an obfuscated version (`client_mappings` present)
   stops here, as `lab.py prepare` does.
2. Unpack the server bundler (`lab.unpack_server`: one version, sha256 per entry).
3. Read the jar's `version.json` into `facts.json`. If its Java major is newer than the JDK on the Mac mini, stop
   with a single work item, "install JDK N" (a Ludovic task: fleet toolchains).
4. Run the data generator (`--reports`) with that JDK into `datagen/reports/`.
5. Index the classes of `client.jar` and `server-game.jar` into `classes.json`: for each class its superclass,
   interfaces, fields, and methods; each method's hash is over its bytecode with every constant-pool reference
   replaced by what it names (class, member, descriptor, literal), so recompiling unchanged code with a shifted
   constant pool hashes the same, and lambda bodies (`lambda$name$N`) are folded into the method that declares them
   so renumbered lambdas don't show as changes. This is exact where a decompiled-source diff is noisy, and takes
   seconds instead of the 10 minutes of a full Vineflower run.

The pinned version is indexed from `shared-local/` the same way, without copying its jars.

### Diff

`version_port.py diff OLD NEW` writes `diff.json` (and readable sections of the report), one section per kind:

| Kind | From | What it lists |
| --- | --- | --- |
| Facts | `facts.json` | protocol, world (data) version, pack formats, Java |
| Packets | `packets.json` | per state and direction: added, removed, renumbered packets |
| Registries | `registries.json` | per registry: added and removed entries, protocol-id shifts, new registries |
| Blocks | `blocks.json` | added and removed blocks; per block: properties and values, state count, default state |
| Item components | `components/item` | per item: added, removed and changed default components |
| Commands | `commands.json` | added and removed nodes, changed arguments and executability |
| Data packs | jar `data/minecraft/**` | added, removed, changed files by kind (recipe, loot table, worldgen, tags...), with a JSON path diff for changed files |
| Assets | jar `assets/minecraft/**` | `en_us.json` keys added, removed, changed; models, blockstates, textures, sounds added, removed, changed (by hash) |
| Classes | `classes.json` | added, removed and changed classes; per changed class the methods added, removed and changed (name and descriptor), field changes, hierarchy changes |

Changed classes are then decompiled, only those, on both sides: Vineflower runs on a small jar holding just those
classes (with the full jar as a library, so types resolve), and a unified diff per class goes into
`diff-OLD-NEW/sources/`. These diffs stay on the Mac mini: decompiled game code is never committed or posted
(`tools/check_public_tree.py`).

### From vanilla classes to Horizon modules

The diff matters only through what it changes in Horizon. `version_port.py map` builds `vanilla_map.json`, from
the repository alone, in seconds:

1. **Oracles → classes.** Each Java oracle and exporter under `mods/horizon/src/test/java/dev/horizonlab/parity/`
   names the vanilla classes it imports and uses (`net.minecraft.*` imports and fully qualified names).
2. **Harnesses → oracles and Rust.** Each `tools/test_*_parity.py` names its oracle class; its probe (a crate's
   example or binary named in the script) and its `PARITY_RUST_SCOPE` give the Rust files it proves.
3. **Rust → classes.** Horizon's doc comments name the vanilla code they port (`/// SharedConstants.getProtocolVersion()`,
   `//! Fixes of 1.21.6 to 26.3`, `EntityType.Builder clientTrackingRange`...): every CamelCase identifier in a
   comment that is the simple name of exactly one vanilla class (or `Outer.Inner`, or `Class.method`) maps that
   Rust file, and the method when named, to the class.

A changed vanilla class then resolves to the Horizon files that port it and the harnesses that prove them; a
changed method resolves more precisely when a comment names it. A class with Horizon code but no harness, or with
neither, is reported as such.

### Regenerate

Each generated output has a generator entry in `version_port.py` (`regen`): its command, its input in the version
store, the files it writes. On a candidate worktree (`port/NEW` from `origin/main`, with `.local/` linking to the
new version), `regen` runs them all and records, per generator: unchanged, regenerated (with the `--stat`), or
failed (the generator's own error: a source pattern no longer matches, a report changed shape). Two additions make
this complete:

- `version_consts`: a generator for the version facts table above, from `facts.json`. Before it, a small refactor
  makes every Rust site use the `horizon-core` / `horizon-protocol` constants instead of its own copy, so the pin
  bump touches one generated file.
- `horizon-prepare` against the new version: its Java exporters compile and run against the new jar, or fail to
  compile, which names exactly the vanilla API they used that changed.

Generators that read decompiled sources (`EntityTypes`, `BlockStateData`) get those two files from the changed-class
decompilation, so no full Vineflower run is needed for them either.

### Run every harness against the new version

In the candidate worktree, with `research-lock.json` pinned to NEW and `.local/` linking to the new store
(`.local/minecraft/NEW` holds `jars/`, `datagen` the new reports, `horizon-client` what the candidate's
`horizon-prepare` made), `tools/parity.py --all` runs every harness. Each result is one of:

- **pass**: Horizon already matches NEW there;
- **oracle does not compile**: vanilla's API changed under the oracle (javac names the symbol);
- **fails**: behaviour changed (or Horizon's data reading broke); the log has the first differing case.

The known failures of `main` (`tools/ci/parity_issues.py`'s audit) are subtracted, so only what NEW broke counts.

### The port report

`diff-OLD-NEW/report.md` and `report.json`:

1. Summary: facts changed, counts per kind, generators (clean / regenerated / failed), harnesses (pass / oracle
   broken / fail).
2. Work items, ranked. An item is one Horizon area, not one vanilla class: changed classes are grouped by the
   Horizon module (or, for unported code, by vanilla package) that owns them, so each item is a PR-sized task. Rank:
   1. blocks the switch: protocol or save format (packets, registries the protocol syncs, data version and the new
      DataFixer schemas), a failed generator, an exporter that no longer compiles;
   2. a harness that fails or whose oracle no longer compiles;
   3. a changed class Horizon ports but no harness covers (the risky kind: nothing will catch a difference);
   4. new content in data packs and assets the game reads at runtime (usually nothing to do: listed so it gets a
      look);
   5. changed vanilla code Horizon doesn't port yet (informational, no issue).

   Within a rank, by the number of changed methods. Each item lists the vanilla classes and methods, the Horizon
   files and harnesses, the data-diff excerpts that concern it, and the local path of its source diffs.
3. Everything regenerated cleanly, so the reader knows what not to look at.

`version_port.py publish` opens one GitHub issue per item of ranks 1 to 3 (labels `version-port` and
`port-NEW`, title `[NEW] <area>: <summary>`; an open issue with that title is updated instead), with the structured
diff attached (names, signatures, data-report diffs, harness output; no decompiled code), and a Kira card per issue
(`card_add` with the issue URL, column `todo`) so contributors and agents can claim them. Rank 4 and 5 go into one
tracking issue.

### The switch and the release

Work items are fixed on `port/NEW` branches stacked on the candidate (or as normal PRs into it). The pin bump PR is
the candidate branch itself: `research-lock.json` (version, sha1s, Java), the regenerated tables and
`version_consts`, the oracle fixes, and the ports. It merges when the report says no rank 1 or 2 item is left: every
harness passes against NEW, CI's vanilla data (`tools/ci/vanilla-data.sh`) and the Mac mini's `shared-local/` have
been switched to NEW (the version store's files are copied in, then `lab.py decompile` runs once for the full
sources), and a playtest shows a world loads and a vanilla NEW client can join a Horizon LAN world.

A Horizon release then ships as usual (`docs/release-flow.md`), labelled `feature` with "Minecraft NEW" in its
title. Its `research-lock.json` makes `horizon-prepare` ask for NEW, so players must have played NEW once in the
official launcher; the launcher already reads each release's game version from its `research-lock.json`, so players
who haven't can keep the previous Horizon release. Supporting two Minecraft versions in one build is out of scope.

## What stays manual, and how small it gets

Automatic: detecting, downloading, verifying, the data generator, every diff, the class-to-module map, regenerating
data-driven tables, rerunning every harness, the report, issues and cards, and, once the report is clean, the pin
bump branch.

Manual, by agents (and Ludovic's review):

- **New mechanics** (a new mob, block behaviour, worldgen feature, UI screen): someone has to port the new code. The
  pipeline hands them the exact classes and methods, the decompiled diff, the Horizon module and the harness to
  extend, so the agent starts at the code instead of searching for it.
- **Changed behaviour under a failing harness**: the harness names the case, the class diff names the method.
- **Oracles whose vanilla API moved**: javac names the symbol; usually a short fix.
- **New harnesses** for rank-3 items, where Horizon ports code that no harness checks.
- **New JDK**: a fleet toolchain install, once per Java bump.
- **Judgment on the switch**: merging the pin bump and releasing stay Ludovic's call.

## Replay: 26.2 → 26.3

Horizon was written against 26.3 from its first commit, so its code is the ground truth of what porting 26.2 to 26.3
takes. On the Mac mini, `prepare 26.2` took 9 s (136 MB), `diff 26.2 26.3` 6 s and `report` under a second.

| What 26.3 needed (in Horizon's code) | What the report found |
| --- | --- |
| Protocol 777, DataVersion 5023, pack formats | rank 1: protocol 776 → 777, world version 4903 → 5023, resource pack 88.0 → 97.1, data pack 107.1 → 121.0, with every site stating them (`PROTOCOL_VERSION`, both `DATA_VERSION`s, the `"26.3"` names) |
| `ids.rs`: `ADD_TRANSIENT_BLOCK`, `POST_EFFECTS` (configuration and play), `SWING_ANIMATION`, serverbound `PUNCH` instead of `SWING`; handlers in `packets/play.rs`, `lan/hub.rs` | rank 1: exactly those packets, 131 renumbered, those files, and "regenerate `ids.rs`". Regenerating from 26.3's report reproduces the committed file byte for byte (after rustfmt); from 26.2's it differs by 274 lines |
| DataFixers 4996-5016 in `datafix/fixes_c.rs` | rank 1: schemas V4996, V4996_1, V4997, V5000 and 16 new fixes (`MergeTerrainChunkStatusFix`, `ExplorerMapItemFix`, `SwingAnimationComponentSplitFix`...), with `datafix/` |
| `swing_animation` split into `attack_animation` and `interact_animation` (`first_person.rs`, `game/hand.rs`, `item_model.rs`) | rank 1: data component types +13 -2, those files among the ones naming them; per component, 1,537 items affected |
| `lan/tracking.rs` and the command tables | rank 1: regenerate `tracking.rs`, `command_suggestions.rs`, `command_order.rs`, `vanilla_order.rs` |
| 133 methods that exist only in 26.3 and that Horizon's comments name (`Minecraft.startAttack`, `MultiPlayerGameMode.piercingAttack`, `SkyRenderer.buildEndSky`, `GameRenderer.render3dHud`...) | all 133: 132 in rank 2-3 items with the method named, on the file that ports it; 1 in the save-format item |
| 49 classes that exist only in 26.3 and that Horizon names (`ComputeCommand`, `PostEffectCommand`, `LivingEntity.SwingState`, `BlockTransformer`, `StrawBedBlock`...) | all 49, in rank 4 "New in <package>" items |

The report has 499 items: 10 of rank 1, 311 of rank 2, 96 of rank 3, 80 of rank 4, and 2 rank-5 summaries. Rank 1 is
exact. Ranks 2 and 3 are leads, not all work: 173 of their 407 items have a changed method that Horizon's comments
name. The rest are classes whose changed methods Horizon doesn't name. Many of those are mechanical (26.3 renamed
`GuiGraphics` to `GuiGraphicsExtractor` and reworked the render pipeline), and 105 are "widely named" buckets
(classes such as `Entity` that many files mention in passing). Running every harness against the new version (with
`--results`) is what turns a rank-2 lead into a confirmed failure; the replay cannot do that, because the oracles are
written against 26.3.

## Implementation status

```sh
python3 tools/version_port.py detect --snapshots     # exit 10: new release, 11: new snapshot
python3 tools/version_port.py prepare 26.3           # the pinned version: indexes .local, downloads nothing
python3 tools/version_port.py prepare 26.4           # downloads, verifies, runs the data generator (about 10 s)
python3 tools/version_port.py diff 26.3 26.4         # summary; the full diff in versions/diff-26.3-26.4/diff.json
python3 tools/version_port.py report 26.3 26.4 [--results .local/parity-runner/state.json]   # report.md, report.json
python3 tools/version_port.py port 26.4             # prepare both, diff, report
python3 tools/version_port.py gc                     # keep the pinned version and the newest two others
python3 tools/vanilla_map.py versions/26.3/classes.json net.minecraft.world.food.FoodData   # its Rust files and harnesses
```

| Slice | Status |
| --- | --- |
| This design | this page |
| `detect`, `prepare`, `diff` (facts, packets, registries, blocks, items, commands, data, assets), `gc` | done |
| Class index (`tools/class_index.py`), class diff with moves, `vanilla_map.json` (`tools/vanilla_map.py`) | done |
| Port report (`tools/port_report.py`, `report`), `port` | done |
| Changed-class decompilation, `regen`, `version_consts` refactor, candidate harness run, `publish`, launchd agent | later |