Merge pull request #359 from ludovic111/feat/version-port-regen

🤖 Agent: merged (version-port tooling, rebased on main). Ludovic asked on 10-21 to clear the PR queue.
This commit is contained in:
Ludovic Marie
2026-10-21 18:51:15 +02:00
committed by GitHub
9 changed files with 518 additions and 33 deletions
@@ -135,7 +135,7 @@ fn tree(d: &Dispatcher, node: usize) -> Value {
fn main() -> horizon_client::Result<()> {
let args: Vec<String> = std::env::args().collect();
if args.get(1).map(String::as_str) == Some("tree") {
let d = vanilla::release_dispatcher();
let d = &vanilla::registered(false);
let mut v = tree(d, 0);
v["type"] = json!("root");
fs::write(&args[2], v.to_string())?;
+8 -1
View File
@@ -36,6 +36,14 @@ pub fn build() -> Dispatcher {
/// The tree Commands registers; `dev` adds the commands vanilla only
/// registers in development builds (raid, warden_spawn_tracker).
pub fn build_with(dev: bool) -> Dispatcher {
let mut d = registered(dev);
apply_vanilla_order(&mut d);
d
}
/// The tree in the order Horizon's code registers it, before
/// [`vanilla_order`](super::vanilla_order) puts children in vanilla's order
/// (tools/generate_command_order.py compares the two).
pub fn registered(dev: bool) -> Dispatcher {
let mut d = Dispatcher::new();
let root = 0;
@@ -403,7 +411,6 @@ pub fn build_with(dev: bool) -> Dispatcher {
super::tree_misc::register(&mut d, root);
super::tree_world::register(&mut d, root);
super::tree_extra::register(&mut d, root);
apply_vanilla_order(&mut d);
d
}
+53 -14
View File
@@ -169,20 +169,35 @@ neither, is reported as such.
### Regenerate
Each generated output has a generator entry in `version_port.py` (`regen`): its command, its input in the version
store, the files it writes. On a candidate worktree (`port/NEW` from `origin/main`, with `.local/` linking to the
new version), `regen` runs them all and records, per generator: unchanged, regenerated (with the `--stat`), or
failed (the generator's own error: a source pattern no longer matches, a report changed shape). Two additions make
this complete:
`version_port.py regen NEW` (`tools/port_regen.py`) makes the candidate and regenerates every table in it:
- `version_consts`: a generator for the version facts table above, from `facts.json`. Before it, a small refactor
makes every Rust site use the `horizon-core` / `horizon-protocol` constants instead of its own copy, so the pin
bump touches one generated file.
- `horizon-prepare` against the new version: its Java exporters compile and run against the new jar, or fail to
compile, which names exactly the vanilla API they used that changed.
1. **The candidate**: a worktree of the branch `port/NEW` (from `--base`, default `origin/main`) in
`~/code/horizon-parity/ports/NEW` (`HORIZON_PORTS`), with `research-lock.json` pinned to NEW: version, the
version JSON's sha1, the jars' sha1s, Java. Every tool reads the version from the lock (`lab.VERSION`,
`lab.GAME`, `lab.RESOURCES`), so nothing else in the tree names the old one.
2. **Its `.local/`**, a real directory: `minecraft/NEW` is prepared by `tools/lab.py prepare` itself, from a
launcher-shaped copy of the download in the version store (`versions/NEW/launcher`: the version JSON, the client
jar and its libraries, verified by sha1) into `versions/NEW/game`, so the candidate's jars, libraries and
manifest are exactly what a player's install gives. `datagen/out/reports` links to NEW's reports, `tools` to the
shared tools. Nothing is written into `shared-local/`.
3. **Decompiled sources**: only the classes a generator parses (`EntityTypes`, `BlockStateData`), each decompiled
alone with the full jar and its libraries on Vineflower's path, into `versions/NEW/sources/` (seconds instead of
a 10-minute full run). The source is not the full run's text (inner classes of the jar's other classes are
imported, chained calls are not wrapped), but what the generators read from it is: from 26.3's they write the
committed tables byte for byte.
4. **The generators**, every row of the table above, each recorded in `diff-OLD-NEW/regen.json` as unchanged,
regenerated (with its `--numstat`) or failed (with the end of its output and, when an exporter or oracle no
longer compiles, javac's errors one per line: they name the vanilla API that changed). Rust outputs are
rustfmt'ed. The pin and what regenerated are committed on `port/NEW`, locally; nothing is pushed.
Generators that read decompiled sources (`EntityTypes`, `BlockStateData`) get those two files from the changed-class
decompilation, so no full Vineflower run is needed for them either.
The report then lists what was regenerated in its summary and makes each failed generator a rank-1 item, instead of
guessing which tables the diff made stale. `--only NAME...` reruns some generators; `port NEW` runs `regen` after
`diff` (`--no-regen` for snapshots).
Three generators run vanilla through Horizon's Java oracles (`VanillaCommandSuggestionsExporter`,
`VanillaCommandParseOracle`) and one builds `command_parse_fixture` (Horizon's command tree in its own registration
order, before `vanilla_order.rs` reorders it), so `regen` needs Java and Cargo. `horizon-prepare` (the resources the
harnesses read) is the next stage's first step.
### Run every harness against the new version
@@ -276,6 +291,27 @@ name. The rest are classes whose changed methods Horizon doesn't name. Many of t
`--results`) is what turns a rank-2 lead into a confirmed failure; the replay cannot do that, because the oracles are
written against 26.3.
### `regen`
`regen 26.2` from a branch pinned to 26.3, then `regen 26.3 --base port/26.2`, both from versions downloaded from
Mojang (`prepare --download`):
| Generator | 26.3 → 26.2 | 26.2 → 26.3 |
| --- | --- | --- |
| `version_consts` | regenerated (+8 -8: protocol 776, data version 4903, packs 88.0 / 107.1, build time) | regenerated, back to the committed file |
| `packet_ids` | regenerated (+134 -142) | regenerated (+142 -134), back to the committed file |
| `entity_tracking` | regenerated (+10 -13) | regenerated (+13 -10), back to the committed file |
| `block_state_data` | regenerated (+1 -1: the version in its header; `BlockStateData` is unchanged) | regenerated, back to the committed file |
| `lan_command_order`, `vanilla_command_order` | failed: `VanillaCommandParseOracle.java:81: no suitable constructor found for CommandSourceStack(...)` and 2 more | unchanged |
| `command_suggestions` | failed: `ParityData.java:45: cannot find symbol: variable WORLD (class RegistryLayer)`, `WORLD_REGISTRIES` | unchanged |
After the round trip, `port/26.3` equals the branch it started from except `research-lock.json`'s five lines of
republished 26.3 version JSONs it accepts (which a port cannot know). So every table Horizon commits is what the
generators make from vanilla's data, and going 26.3 → 26.2 fails exactly where Horizon's oracles use 26.3-only API,
with javac naming it. The report for 26.2 → 26.3 then has 5 rank-1 items instead of 10: the 5 "regenerate" guesses
are replaced by the generators' results in its summary. Both regens took under three minutes, most of it building
`command_parse_fixture`.
## Implementation status
```sh
@@ -283,8 +319,9 @@ python3 tools/version_port.py detect --snapshots # exit 10: new release, 11:
python3 tools/version_port.py prepare 26.3 # the pinned version: indexes .local, downloads nothing
python3 tools/version_port.py prepare 26.4 # downloads, verifies, runs the data generator (about 10 s)
python3 tools/version_port.py diff 26.3 26.4 # summary; the full diff in versions/diff-26.3-26.4/diff.json
python3 tools/version_port.py regen 26.4 # ports/26.4 on port/26.4: pinned, every table regenerated
python3 tools/version_port.py report 26.3 26.4 [--results .local/parity-runner/state.json] # report.md, report.json
python3 tools/version_port.py port 26.4 # prepare both, diff, report
python3 tools/version_port.py port 26.4 # prepare both, diff, regen, report
python3 tools/version_port.py gc # keep the pinned version and the newest two others
python3 tools/vanilla_map.py versions/26.3/classes.json net.minecraft.world.food.FoodData # its Rust files and harnesses
```
@@ -296,4 +333,6 @@ python3 tools/vanilla_map.py versions/26.3/classes.json net.minecraft.world.food
| Class index (`tools/class_index.py`), class diff with moves, `vanilla_map.json` (`tools/vanilla_map.py`) | done |
| Port report (`tools/port_report.py`, `report`), `port` | done |
| `version_consts`: one generated file for the version facts (`tools/gen_version_consts.py`) | done |
| Changed-class decompilation, `regen`, candidate harness run, `publish`, launchd agent | later |
| Tools read the version from the lock (`lab.VERSION`, `lab.GAME`, `lab.RESOURCES`, `lab.version_info()`) | done |
| `regen` (`tools/port_regen.py`): candidate worktree, its `.local/`, the generators' classes decompiled alone, every generator, `regen.json` in the report | done |
| Candidate harness run, `publish`, launchd agent | later |
+52
View File
@@ -0,0 +1,52 @@
import unittest
from tools import port_regen as regen
LOCK = {"minecraft": {"version": "26.3", "manifest_sha1": "m3", "accepted_manifest_sha1": ["m3old"],
"accepted_manifest_note": "republished", "client_sha1": "c3", "server_sha1": "s3",
"java_major": 25},
"vineflower": {"version": "1.12.0"}}
def manifest(version, client, server, java=25):
return {"id": version, "downloads": {"client": {"sha1": client}, "server": {"sha1": server}},
"javaVersion": {"majorVersion": java}}
class PinTests(unittest.TestCase):
def test_another_version_gets_a_new_block(self):
pinned = regen.pinned_lock(LOCK, manifest("26.4", "c4", "s4", 26), "m4")
self.assertEqual(pinned["minecraft"], {"version": "26.4", "manifest_sha1": "m4", "client_sha1": "c4",
"server_sha1": "s4", "java_major": 26})
self.assertEqual(pinned["vineflower"], LOCK["vineflower"])
def test_the_same_version_keeps_its_block(self):
for sha1 in ["m3", "m3old"]:
self.assertIs(regen.pinned_lock(LOCK, manifest("26.3", "c3", "s3"), sha1), LOCK)
def test_the_same_version_with_other_jars_is_repinned(self):
self.assertEqual(regen.pinned_lock(LOCK, manifest("26.3", "c3b", "s3"), "m3b")["minecraft"]["client_sha1"],
"c3b")
class CompileErrorTests(unittest.TestCase):
OUTPUT = """/x/mods/horizon/src/test/java/dev/horizonlab/parity/ParityData.java:45: error: cannot find symbol
var context = TagLoader.buildUpdatedLookups(layers.getAccessForLoading(RegistryLayer.WORLD), staticTags);
^
symbol: variable WORLD
location: class RegistryLayer
/x/VanillaCommandParseOracle.java:12: error: incompatible types: int cannot be converted to String
2 errors
"""
def test_names_the_symbols(self):
self.assertEqual(regen.compile_errors(self.OUTPUT), [
"ParityData.java:45: cannot find symbol: variable WORLD (class RegistryLayer)",
"VanillaCommandParseOracle.java:12: incompatible types: int cannot be converted to String"])
def test_nothing_from_other_failures(self):
self.assertEqual(regen.compile_errors("expected the root's order to differ"), [])
if __name__ == "__main__":
unittest.main()
+23 -2
View File
@@ -62,11 +62,32 @@ def diff():
class PortReportTests(unittest.TestCase):
def build(self, results=None):
def build(self, results=None, regen=None):
with mock.patch.object(report, "version_sites", return_value=["crates/p/src/lib.rs:20"]), \
mock.patch.object(report, "files_naming", return_value=["crates/p/src/play.rs"]), \
mock.patch.object(report, "files_containing", return_value=["crates/w/src/blocks.rs"]):
return report.build(diff(), mapping(), "26.2", "26.3", {"id": "26.3"}, results)
return report.build(diff(), mapping(), "26.2", "26.3", {"id": "26.3"}, results, regen=regen)
def test_regen_results_replace_the_stale_tables(self):
regen = {"new": "26.3", "candidate": "/ports/26.3", "commit": "0123456789abcdef", "generators": {
"packet_ids": {"status": "regenerated", "outputs": ["crates/horizon-protocol/src/ids.rs"],
"stat": {"crates/horizon-protocol/src/ids.rs": [142, 134]}},
"entity_tracking": {"status": "unchanged", "outputs": ["crates/horizon-client/src/lan/tracking.rs"],
"stat": {}},
"lan_command_order": {"status": "failed", "outputs": ["crates/horizon-client/src/lan/command_order.rs"],
"error": "javac exited 1",
"compile_errors": ["ParityData.java:45: cannot find symbol: variable WORLD"]}}}
built = self.build(regen=regen)
self.assertFalse([i for i in built["items"] if i["kind"] == "regenerate"])
[failed] = [i for i in built["items"] if i["kind"] == "regen-failed"]
self.assertEqual(failed["rank"], 1)
self.assertIn("no longer compiles", failed["title"])
self.assertEqual(built["summary"]["generators"],
{"packet_ids": "regenerated", "entity_tracking": "unchanged", "lan_command_order": "failed"})
text = report.markdown(built)
self.assertIn("regenerated 1, unchanged 1, failed 1", text)
self.assertIn("`crates/horizon-protocol/src/ids.rs` (+142 -134)", text)
self.assertIn("- javac: `ParityData.java:45: cannot find symbol: variable WORLD`", text)
def test_ranks_and_kinds(self):
built = self.build()
+1 -1
View File
@@ -5,7 +5,7 @@ registers them in a different order. Brigadier keeps children in
registration order, and /help (smart usage) lists them that way.
python3 tools/test_command_parse_parity.py # exports .local/command-parse-parity/tree.json
command_parse_fixture tree rust_tree.json # Horizon's tree with child order
command_parse_fixture tree rust_tree.json # Horizon's tree in its registration order
generate_command_order.py VANILLA_TREE RUST_TREE
"""
import argparse
+311
View File
@@ -0,0 +1,311 @@
#!/usr/bin/env python3
"""Regenerate what Horizon derives from vanilla, for a new version, in a candidate worktree
(docs/version-ports.md, "Regenerate").
Used through `version_port.py regen NEW`:
1. the candidate: a worktree of `port/NEW` (from --base, default origin/main), its research-lock.json pinned to
NEW (version, version JSON, jars, Java);
2. its .local: NEW's jars, libraries and manifest prepared by tools/lab.py prepare from the version store (a
launcher-shaped copy of the download), NEW's data generator reports, the shared tools;
3. the two vanilla classes the generators parse (EntityTypes, BlockStateData) decompiled alone, with the full
jar and its libraries for types, into the version store;
4. every generator run there, each recorded as unchanged, regenerated (with its diff stat) or failed (with the
end of its output); the regenerated tree is committed on port/NEW, never pushed.
Nothing here writes into the shared .local (~/code/horizon-parity/shared-local): the candidate links only what
it reads.
"""
import json
import os
from pathlib import Path
import re
import shutil
import subprocess
import tempfile
import zipfile
try:
from . import lab
except ImportError:
import lab
# Classes whose decompiled source a generator parses: (side, class).
SOURCES = [("client", "net/minecraft/world/entity/EntityTypes"),
("server", "net/minecraft/util/datafix/fixes/BlockStateData")]
PARITY = "mods/horizon/src/test/java/dev/horizonlab/parity"
# --- the candidate -----------------------------------------------------------------------------------------
def git(*args, cwd=lab.ROOT, check=True):
return subprocess.run(["git", *args], cwd=cwd, capture_output=True, text=True, check=check).stdout
def candidate(version, base, path):
"""The port/VERSION worktree at path, made from base unless it exists."""
if (path / ".git").exists():
return path
path.parent.mkdir(parents=True, exist_ok=True)
branch = f"port/{version}"
exists = git("rev-parse", "--verify", "--quiet", f"refs/heads/{branch}", check=False).strip()
git("worktree", "add", *([str(path), branch] if exists else ["-b", branch, str(path), base]))
return path
def pinned_lock(lock, manifest, manifest_sha1):
"""research-lock.json pinned to the version this manifest describes. The same version keeps its block (and
the republished version JSONs it accepts); another version gets a new one."""
game = lock["minecraft"]
same = (game["version"] == manifest["id"]
and manifest_sha1 in [game["manifest_sha1"], *game.get("accepted_manifest_sha1", [])]
and game["client_sha1"] == manifest["downloads"]["client"]["sha1"]
and game["server_sha1"] == manifest["downloads"]["server"]["sha1"])
if same:
return lock
return {**lock, "minecraft": {
"version": manifest["id"], "manifest_sha1": manifest_sha1,
"client_sha1": manifest["downloads"]["client"]["sha1"],
"server_sha1": manifest["downloads"]["server"]["sha1"],
"java_major": manifest["javaVersion"]["majorVersion"],
}}
def pin(root, version_json):
lock_path = root / "research-lock.json"
lock = json.loads(lock_path.read_text())
pinned = pinned_lock(lock, json.loads(version_json.read_text()), lab.digest(version_json, "sha1"))
if pinned is not lock:
lab.write_json(lock_path, pinned)
def link(target, link_path):
link_path.parent.mkdir(parents=True, exist_ok=True)
if link_path.is_symlink() or link_path.exists():
if link_path.is_symlink() and Path(os.readlink(link_path)) == target:
return
raise SystemExit(f"{link_path} exists and is not a link to {target}; remove it")
link_path.symlink_to(target)
def launcher_tree(store, manifest):
"""A launcher-shaped copy of a downloaded version (versions/ID/ID.json and .jar, libraries/), so that
tools/lab.py prepare imports it exactly as it imports a player's install."""
tree, version = store / "launcher", manifest["id"]
versions = tree / "versions" / version
versions.mkdir(parents=True, exist_ok=True)
shutil.copyfile(store / "version.json", versions / f"{version}.json")
jar = versions / f"{version}.jar"
if not jar.exists():
os.link(store / "jars" / "client.jar", jar)
for artifact in lab.client_library_artifacts(manifest):
lab.download(artifact["url"], tree / "libraries" / artifact["path"], artifact["sha1"])
return tree
def prepare_local(root, version, found, store):
"""The candidate's .local: tools from the shared one, the version's prepared jars and its reports."""
shared = Path(os.environ.get("HORIZON_SHARED_LOCAL", Path.home() / "code/horizon-parity/shared-local"))
local = root / ".local"
if local.is_symlink():
raise SystemExit(f"{local} links to a shared directory; a candidate needs its own")
link(shared / "tools", local / "tools")
link(found["reports"], local / "datagen" / "out" / "reports")
jars = found["client"].parent
if jars.parent.name == version and (jars.parent / "prepared.json").exists() and store not in jars.parents:
link(jars.parent, local / "minecraft" / version) # The pinned version, prepared by lab.py already.
return
game = store / "game"
game.mkdir(exist_ok=True)
link(game, local / "minecraft" / version)
if not (game / "prepared.json").exists():
tree = launcher_tree(store, json.loads(found["version_json"].read_text()))
run([python(), "tools/lab.py", "prepare", "--minecraft-dir", tree, "--server-jar", found["server"]], root)
def python():
return shutil.which("python3") or "python3"
class Failed(RuntimeError):
"""A command that exited non-zero, with all it printed."""
def __init__(self, command, code, output):
super().__init__(f"{' '.join(map(str, command))[:300]} exited {code}:\n{output[-2500:]}")
self.output = output
def run(command, cwd, log=None, env=None):
"""Run a command; on failure raise Failed."""
result = subprocess.run([str(c) for c in command], cwd=cwd, capture_output=True, text=True, env=env)
if log:
log.write_text(result.stdout + result.stderr)
if result.returncode:
raise Failed(command, result.returncode, result.stdout + result.stderr)
return result.stdout
def compile_errors(output, limit=12):
"""javac's errors, one line each: "ParityData.java:45: cannot find symbol: variable WORLD (class
RegistryLayer)" names the vanilla API an oracle used that the new version changed."""
lines = output.splitlines()
found = []
for index, line in enumerate(lines):
match = re.match(r".*?([\w$]+\.java):(\d+): error: (.*)", line)
if not match:
continue
entry = f"{match[1]}:{match[2]}: {match[3]}"
after = [text.strip() for text in lines[index + 1:index + 6]]
symbol = next((text.split(":", 1)[1].strip() for text in after if text.startswith("symbol:")), None)
where = next((text.split(":", 1)[1].strip() for text in after if text.startswith("location:")), None)
found.append(entry + (f": {symbol}" if symbol else "") + (f" ({where})" if where else ""))
return found[:limit]
# --- decompiling the classes the generators parse ------------------------------------------------------------
def decompile_classes(root, version, store):
"""Decompile only SOURCES' classes (with their inner classes) into store/sources/SIDE, with the full jar and
the side's libraries on the decompiler's path so types resolve as in a full run."""
lock = json.loads((root / "research-lock.json").read_text())
vineflower = root / ".local" / "tools" / f"vineflower-{lock['vineflower']['version']}.jar"
prepared = json.loads((root / ".local" / "minecraft" / version / "prepared.json").read_text())
for side, name in SOURCES:
output = store / "sources" / side
if (output / f"{name}.java").exists():
continue
jar = root / prepared["jars"][side]
with tempfile.TemporaryDirectory(prefix=".decompile-", dir=store) as scratch:
scratch = Path(scratch)
small = scratch / "classes.jar"
with zipfile.ZipFile(jar) as full, zipfile.ZipFile(small, "w") as out:
for entry in full.namelist():
if entry == f"{name}.class" or (entry.startswith(f"{name}$") and entry.endswith(".class")):
out.writestr(entry, full.read(entry))
command = ["java", "-Xmx2G", "-jar", vineflower, "--folder", "--skip-extra-files=1", "--log-level=warn",
f"--add-external={jar}", *(f"--add-external={root / p}" for p in prepared["libraries"][side]),
small, scratch / "out"]
run(command, root)
target = output / f"{name}.java"
target.parent.mkdir(parents=True, exist_ok=True)
shutil.move(scratch / "out" / f"{name}.java", target)
# --- the generators ------------------------------------------------------------------------------------------
def compile_oracle(root, version, classpath, sources, build):
lock = json.loads((root / "research-lock.json").read_text())
classes = build / "classes"
classes.mkdir(parents=True, exist_ok=True)
run(["javac", "--release", lock["minecraft"]["java_major"], "-proc:none", "-sourcepath",
root / "mods/horizon/src/test/java", "-cp", os.pathsep.join(map(str, classpath)), "-d", classes,
*sources], root)
return classes
def classpaths(root, version):
prepared = json.loads((root / ".local" / "minecraft" / version / "prepared.json").read_text())
return {side: [root / prepared["jars"][side], *(root / p for p in prepared["libraries"][side])]
for side in ("client", "server")}
def gen_suggestions(root, version, store, build):
client = classpaths(root, version)["client"]
classes = compile_oracle(root, version, client, [root / PARITY / "VanillaCommandSuggestionsExporter.java"],
build / "suggestions")
tsv = build / "suggestions.tsv"
run(["java", "-Xmx1G", "-cp", os.pathsep.join(map(str, [classes, *client])),
"dev.horizonlab.parity.VanillaCommandSuggestionsExporter", tsv], build)
return [python(), "tools/generate_command_suggestions.py", tsv]
def gen_vanilla_order(root, version, store, build):
"""vanilla's tree (the command-parse oracle's export) against Horizon's (command_parse_fixture tree)."""
server = classpaths(root, version)["server"]
classes = compile_oracle(root, version, server, [root / PARITY / "VanillaCommandParseOracle.java"],
build / "parse")
run(["java", "-Xmx1G", "-cp", os.pathsep.join(map(str, [classes, *server])),
"dev.horizonlab.parity.VanillaCommandParseOracle", "tree", build], build)
env = dict(os.environ, CARGO_TARGET_DIR=str(root / ".local" / "port-target"))
jobs = os.environ.get("FLEET_LOAD_JOBS")
run(["cargo", "build", "-p", "horizon-client", "--example", "command_parse_fixture",
*(["-j", jobs] if jobs else [])], root, env=env)
fixture = root / ".local/port-target/debug/examples/command_parse_fixture"
run([fixture, "tree", build / "rust_tree.json"], root)
return [python(), "tools/generate_command_order.py", build / "tree.json", build / "rust_tree.json"]
# (name, outputs, how to get the generator's command: a list, or a function that prepares its input first).
GENERATORS = [
("version_consts", ["crates/horizon-core/src/version.rs"],
lambda root, version, store, build: [python(), "tools/gen_version_consts.py"]),
("packet_ids", ["crates/horizon-protocol/src/ids.rs"],
lambda root, version, store, build: [python(), "tools/generate_packet_ids.py"]),
("entity_tracking", ["crates/horizon-client/src/lan/tracking.rs"],
lambda root, version, store, build: [python(), "tools/generate_entity_tracking.py", "--sources",
store / "sources" / "client"]),
("block_state_data", ["crates/horizon-world/src/datafix/block_state_data.txt"],
lambda root, version, store, build: [python(), "tools/gen_block_state_data.py",
store / "sources/server" / f"{SOURCES[1][1]}.java"]),
("lan_command_order", ["crates/horizon-client/src/lan/command_order.rs"],
lambda root, version, store, build: [python(), "tools/generate_lan_command_order.py"]),
("command_suggestions", ["crates/horizon-client/src/lan/command_suggestions.rs"], gen_suggestions),
("vanilla_command_order", ["crates/horizon-client/src/command/vanilla_order.rs"], gen_vanilla_order),
]
def status(root, outputs):
"""unchanged, or regenerated with {path: [added, removed]} against the candidate's last commit."""
stat = {}
for line in git("diff", "--numstat", "HEAD", "--", *outputs, cwd=root).splitlines():
added, removed, path = line.split("\t")
stat[path] = [int(added), int(removed)]
return ("regenerated", stat) if stat else ("unchanged", {})
def run_generators(root, version, store, names=None):
"""Every generator in the candidate: {name: {status, outputs, stat | error}}."""
build = root / ".local" / "port-regen"
build.mkdir(parents=True, exist_ok=True)
results = {}
for name, outputs, command in GENERATORS:
if names and name not in names:
continue
print(f" {name}", flush=True)
try:
run(command(root, version, store, build), root, log=build / f"{name}.log")
rust = [root / o for o in outputs if o.endswith(".rs")]
if rust:
run(["rustfmt", "--edition", "2024", *rust], root)
state, stat = status(root, outputs)
results[name] = {"status": state, "outputs": outputs, "stat": stat}
except (RuntimeError, OSError, subprocess.CalledProcessError, KeyError, ValueError) as error:
git("checkout", "HEAD", "--", *outputs, cwd=root, check=False)
results[name] = {"status": "failed", "outputs": outputs, "error": str(error)[-2500:],
"compile_errors": compile_errors(getattr(error, "output", str(error)))}
print(f" {name}: {results[name]['status']}", flush=True)
return results
def commit(root, version, results):
"""Commit the pin and what regenerated cleanly on port/VERSION (local only)."""
paths = ["research-lock.json"] + [o for r in results.values() if r["status"] == "regenerated" for o in r["outputs"]]
git("add", "--", *paths, cwd=root)
if git("diff", "--cached", "--name-only", cwd=root).strip():
git("commit", "-q", "-m", f"Port to Minecraft {version}: pin it and regenerate (version_port.py regen)",
cwd=root)
return git("rev-parse", "HEAD", cwd=root).strip()
def regen(version, found, store, base, path, names=None):
"""The whole stage: {old, new, candidate, commit, generators}."""
old = json.loads(git("show", f"{base}:research-lock.json"))["minecraft"]["version"]
root = candidate(version, base, path)
pin(root, found["version_json"])
print(f"Candidate {root} (port/{version}, was {old})", flush=True)
prepare_local(root, version, found, store)
decompile_classes(root, version, store)
results = run_generators(root, version, store, names)
return {"old": old, "new": version, "candidate": str(root), "commit": commit(root, version, results),
"generators": results}
+32 -8
View File
@@ -104,7 +104,7 @@ def item(rank, kind, title, **details):
return {"rank": rank, "kind": kind, "title": title, **details}
def blocking_items(diff, mapping, pinned_facts, root):
def blocking_items(diff, mapping, pinned_facts, root, regen=None):
items = []
if diff["facts"]:
changes = ", ".join(f"{k} {a} -> {b}" for k, (a, b) in diff["facts"].items())
@@ -147,9 +147,17 @@ def blocking_items(diff, mapping, pinned_facts, root):
items.append(item(1, "report-format", f"blocks.json format: {', '.join(diff['blocks']['format'])}",
files=files_containing("blocks.json", root),
details={"fields": diff["blocks"]["format"]}))
for output, command, stale in GENERATORS:
if stale(diff):
items.append(item(1, "regenerate", f"Regenerate {output}", files=[output], command=command))
if regen is None:
for output, command, stale in GENERATORS:
if stale(diff):
items.append(item(1, "regenerate", f"Regenerate {output}", files=[output], command=command))
else:
for name, entry in regen["generators"].items():
if entry["status"] == "failed":
errors = entry.get("compile_errors", [])
what = "its vanilla exporter no longer compiles" if errors else "failed"
items.append(item(1, "regen-failed", f"Generator {name}: {what} on {regen['new']}",
files=entry["outputs"], error=entry["error"], compile_errors=errors))
return items
@@ -237,10 +245,11 @@ def content_items(diff, mapping):
return items
def build(diff, mapping, old, new, pinned_facts, results=None, root=ROOT):
def build(diff, mapping, old, new, pinned_facts, results=None, root=ROOT, regen=None):
"""The report: summary, ranked items and what came out clean. pinned_facts: the facts of the version the
code states today (to find where it states them); results: {harness: "pass" | "fail"} against NEW."""
items = blocking_items(diff, mapping, pinned_facts, root)
code states today (to find where it states them); results: {harness: "pass" | "fail"} against NEW; regen:
version_port.py regen's result (without it, the tables a diff makes stale are listed to regenerate)."""
items = blocking_items(diff, mapping, pinned_facts, root, regen)
items += code_items(diff, mapping, results) + content_items(diff, mapping)
items.sort(key=lambda i: (i["rank"], -i.get("score", 0), i["title"]))
for number, entry in enumerate(items, 1):
@@ -256,8 +265,12 @@ def build(diff, mapping, old, new, pinned_facts, results=None, root=ROOT):
"classes": {"added": len(classes["added"]), "removed": len(classes["removed"]),
"moved": len(classes["moved"]), "changed": len(classes["changed"]),
"horizon": len(classes["horizon"])},
"generators": [i["files"][0] for i in items if i["kind"] == "regenerate"],
"generators": ({name: e["status"] for name, e in regen["generators"].items()} if regen
else {i["files"][0]: "stale" for i in items if i["kind"] == "regenerate"}),
},
"regen": {"candidate": regen["candidate"], "commit": regen["commit"],
"stat": {p: s for e in regen["generators"].values() for p, s in e.get("stat", {}).items()}}
if regen else None,
"items": items, "clean": clean,
}
@@ -278,6 +291,13 @@ def markdown(report, limit=12):
lines.append(f"- work items by rank: " + ", ".join(f"{r}: {n}" for r, n in summary["ranks"].items()))
if report["clean"]:
lines.append(f"- unchanged: {', '.join(report['clean'])}")
if report.get("regen"):
generators = summary["generators"]
lines.append(f"- generators in `{report['regen']['candidate']}` at {report['regen']['commit'][:10]}: "
+ ", ".join(f"{s} {sum(1 for v in generators.values() if v == s)}"
for s in ("regenerated", "unchanged", "failed")))
for path, (added, removed) in report["regen"]["stat"].items():
lines.append(f" - regenerated `{path}` (+{added} -{removed})")
names = {1: "Blocks the switch", 2: "Proved by a harness: rerun it", 3: "Ported, no harness",
4: "New content", 5: "Not ported"}
for rank in sorted({i["rank"] for i in report["items"]}):
@@ -286,6 +306,10 @@ def markdown(report, limit=12):
lines.append(f"### #{entry['id']} {entry['title']}")
if entry.get("command"):
lines.append(f"- run: `{entry['command']}`")
if entry.get("compile_errors"):
lines += [f"- javac: `{e}`" for e in entry["compile_errors"][:limit]]
elif entry.get("error"):
lines.append("- error: `" + entry["error"].strip().splitlines()[-1][:300] + "`")
if entry.get("failed"):
lines.append(f"- failing harnesses: {', '.join(entry['failed'])}")
if entry.get("harnesses"):
+37 -6
View File
@@ -8,12 +8,15 @@
version_port.py diff OLD NEW facts, packets, registries, blocks, items, commands,
data packs, assets, lang and classes, with the Horizon
code each changed class concerns: diff-OLD-NEW/diff.json
version_port.py regen NEW [--base REF] a port/NEW worktree pinned to NEW with every generated table
regenerated (tools/port_regen.py): diff-OLD-NEW/regen.json
version_port.py report OLD NEW [--results S] the ranked port report: diff-OLD-NEW/report.md and .json
version_port.py port NEW [--old OLD] prepare both, diff, report
version_port.py port NEW [--old OLD] prepare both, diff, regen, report
version_port.py gc [--keep VERSION...] keep the pinned, the newest two others and --keep
The downloads are for development only: they go to the version store (~/code/horizon-parity/versions,
HORIZON_VERSIONS overrides it), never into the repository, and nothing from them is shipped. The pinned
HORIZON_VERSIONS overrides it), never into the repository, and nothing from them is shipped. Candidate
worktrees go to ~/code/horizon-parity/ports/NEW (HORIZON_PORTS overrides it). The pinned
version is not downloaded: its files are read where tools/lab.py prepared them (.local/).
detect exits 0 when nothing is new, 10 for a new release and 11 for a new snapshot (with --snapshots).
@@ -33,15 +36,17 @@ import urllib.request
import zipfile
try:
from . import class_index, lab, port_report, vanilla_map
from . import class_index, lab, port_regen, port_report, vanilla_map
except ImportError:
import class_index
import lab
import port_regen
import port_report
import vanilla_map
MANIFEST_URL = "https://piston-meta.mojang.com/mc/game/version_manifest_v2.json"
STORE = Path(os.environ.get("HORIZON_VERSIONS", Path.home() / "code" / "horizon-parity" / "versions"))
PORTS = Path(os.environ.get("HORIZON_PORTS", STORE.parent / "ports"))
STATES = ["handshake", "status", "login", "configuration", "play"]
# How many changed JSON paths one changed file keeps in diff.json.
PATH_LIMIT = 50
@@ -545,6 +550,20 @@ def diff(args):
return 0
# --- regen -------------------------------------------------------------------------------------------------
def regen(args):
version = safe_version(args.new)
result = port_regen.regen(version, paths(version), version_dir(version), args.base,
Path(args.worktree) if args.worktree else PORTS / version, args.only)
lab.write_json(diff_dir(result["old"], version) / "regen.json", result)
for name, entry in result["generators"].items():
stat = ", ".join(f"{p} +{a} -{r}" for p, (a, r) in entry.get("stat", {}).items())
print(f"{name}: {entry['status']}" + (f" ({stat})" if stat else ""))
print(f"\n{result['candidate']} at {result['commit'][:10]}; {diff_dir(result['old'], version) / 'regen.json'}")
return 1 if any(e["status"] == "failed" for e in result["generators"].values()) else 0
# --- report ------------------------------------------------------------------------------------------------
def harness_results(state_path):
@@ -563,7 +582,10 @@ def report(args):
result = json.loads((directory / "diff.json").read_text())
mapping = json.loads((directory / "vanilla_map.json").read_text())
results = harness_results(args.results) if args.results else None
built = port_report.build(result, mapping, args.old, args.new, load(pinned(), "facts.json"), results)
regenerated = directory / "regen.json"
regenerated = json.loads(regenerated.read_text()) if regenerated.exists() else None
built = port_report.build(result, mapping, args.old, args.new, load(pinned(), "facts.json"), results,
regen=regenerated)
lab.write_json(directory / "report.json", built)
(directory / "report.md").write_text(port_report.markdown(built))
print("\n".join(port_report.markdown(built).splitlines()[:args.lines]))
@@ -572,11 +594,13 @@ def report(args):
def port(args):
"""prepare both, diff, report."""
"""prepare both, diff, regen, report."""
old = args.old or pinned()
for version in (old, args.new):
prepare(argparse.Namespace(version=version, download=False, jobs=args.jobs))
diff(argparse.Namespace(old=old, new=args.new))
if not args.no_regen:
regen(argparse.Namespace(new=args.new, base=args.base, worktree=None, only=None))
return report(argparse.Namespace(old=old, new=args.new, results=args.results, lines=args.lines))
@@ -623,6 +647,11 @@ def main():
command = commands.add_parser("diff", help="diff two prepared versions")
command.add_argument("old")
command.add_argument("new")
command = commands.add_parser("regen", help="a candidate worktree pinned to NEW, every table regenerated")
command.add_argument("new")
command.add_argument("--base", default="origin/main", help="what port/NEW starts from (default origin/main)")
command.add_argument("--worktree", help="where (default: $HORIZON_PORTS/NEW)")
command.add_argument("--only", nargs="*", help="these generators only (names in tools/port_regen.py)")
command = commands.add_parser("report", help="the ranked port report of a diff")
command.add_argument("old")
command.add_argument("new")
@@ -633,13 +662,15 @@ def main():
command.add_argument("--old", help="default: the pinned version")
command.add_argument("--results", help="tools/parity.py's state.json from a run against NEW")
command.add_argument("--lines", type=int, default=60)
command.add_argument("--base", default="origin/main", help="what the candidate starts from")
command.add_argument("--no-regen", action="store_true", help="diff and report only (snapshots)")
command.add_argument("-j", "--jobs", type=int, default=int(os.environ.get("FLEET_LOAD_JOBS", 0)) or None)
command = commands.add_parser("gc", help="delete versions and scratch the store doesn't need")
command.add_argument("--keep", nargs="*", default=[])
command.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
STORE.mkdir(parents=True, exist_ok=True)
return {"detect": detect, "prepare": prepare, "diff": diff, "report": report, "port": port,
return {"detect": detect, "prepare": prepare, "diff": diff, "regen": regen, "report": report, "port": port,
"gc": gc}[args.command](args)