Merge pull request #11 from ludovic111/priority-dispatch-backoff
Bound rejected terrain dispatch retries without retaining the dispatcher lock
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
name: checks
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
permissions:
|
||||
contents: read
|
||||
@@ -56,6 +57,14 @@ jobs:
|
||||
run: python -c "from tools.build_mod import benchmark_headers; benchmark_headers()"
|
||||
- name: Test pack integrity and comparisons
|
||||
run: python -m unittest discover -s tests -v
|
||||
- name: Compile the private dispatch fixture
|
||||
run: python -c "from tools.build_mod import build_dispatch_fixture; build_dispatch_fixture()"
|
||||
- name: Install headless graphics libraries for applied-hook checks
|
||||
if: runner.os == 'Linux'
|
||||
run: sudo apt-get update && sudo apt-get install -y xvfb libgl1 libegl1 libvulkan1 mesa-vulkan-drivers libasound2t64 libxkbcommon0
|
||||
- name: Exercise the applied dispatcher hooks and upstream rejection path
|
||||
if: runner.os == 'Linux'
|
||||
run: python tools/test_dispatch.py
|
||||
- name: Upload original mod and pack manifests only
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
@@ -67,3 +76,4 @@ jobs:
|
||||
dist/debounce-queue-benchmark.txt
|
||||
dist/idle-queue-benchmark.txt
|
||||
dist/object-header-benchmark.txt
|
||||
dist/dispatch-integration.json
|
||||
|
||||
@@ -6,7 +6,7 @@ Client-first performance research for Minecraft Java **26.3**, preserving vanill
|
||||
|
||||
## Play on your Mac
|
||||
|
||||
Download **horizon-mac-starter-0.9.0.zip** from [Releases](https://github.com/ludovic111/minecraft-optimization-lab/releases), extract it and open `Mac-Start.command`. It verifies and installs official universal Prism Launcher in `~/Applications` if needed, then opens the balanced pack import. Sign into Microsoft **inside Prism yourself**. This project needs no credentials.
|
||||
Download **horizon-mac-starter-0.10.0.zip** from [Releases](https://github.com/ludovic111/minecraft-optimization-lab/releases), extract it and open `Mac-Start.command`. It verifies and installs official universal Prism Launcher in `~/Applications` if needed, then opens the balanced pack import. Sign into Microsoft **inside Prism yourself**. This project needs no credentials.
|
||||
|
||||
Use native **Java 25 ARM64**, maximum heap **6144 MiB**, minimum **512 MiB** on your 16 GB M2 Pro. Prism can download Java automatically. Start at 1280×800 or 1920×1200; full Retina resolution is a larger GPU workload. [START-HERE.txt](packs/START-HERE.txt) has details. The same `.mrpack` files import into Prism on Windows and Linux.
|
||||
|
||||
@@ -19,6 +19,7 @@ Use native **Java 25 ARM64**, maximum heap **6144 MiB**, minimum **512 MiB** on
|
||||
| adaptive-pacing (experiment) | 12 | 128 | 6144 MiB |
|
||||
| queue-wait (experimental) | 12 | 128 | 6144 MiB |
|
||||
| adaptive-queue (experimental) | 12 | 128 | 6144 MiB |
|
||||
| dispatch-retry (experimental) | 12 | 128 | 6144 MiB |
|
||||
| low-memory | 8 | none | 3072 MiB |
|
||||
| far-distance | 12 | 256 | 8192 MiB, with sufficient physical RAM |
|
||||
| cinematic | 12 | 128 | 6144 MiB |
|
||||
@@ -30,6 +31,10 @@ Distant LOD is simplified terrain. Nearby blocks retain full detail. Initial gen
|
||||
|
||||
The optional `queue-wait` pack yields during pending distant-terrain debounce retries; `adaptive-queue` combines it with the existing controllers. [Queue experiment and measurement limits](docs/debounce-queue.md).
|
||||
|
||||
Version 0.10 adds an isolated [rejected-dispatch retry experiment](docs/dispatch-retry.md).
|
||||
It bounds retry passes when a DH worker rejects queued work, preserving original
|
||||
requeue handling and allowing other executors to progress. Balanced keeps it off.
|
||||
|
||||
Version 0.9 also makes the queue's idle wait wakeable for incoming work, preserving
|
||||
its debounce rules and bounded cache-maintenance cadence. It remains part of the
|
||||
optional queue profiles, with separate wake-latency measurements.
|
||||
@@ -77,7 +82,7 @@ python3 tools/build_mod.py
|
||||
python3 tools/build_release.py
|
||||
```
|
||||
|
||||
Gitignored `dist/` contains eleven packs, our original instrumentation JAR, client/server starter ZIPs and SHA256SUMS. [mods.lock.json](packs/mods.lock.json) pins mod versions, exact dependency closure, SHA-1/SHA-512 and source/license; [fabric.lock.json](packs/fabric.lock.json) pins loader/library SHA-256. Packs use the [Modrinth format](https://support.modrinth.com/en/articles/8802351-modrinth-modpack-format-mrpack). They embed only original project code; third-party mods, game and assets download from their publishers.
|
||||
Gitignored `dist/` contains twelve packs, our original instrumentation JAR, client/server starter ZIPs and SHA256SUMS. [mods.lock.json](packs/mods.lock.json) pins mod versions, exact dependency closure, SHA-1/SHA-512 and source/license; [fabric.lock.json](packs/fabric.lock.json) pins loader/library SHA-256. Packs use the [Modrinth format](https://support.modrinth.com/en/articles/8802351-modrinth-modpack-format-mrpack). They embed only original project code; third-party mods, game and assets download from their publishers.
|
||||
|
||||
The server starter uses Lithium/FerriteCore and `/horizonserver bench <label> <seconds>`. [Server instructions](docs/server-quickstart.txt) cover setup and a baseline. Files are prepared before you accept Minecraft's EULA locally. A 50-active-player workload remains to be measured.
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"platform": "Linux-6.17.0-1022-azure-x86_64-with-glibc2.39",
|
||||
"architecture": "x86_64",
|
||||
"minecraft": "26.3",
|
||||
"distant_horizons": "3.3.4",
|
||||
"results": [
|
||||
{
|
||||
"enabled": false,
|
||||
"submission_allocated_bytes": 216987192,
|
||||
"submission_cpu_ns": 372949442,
|
||||
"submission_wall_ns": 401548875,
|
||||
"rejections": 63790,
|
||||
"tasks_completed": 2,
|
||||
"occupied_slots": 0
|
||||
},
|
||||
{
|
||||
"enabled": true,
|
||||
"submission_allocated_bytes": 103384,
|
||||
"submission_cpu_ns": 941367,
|
||||
"submission_wall_ns": 959453,
|
||||
"rejections": 6,
|
||||
"tasks_completed": 2,
|
||||
"occupied_slots": 0
|
||||
}
|
||||
],
|
||||
"notes": [
|
||||
"Synthetic full physical worker queue; both modes use the actual publisher dispatcher inside Fabric.",
|
||||
"Allocation and CPU measure the submission caller only; retry scheduler activity is excluded.",
|
||||
"Upstream keeps the caller blocked for the requested 200 ms observation window; patched returns earlier.",
|
||||
"Both modes verify exact task counts, recovered occupied slots and fixture worker shutdown.",
|
||||
"Not a game FPS, terrain completion, whole-game RAM, or hardware compatibility benchmark."
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
# Rejected dispatch retry experiment
|
||||
|
||||
A local game recording showed allocation samples repeatedly entering DH's
|
||||
`ThreadPoolExecutor.AbortPolicy`, exception construction and diagnostic strings.
|
||||
The direct allocation counters showed substantial allocation on render-loader
|
||||
workers. This was a scheduler retry path, rather than evidence of useful terrain
|
||||
generation. Inspecting the pinned DH 3.3.4 dispatcher confirmed that rejection
|
||||
requeues the same tracked task and immediately tries it again while retaining
|
||||
the dispatcher lock. A busy or unavailable physical pool can make that loop
|
||||
allocate and prevent other executors from progressing.
|
||||
|
||||
The optional `dispatch-retry` profile uses balanced graphics, mods, distances,
|
||||
worker count and heap guidance. It enables only the new dispatcher hook; adaptive
|
||||
budgeting, precise worker pacing and queue waiting stay disabled. Other profiles
|
||||
explicitly disable the new hook, so existing isolated comparisons remain useful.
|
||||
It requires exact DH 3.3.4. Set `enabled=false` in
|
||||
`config/horizon-dispatch-retry.properties` and restart to disable it.
|
||||
|
||||
After one rejection for an executor in a dispatch pass, the hook ends that
|
||||
executor's inner submission loop. Other executors can consume available logical
|
||||
slots in the same pass. The original catch still requeues the original task or
|
||||
clears it during shutdown. The original lock, priority ordering, occupied-slot
|
||||
increments/decrements, exception handling and scheduled-future cancellation
|
||||
remain in use. A rejected pass requests a later retry after 1 ms; a pass with
|
||||
only paused executors retains DH's original 1000 ms interval. Shutdown prevents
|
||||
new retry scheduling. OS timers and lock contention can delay the requested retry.
|
||||
|
||||
This bounds repeated rejection within each pass. It still constructs an exception
|
||||
for a rejected submission, and frequent callers can cause additional passes.
|
||||
It is not a complete replacement scheduler or proof of faster terrain completion.
|
||||
No simulation rule, server tick rate, render distance or voxel content is changed.
|
||||
|
||||
## Applied-hook integration test
|
||||
|
||||
Build with `python tools/build_mod.py`, then run `python tools/test_dispatch.py`.
|
||||
On Linux the latter uses Xvfb; `--native-display` uses an existing display.
|
||||
The tool builds a private fixture mod under `.local/mod-build`, runs two isolated
|
||||
Fabric clients with the dispatcher hook disabled and enabled, and exits before
|
||||
opening a world. It uses the local demo launcher and requires no account.
|
||||
Official game libraries/assets and publisher mods remain private downloads.
|
||||
The fixture never enters a release pack or production JAR.
|
||||
|
||||
The fixture creates an actual publisher dispatcher with two executors. It replaces
|
||||
one executor's physical pool with a one-worker, one-slot pool and deliberately
|
||||
fills it with blocking work outside the dispatcher. This reproduces rejection
|
||||
while logical slots remain free. Upstream retains the caller in the retry loop
|
||||
during a 200 ms observation window and stalls independent queued work. With the
|
||||
hook, the caller returns and the other executor progresses before releasing the
|
||||
blocked physical worker. Both runs then require each tracked task to run exactly
|
||||
once, empty logical queues, zero occupied slots and physical-worker shutdown.
|
||||
|
||||
The sanitized output reports the submission caller's wall time, CPU time and
|
||||
allocated bytes, plus total observed physical-pool rejections. Retry-scheduler
|
||||
CPU/allocation is excluded from the caller counters. The observation window and
|
||||
earlier patched return produce different caller durations; these measurements
|
||||
describe the rejected-submission path, not normal task throughput. Linux CI runs
|
||||
the applied hooks; Windows and ARM Mac CI compile the fixture and production mod.
|
||||
|
||||
In [Linux CI run 37158922187](https://github.com/ludovic111/minecraft-optimization-lab/actions/runs/37158922187),
|
||||
[the raw fixture result](../benchmarks/dispatch-retry-2026-10-04/linux-applied-hooks.json)
|
||||
reported 216,987,192 caller allocation bytes and 63,790 rejections upstream,
|
||||
versus 103,384 caller bytes and 6 rejections with the hook. Caller wall times were
|
||||
401.549 ms and 0.959 ms respectively. These are single, unwarmed failure-path
|
||||
observations, including initialization/JIT effects, rather than normal throughput
|
||||
or a fixed-duration whole-process comparison. Both modes completed two tasks.
|
||||
|
||||
An initial patched Linux game also loaded, recorded 120 seconds and shut down
|
||||
normally. Its subsequent JFR capture avoided the earlier allocation storm.
|
||||
The captures had different cache histories; they do not establish an FPS or
|
||||
whole-game memory improvement, and this is not an M2 graphics benchmark.
|
||||
@@ -73,7 +73,7 @@ continued updating, and worlds/databases/workers closed cleanly. The actual
|
||||
publisher queue microbenchmark passes identity/priority/count checks on Linux,
|
||||
Windows and Apple Silicon macOS; its raw results and timing overshoot are documented
|
||||
in [the queue experiment](debounce-queue.md). These are separate from game-FPS,
|
||||
block-update-latency or RAM-reduction claims. All eleven pack overrides retain
|
||||
block-update-latency or RAM-reduction claims. All twelve pack overrides retain
|
||||
simulation distance 12 and explicitly set the intended experimental flags.
|
||||
|
||||
The 0.6.0 resident-memory sampler ran in the actual adaptive-pacing demo world.
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
package dev.horizonlab.budget;
|
||||
|
||||
/** Per-picker state, used only while its existing dispatch lock is held. */
|
||||
public final class DispatchRetry {
|
||||
private boolean rejectedExecutor;
|
||||
private boolean rejectedPass;
|
||||
|
||||
public void beginPass() { rejectedExecutor = false; rejectedPass = false; }
|
||||
public void beginExecutor() { rejectedExecutor = false; }
|
||||
public void rejected() { rejectedExecutor = true; rejectedPass = true; }
|
||||
public int visibleSlots(int occupied) { return rejectedExecutor ? Integer.MAX_VALUE : occupied; }
|
||||
public boolean needsRetry(boolean original, boolean shutdown) { return !shutdown && (original || rejectedPass); }
|
||||
public long retryMilliseconds(long original) { return rejectedPass ? 1 : original; }
|
||||
}
|
||||
@@ -94,6 +94,7 @@ public final class HorizonClient implements ClientModInitializer {
|
||||
context.put("precise_worker_pacing", dev.horizonlab.mixin.HorizonMixinPlugin.workerPacingEnabled());
|
||||
context.put("debounce_queue_wait", dev.horizonlab.mixin.HorizonMixinPlugin.debounceWaitEnabled());
|
||||
context.put("wakeable_idle_wait", dev.horizonlab.mixin.HorizonMixinPlugin.debounceWaitEnabled());
|
||||
context.put("dispatch_retry_backoff", dev.horizonlab.mixin.HorizonMixinPlugin.dispatchRetryEnabled());
|
||||
context.put("adaptive_target_fps", targetFps);
|
||||
context.put("render_distance", client.options.renderDistance().get());
|
||||
context.put("simulation_distance", client.options.simulationDistance().get());
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
package dev.horizonlab.mixin;
|
||||
|
||||
import com.seibel.distanthorizons.core.util.threading.PriorityTaskPicker;
|
||||
import dev.horizonlab.budget.DispatchRetry;
|
||||
import java.util.concurrent.RejectedExecutionException;
|
||||
import java.util.concurrent.atomic.AtomicBoolean;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import org.spongepowered.asm.mixin.Final;
|
||||
import org.spongepowered.asm.mixin.Mixin;
|
||||
import org.spongepowered.asm.mixin.Pseudo;
|
||||
import org.spongepowered.asm.mixin.Shadow;
|
||||
import org.spongepowered.asm.mixin.Unique;
|
||||
import org.spongepowered.asm.mixin.injection.At;
|
||||
import org.spongepowered.asm.mixin.injection.Constant;
|
||||
import org.spongepowered.asm.mixin.injection.Inject;
|
||||
import org.spongepowered.asm.mixin.injection.ModifyConstant;
|
||||
import org.spongepowered.asm.mixin.injection.ModifyVariable;
|
||||
import org.spongepowered.asm.mixin.injection.Redirect;
|
||||
import org.spongepowered.asm.mixin.injection.callback.CallbackInfo;
|
||||
|
||||
@Pseudo
|
||||
@Mixin(targets = "com.seibel.distanthorizons.core.util.threading.PriorityTaskPicker", remap = false)
|
||||
public abstract class DhPriorityDispatchMixin {
|
||||
@Shadow @Final private AtomicInteger occupiedThreadsRef;
|
||||
@Shadow @Final private AtomicBoolean isShutDownRef;
|
||||
@Unique private final DispatchRetry horizonDispatch = new DispatchRetry();
|
||||
|
||||
// This site is after the original lock acquisition, including the failed-tryLock early return.
|
||||
@Inject(method = "startNextTask", at = @At(value = "INVOKE",
|
||||
target = "Lcom/seibel/distanthorizons/core/util/threading/PriorityTaskPicker;getExecutorIteratorSortedByShortestTotalRunTime()Ljava/util/Iterator;"),
|
||||
require = 1, allow = 1, remap = false)
|
||||
private void horizonBeginDispatchPass(boolean blocking, CallbackInfo callback) {
|
||||
horizonDispatch.beginPass();
|
||||
}
|
||||
|
||||
@Redirect(method = "startNextTask", at = @At(value = "INVOKE",
|
||||
target = "Lcom/seibel/distanthorizons/core/util/threading/PriorityTaskPicker$Executor;canRun()Z"),
|
||||
require = 1, allow = 1, remap = false)
|
||||
private boolean horizonBeginExecutor(PriorityTaskPicker.Executor executor) {
|
||||
horizonDispatch.beginExecutor();
|
||||
return executor.canRun();
|
||||
}
|
||||
|
||||
@Redirect(method = "startNextTask", at = @At(value = "INVOKE",
|
||||
target = "Lcom/seibel/distanthorizons/core/util/threading/PriorityTaskPicker$Executor;runTask(Ljava/lang/Runnable;)V"),
|
||||
require = 1, allow = 1, remap = false)
|
||||
private void horizonMarkRejected(PriorityTaskPicker.Executor executor, Runnable task) {
|
||||
try {
|
||||
executor.runTask(task);
|
||||
} catch (RejectedExecutionException rejected) {
|
||||
horizonDispatch.rejected();
|
||||
throw rejected; // Original catch requeues the same task or clears it during shutdown.
|
||||
}
|
||||
}
|
||||
|
||||
@Redirect(method = "startNextTask", at = @At(value = "INVOKE",
|
||||
target = "Ljava/util/concurrent/atomic/AtomicInteger;get()I"), require = 1, allow = 1, remap = false)
|
||||
private int horizonLeaveRejectedExecutor(AtomicInteger counter) {
|
||||
int occupied = counter.get();
|
||||
return counter == occupiedThreadsRef
|
||||
? horizonDispatch.visibleSlots(occupied) : occupied;
|
||||
}
|
||||
|
||||
// Inspected DH 3.3.4 local slot 2 is the existing "executor paused" retry flag.
|
||||
@ModifyVariable(method = "startNextTask", at = @At("LOAD"), index = 2,
|
||||
require = 1, allow = 1, remap = false)
|
||||
private boolean horizonRetryRejectedPass(boolean original) {
|
||||
return horizonDispatch.needsRetry(original, isShutDownRef.get());
|
||||
}
|
||||
|
||||
@ModifyConstant(method = "startNextTask", constant = @Constant(longValue = 1000L),
|
||||
require = 1, allow = 1, remap = false)
|
||||
private long horizonRetryDelay(long original) {
|
||||
return horizonDispatch.retryMilliseconds(original);
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,7 @@ import org.spongepowered.asm.mixin.extensibility.IMixinInfo;
|
||||
public final class HorizonMixinPlugin implements IMixinConfigPlugin {
|
||||
private static boolean workerPacing;
|
||||
private static boolean debounceWait;
|
||||
private static boolean dispatchRetry;
|
||||
|
||||
@Override public void onLoad(String mixinPackage) {
|
||||
var loader = FabricLoader.getInstance();
|
||||
@@ -47,13 +48,30 @@ public final class HorizonMixinPlugin implements IMixinConfigPlugin {
|
||||
} catch (Exception error) {
|
||||
LoggerFactory.getLogger("horizonlab").warn("Debounce queue wait disabled: invalid configuration", error);
|
||||
}
|
||||
try {
|
||||
Properties properties = new Properties();
|
||||
var dispatchFile = loader.getConfigDir().resolve("horizon-dispatch-retry.properties");
|
||||
if (Files.exists(dispatchFile)) {
|
||||
try (var reader = Files.newBufferedReader(dispatchFile)) { properties.load(reader); }
|
||||
}
|
||||
boolean requested = Boolean.parseBoolean(properties.getProperty("enabled", "false"));
|
||||
boolean supported = loader.getModContainer("distanthorizons")
|
||||
.map(mod -> mod.getMetadata().getVersion().getFriendlyString().equals("3.3.4")).orElse(false);
|
||||
dispatchRetry = requested && supported;
|
||||
if (requested) LoggerFactory.getLogger("horizonlab").info(
|
||||
"DH rejected dispatch retry: {} (requires pinned DH 3.3.4)", dispatchRetry ? "enabled" : "unavailable");
|
||||
} catch (Exception error) {
|
||||
LoggerFactory.getLogger("horizonlab").warn("Dispatch retry disabled: invalid configuration", error);
|
||||
}
|
||||
}
|
||||
|
||||
public static boolean workerPacingEnabled() { return workerPacing; }
|
||||
public static boolean debounceWaitEnabled() { return debounceWait; }
|
||||
public static boolean dispatchRetryEnabled() { return dispatchRetry; }
|
||||
@Override public boolean shouldApplyMixin(String targetClassName, String mixinClassName) {
|
||||
if (mixinClassName.endsWith("DhWorkerPacingMixin")) return workerPacing;
|
||||
if (mixinClassName.endsWith("DhDebounceWaitMixin")) return debounceWait;
|
||||
if (mixinClassName.endsWith("DhPriorityDispatchMixin")) return dispatchRetry;
|
||||
return true;
|
||||
}
|
||||
@Override public String getRefMapperConfig() { return null; }
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"id": "horizonlab",
|
||||
"version": "0.9.0",
|
||||
"version": "0.10.0",
|
||||
"name": "Horizon Lab",
|
||||
"description": "Client frame pacing and server tick measurements for reproducible optimization experiments.",
|
||||
"authors": [
|
||||
|
||||
@@ -4,6 +4,6 @@
|
||||
"plugin": "dev.horizonlab.mixin.HorizonMixinPlugin",
|
||||
"compatibilityLevel": "JAVA_25",
|
||||
"mixins": ["ServerTickMixin"],
|
||||
"client": ["FrameMixin", "DhWorkerPacingMixin", "DhDebounceWaitMixin"],
|
||||
"client": ["FrameMixin", "DhWorkerPacingMixin", "DhDebounceWaitMixin", "DhPriorityDispatchMixin"],
|
||||
"injectors": {"defaultRequire": 1}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
package dev.horizonlab.budget;
|
||||
|
||||
public final class DispatchRetryTest {
|
||||
private static void check(boolean condition) { if (!condition) throw new AssertionError(); }
|
||||
public static void main(String[] args) {
|
||||
DispatchRetry policy = new DispatchRetry();
|
||||
policy.beginPass();
|
||||
policy.beginExecutor();
|
||||
check(policy.visibleSlots(0) == 0 && policy.visibleSlots(3) == 3);
|
||||
check(!policy.needsRetry(false, false) && policy.needsRetry(true, false));
|
||||
check(policy.retryMilliseconds(1000) == 1000);
|
||||
policy.rejected();
|
||||
check(policy.visibleSlots(0) == Integer.MAX_VALUE && policy.needsRetry(false, false));
|
||||
check(policy.retryMilliseconds(1000) == 1);
|
||||
policy.beginExecutor();
|
||||
check(policy.visibleSlots(1) == 1); // Another executor can consume a real free slot.
|
||||
check(policy.needsRetry(false, false)); // Still retry the earlier rejected executor.
|
||||
check(!policy.needsRetry(false, true) && !policy.needsRetry(true, true));
|
||||
policy.beginPass();
|
||||
check(!policy.needsRetry(false, false) && policy.retryMilliseconds(1000) == 1000);
|
||||
System.out.println("Dispatch retry policy tests passed.");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
package dev.horizonlab.dispatch;
|
||||
|
||||
import com.seibel.distanthorizons.core.util.threading.PriorityTaskPicker;
|
||||
import com.seibel.distanthorizons.core.util.threading.RateLimitedThreadPoolExecutor;
|
||||
import dev.horizonlab.mixin.HorizonMixinPlugin;
|
||||
import java.lang.reflect.Field;
|
||||
import java.lang.management.ManagementFactory;
|
||||
import java.util.concurrent.ArrayBlockingQueue;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.ThreadPoolExecutor;
|
||||
import java.util.concurrent.atomic.AtomicInteger;
|
||||
import java.util.concurrent.atomic.AtomicLong;
|
||||
import java.util.concurrent.atomic.AtomicReference;
|
||||
import net.fabricmc.api.ClientModInitializer;
|
||||
import net.fabricmc.fabric.api.client.event.lifecycle.v1.ClientLifecycleEvents;
|
||||
|
||||
/** Private fixture mod, excluded from the production JAR. Run inside Fabric to test applied hooks. */
|
||||
public final class DispatchIntegration implements ClientModInitializer {
|
||||
@Override public void onInitializeClient() {
|
||||
ClientLifecycleEvents.CLIENT_STARTED.register(client -> {
|
||||
try { exercise(HorizonMixinPlugin.dispatchRetryEnabled()); }
|
||||
catch (Exception failure) { throw new AssertionError("Dispatch integration failed", failure); }
|
||||
if (Boolean.getBoolean("horizon.fixture.exit")) System.exit(0);
|
||||
});
|
||||
}
|
||||
|
||||
private static void exercise(boolean enabled) throws Exception {
|
||||
PriorityTaskPicker picker = new PriorityTaskPicker();
|
||||
var blocked = picker.createExecutor("Horizon fixture blocked", 1);
|
||||
var independent = picker.createExecutor("Horizon fixture independent", 2);
|
||||
Field poolField = blocked.getClass().getDeclaredField("threadPoolExecutor");
|
||||
poolField.setAccessible(true);
|
||||
((RateLimitedThreadPoolExecutor) poolField.get(blocked)).shutdownNow();
|
||||
var physical = new RateLimitedThreadPoolExecutor(1, runnable -> {
|
||||
Thread thread = new Thread(runnable, "Horizon fixture physical");
|
||||
thread.setDaemon(true);
|
||||
return thread;
|
||||
}, new ArrayBlockingQueue<Runnable>(1));
|
||||
AtomicInteger rejections = new AtomicInteger();
|
||||
var abort = new ThreadPoolExecutor.AbortPolicy();
|
||||
physical.setRejectedExecutionHandler((task, pool) -> {
|
||||
rejections.incrementAndGet();
|
||||
abort.rejectedExecution(task, pool);
|
||||
});
|
||||
poolField.set(blocked, physical);
|
||||
CountDownLatch entered = new CountDownLatch(1), release = new CountDownLatch(1);
|
||||
CountDownLatch submitted = new CountDownLatch(1), completed = new CountDownLatch(2);
|
||||
AtomicInteger firstRuns = new AtomicInteger(), secondRuns = new AtomicInteger();
|
||||
AtomicLong callerBytes = new AtomicLong(), callerCpu = new AtomicLong(), callerWall = new AtomicLong();
|
||||
AtomicReference<Throwable> callerFailure = new AtomicReference<>();
|
||||
var threads = (com.sun.management.ThreadMXBean) ManagementFactory.getThreadMXBean();
|
||||
threads.setThreadAllocatedMemoryEnabled(true);
|
||||
threads.setThreadCpuTimeEnabled(true);
|
||||
try {
|
||||
// Physical work is deliberately outside the picker: reproduce a busy executor with free logical slots.
|
||||
physical.execute(() -> {
|
||||
entered.countDown();
|
||||
try { release.await(10, TimeUnit.SECONDS); }
|
||||
catch (InterruptedException interrupted) { Thread.currentThread().interrupt(); }
|
||||
});
|
||||
if (!entered.await(5, TimeUnit.SECONDS)) throw new AssertionError("Physical worker did not start");
|
||||
physical.execute(() -> {}); // Fill its only physical queue slot.
|
||||
Thread caller = new Thread(() -> {
|
||||
long id = Thread.currentThread().threadId();
|
||||
long bytes = threads.getThreadAllocatedBytes(id), cpu = threads.getCurrentThreadCpuTime(), wall = System.nanoTime();
|
||||
try { blocked.execute(() -> { firstRuns.incrementAndGet(); completed.countDown(); }); }
|
||||
catch (Throwable failure) { callerFailure.set(failure); }
|
||||
finally {
|
||||
callerWall.set(System.nanoTime() - wall);
|
||||
callerBytes.set(threads.getThreadAllocatedBytes(id) - bytes);
|
||||
callerCpu.set(threads.getCurrentThreadCpuTime() - cpu);
|
||||
submitted.countDown();
|
||||
}
|
||||
}, "Horizon fixture caller");
|
||||
caller.setDaemon(true);
|
||||
caller.start();
|
||||
boolean returned = submitted.await(enabled ? 2000 : 200, TimeUnit.MILLISECONDS);
|
||||
if (returned != enabled) throw new AssertionError("Unexpected rejection lock behavior: hook=" + enabled);
|
||||
independent.execute(() -> { secondRuns.incrementAndGet(); completed.countDown(); });
|
||||
long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
|
||||
if (enabled) {
|
||||
while (secondRuns.get() == 0 && System.nanoTime() < deadline) Thread.sleep(1);
|
||||
if (secondRuns.get() != 1 || firstRuns.get() != 0) throw new AssertionError("Blocked executor prevented independent work");
|
||||
} else if (secondRuns.get() != 0) throw new AssertionError("Upstream did not reproduce dispatcher blocking");
|
||||
release.countDown();
|
||||
if (!completed.await(5, TimeUnit.SECONDS)) throw new AssertionError("Requeued task was lost");
|
||||
caller.join(5000);
|
||||
if (caller.isAlive() || callerFailure.get() != null) throw new AssertionError("Submission failed", callerFailure.get());
|
||||
Field occupiedField = PriorityTaskPicker.class.getDeclaredField("occupiedThreadsRef");
|
||||
occupiedField.setAccessible(true);
|
||||
AtomicInteger occupied = (AtomicInteger) occupiedField.get(picker);
|
||||
deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
|
||||
while (occupied.get() != 0 && System.nanoTime() < deadline) Thread.sleep(1);
|
||||
if (occupied.get() != 0 || firstRuns.get() != 1 || secondRuns.get() != 1
|
||||
|| blocked.getQueueSize() != 0 || independent.getQueueSize() != 0)
|
||||
throw new AssertionError("Task count, queue, or occupied slots corrupted");
|
||||
System.out.println("HORIZON_DISPATCH_RESULT {\"enabled\":" + enabled
|
||||
+ ",\"submission_allocated_bytes\":" + callerBytes.get()
|
||||
+ ",\"submission_cpu_ns\":" + callerCpu.get() + ",\"submission_wall_ns\":" + callerWall.get()
|
||||
+ ",\"rejections\":" + rejections.get() + ",\"tasks_completed\":2,\"occupied_slots\":0}");
|
||||
} finally {
|
||||
release.countDown();
|
||||
picker.shutdownNow();
|
||||
if (!physical.awaitTermination(5, TimeUnit.SECONDS)) throw new AssertionError("Fixture worker did not shut down");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,7 +9,8 @@ YOUR MAC
|
||||
2. Sign into your Microsoft account in Prism yourself. No credentials are
|
||||
shared with this project. You need an owned Minecraft Java license.
|
||||
3. Enable Prism automatic Java downloads, using Java 25 ARM64 on Apple Silicon.
|
||||
Instance > Edit > Settings > Java: maximum memory 6144 MiB, minimum 512 MiB.
|
||||
Instance > Edit > Settings > Java: check the Memory override box, then set
|
||||
maximum memory 6144 MiB and minimum 512 MiB.
|
||||
If auto Java fails, install a native ARM64 Java 25 JDK and select it there.
|
||||
4. Launch. Start with a new test world or a COPY of a world from Minecraft 26.3.
|
||||
This release is 26.3; do not load an older-version world for a fair comparison.
|
||||
@@ -52,6 +53,11 @@ PROFILES
|
||||
For an isolated comparison use queue-wait versus m2-balanced. For the combined
|
||||
controls use adaptive-queue versus adaptive-pacing. Disable the queue hook in
|
||||
config/horizon-queue-wait.properties (enabled=false), then restart.
|
||||
- dispatch-retry: same balanced settings, with bounded retries when DH rejects
|
||||
a task submission. The original task is requeued; other executors can progress.
|
||||
This is a separate experiment: adaptive, worker-pacing and queue-wait are off.
|
||||
Set enabled=false in config/horizon-dispatch-retry.properties and restart
|
||||
to disable it. A synthetic applied-hook test checks exact task completion.
|
||||
- low-memory: 8 full-detail chunks, no LOD. Set maximum memory 3072 MiB.
|
||||
For a matched comparison set both instances to the same render distance.
|
||||
- far-distance: 12 full-detail plus 256 LOD chunks. Start with 8192 MiB ONLY on
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"minecraft": "26.3",
|
||||
"java_major": 25,
|
||||
"fabric_loader": "0.19.5",
|
||||
"pack_version": "0.9.0",
|
||||
"pack_version": "0.10.0",
|
||||
"options_data_version": 5023,
|
||||
"projects": {
|
||||
"sodium": {
|
||||
|
||||
@@ -50,6 +50,18 @@ class PackTests(unittest.TestCase):
|
||||
with self.assertRaisesRegex(ValueError, "Missing pinned dependency"):
|
||||
build_pack.validate_selection(build_pack.read_lock(), ["iris"])
|
||||
|
||||
def test_dispatch_experiment_does_not_enable_other_controllers(self):
|
||||
for name, profile in build_pack.PROFILES.items():
|
||||
files = build_pack.overrides(name, b"original fixture")
|
||||
dispatch = files["config/horizon-dispatch-retry.properties"].decode()
|
||||
self.assertEqual(dispatch, "enabled=true\n" if name == "dispatch-retry" else "enabled=false\n")
|
||||
if name == "dispatch-retry":
|
||||
for setting in ["worker-pacing", "queue-wait"]:
|
||||
self.assertEqual(files[f"config/horizon-{setting}.properties"], b"enabled=false\n")
|
||||
self.assertTrue(files["config/horizon-adaptive.properties"].startswith(b"enabled=false\n"))
|
||||
self.assertEqual(files["config/DistantHorizons.toml"],
|
||||
build_pack.overrides("m2-balanced", b"original fixture")["config/DistantHorizons.toml"])
|
||||
|
||||
def test_wrong_game_or_download_path_is_rejected(self):
|
||||
for field, value in [("game_versions", ["1.21"]), ("filename", "../bad.jar"), ("url", "http://unsafe.invalid")]:
|
||||
lock = copy.deepcopy(build_pack.read_lock())
|
||||
|
||||
@@ -100,6 +100,7 @@ function renderDetails(side){
|
||||
const r=entry.report,fields=[['Profile',r.profile||r.label||'Unlabeled'],['Recording',format(r.elapsed_seconds)+' s · '+format(r.stats.count)+' samples'],['Platform',[r.os,r.architecture].filter(Boolean).join(' ')||'Not captured']];
|
||||
if(r.kind==='frames')fields.push(['Graphics',[r.gpu,r.graphics_backend].filter(Boolean).join(' · ')||'Not captured'],['Window',number(r.framebuffer_width)?r.framebuffer_width+' × '+r.framebuffer_height:'Not captured'],['Experiments','Adaptive: '+(r.adaptive_budget_enabled?'on':'off')+' · precise pacing: '+(r.precise_worker_pacing?'on':'off')+' · queue wait: '+(r.debounce_queue_wait?'on':'off')]);
|
||||
fields.push(['JVM', 'Compact headers: '+(r.compact_object_headers===true?'on':r.compact_object_headers===false?'off':'not captured')+' · GC: '+(Array.isArray(r.garbage_collectors)?r.garbage_collectors.join(', '):'not captured')]);
|
||||
fields.push(['Rejected dispatch retry', r.dispatch_retry_backoff===true?'on':r.dispatch_retry_backoff===false?'off':'not captured']);
|
||||
fields.push(['Wakeable queue idle', r.wakeable_idle_wait===true?'on':r.wakeable_idle_wait===false?'off':'not captured']);
|
||||
for(const [label,value]of fields)target.append(text('dt',label),text('dd',value));
|
||||
}
|
||||
|
||||
@@ -47,6 +47,10 @@ def test_metrics():
|
||||
SOURCE / "test/java/dev/horizonlab/budget/WakeableIdleWaitTest.java"]
|
||||
subprocess.run(["javac", "--release", "25", "-d", str(destination), *map(str, idle_sources)], check=True)
|
||||
subprocess.run(["java", "-cp", str(destination), "dev.horizonlab.budget.WakeableIdleWaitTest"], check=True)
|
||||
dispatch_sources = [SOURCE / "main/java/dev/horizonlab/budget/DispatchRetry.java",
|
||||
SOURCE / "test/java/dev/horizonlab/budget/DispatchRetryTest.java"]
|
||||
subprocess.run(["javac", "--release", "25", "-d", str(destination), *map(str, dispatch_sources)], check=True)
|
||||
subprocess.run(["java", "-cp", str(destination), "dev.horizonlab.budget.DispatchRetryTest"], check=True)
|
||||
|
||||
|
||||
def build():
|
||||
@@ -109,6 +113,28 @@ def build():
|
||||
return output
|
||||
|
||||
|
||||
def build_dispatch_fixture():
|
||||
"""Create a private integration-test mod; never include it in release packs."""
|
||||
arguments = [json.loads(line) for line in (BUILD / "javac.args").read_text().splitlines()]
|
||||
classpath = arguments[arguments.index("-classpath") + 1] + os.pathsep + str(BUILD / "classes")
|
||||
classes = BUILD / "dispatch-fixture"
|
||||
if classes.exists():
|
||||
shutil.rmtree(classes)
|
||||
classes.mkdir(parents=True)
|
||||
subprocess.run(["javac", "--release", "25", "-proc:none", "-classpath", classpath,
|
||||
"-d", str(classes), str(SOURCE / "test/java/dev/horizonlab/dispatch/DispatchIntegration.java")], check=True)
|
||||
output = BUILD / "dispatch-fixture.jar"
|
||||
with zipfile.ZipFile(output, "w", compression=zipfile.ZIP_DEFLATED) as archive:
|
||||
for path in sorted(classes.rglob("*.class")):
|
||||
archive.write(path, path.relative_to(classes).as_posix())
|
||||
archive.writestr("fabric.mod.json", json.dumps({"schemaVersion": 1, "id": "horizon_dispatch_fixture",
|
||||
"version": "1.0.0", "environment": "client",
|
||||
"entrypoints": {"client": ["dev.horizonlab.dispatch.DispatchIntegration"]},
|
||||
"depends": {"horizonlab": "*", "distanthorizons": "3.3.4"}}))
|
||||
print(f"Private dispatch integration fixture: {output}")
|
||||
return output
|
||||
|
||||
|
||||
def benchmark_workers():
|
||||
arguments = [json.loads(line) for line in (BUILD / "javac.args").read_text().splitlines()]
|
||||
classpath = os.pathsep.join([str(BUILD / "classes"), arguments[arguments.index("-classpath") + 1]])
|
||||
|
||||
+4
-1
@@ -23,6 +23,7 @@ PROFILES = {
|
||||
"adaptive-pacing": {"title": "Adaptive precise pacing experiment", "mods": CORE + ["distanthorizons"], "render": 12, "lod": 128, "heap": 6144, "adaptive": True, "pacing": True},
|
||||
"queue-wait": {"title": "Debounce queue wait experiment", "mods": CORE + ["distanthorizons"], "render": 12, "lod": 128, "heap": 6144, "queue_wait": True},
|
||||
"adaptive-queue": {"title": "Adaptive pacing and queue wait experiment", "mods": CORE + ["distanthorizons"], "render": 12, "lod": 128, "heap": 6144, "adaptive": True, "pacing": True, "queue_wait": True},
|
||||
"dispatch-retry": {"title": "Rejected dispatch retry experiment", "mods": CORE + ["distanthorizons"], "render": 12, "lod": 128, "heap": 6144, "dispatch_retry": True},
|
||||
"low-memory": {"title": "Low memory", "mods": CORE, "render": 8, "lod": 0, "heap": 3072},
|
||||
"far-distance": {"title": "Far distance", "mods": CORE + ["distanthorizons"], "render": 12, "lod": 256, "heap": 8192},
|
||||
"cinematic": {"title": "Cinematic (optional)", "mods": CORE + ["distanthorizons", "iris", "complementary-reimagined"], "render": 12, "lod": 128, "heap": 6144},
|
||||
@@ -76,7 +77,7 @@ def overrides(profile_name, mod_bytes):
|
||||
files = {f"mods/horizon-lab-{read_lock()['pack_version']}.jar": mod_bytes,
|
||||
"config/horizon-profile.txt": (profile_name + "\n").encode(),
|
||||
"options.txt": options(profile).encode()}
|
||||
experiment = any(profile.get(key, False) for key in ["adaptive", "pacing", "queue_wait"])
|
||||
experiment = any(profile.get(key, False) for key in ["adaptive", "pacing", "queue_wait", "dispatch_retry"])
|
||||
config = ROOT / "packs/config" / f"{'m2-balanced' if experiment else profile_name}.toml"
|
||||
files["config/horizon-adaptive.properties"] = (
|
||||
f"enabled={str(profile.get('adaptive', False)).lower()}\ntargetFps=60\n").encode()
|
||||
@@ -84,6 +85,8 @@ def overrides(profile_name, mod_bytes):
|
||||
f"enabled={str(profile.get('pacing', False)).lower()}\n").encode()
|
||||
files["config/horizon-queue-wait.properties"] = (
|
||||
f"enabled={str(profile.get('queue_wait', False)).lower()}\n").encode()
|
||||
files["config/horizon-dispatch-retry.properties"] = (
|
||||
f"enabled={str(profile.get('dispatch_retry', False)).lower()}\n").encode()
|
||||
if profile["lod"]:
|
||||
if not config.exists():
|
||||
raise FileNotFoundError(f"Missing verified DH profile config: {config}")
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Exercise applied DH dispatch hooks in isolated Fabric clients; no world or account needed."""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
try:
|
||||
from . import build_mod, build_pack
|
||||
except ImportError:
|
||||
import build_mod
|
||||
import build_pack
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--native-display", action="store_true", help="Use the existing display instead of Linux Xvfb")
|
||||
args = parser.parse_args()
|
||||
root = build_pack.ROOT
|
||||
fixture = build_mod.build_dispatch_fixture()
|
||||
version = build_pack.read_lock()["pack_version"]
|
||||
mod = root / "dist" / f"horizon-lab-{version}.jar"
|
||||
results = []
|
||||
for enabled in [False, True]:
|
||||
instance = root / ".local/runtime" / ("dispatch-fixture-patched" if enabled else "dispatch-fixture-upstream")
|
||||
build_pack.prepare_instance("dispatch-retry", instance, mod)
|
||||
shutil.copy2(fixture, instance / "mods/dispatch-fixture.jar")
|
||||
(instance / "config/horizon-dispatch-retry.properties").write_text(f"enabled={str(enabled).lower()}\n")
|
||||
command = [sys.executable, str(root / "tools/runtime.py"), "demo", "--instance", str(instance),
|
||||
"--heap", "3G", "--compact-headers"]
|
||||
if platform.system() == "Linux" and not args.native_display:
|
||||
command = ["xvfb-run", "-a", *command]
|
||||
environment = os.environ.copy()
|
||||
environment["_JAVA_OPTIONS"] = "-Dhorizon.fixture.exit=true"
|
||||
run = subprocess.run(command, cwd=root, env=environment, text=True, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, timeout=240)
|
||||
(instance / "fixture.log").write_text(run.stdout)
|
||||
marker = "HORIZON_DISPATCH_RESULT "
|
||||
reports = [json.loads(line.split(marker, 1)[1]) for line in run.stdout.splitlines() if marker in line]
|
||||
if run.returncode or len(reports) != 1:
|
||||
raise RuntimeError(f"Dispatch fixture failed; inspect {instance / 'fixture.log'}")
|
||||
report = reports[0]
|
||||
if report["enabled"] != enabled or report["tasks_completed"] != 2 or report["occupied_slots"] != 0:
|
||||
raise AssertionError("Fixture outcome did not match the selected hook")
|
||||
results.append(report)
|
||||
print(json.dumps(report), flush=True)
|
||||
output = root / "dist/dispatch-integration.json"
|
||||
output.write_text(json.dumps({"schema": 1, "platform": platform.platform(), "architecture": platform.machine(),
|
||||
"minecraft": "26.3", "distant_horizons": "3.3.4", "results": results,
|
||||
"notes": ["Synthetic full physical worker queue; both modes use the actual publisher dispatcher inside Fabric.",
|
||||
"Allocation and CPU measure the submission caller only; retry scheduler activity is excluded.",
|
||||
"Upstream keeps the caller blocked for the requested 200 ms observation window; patched returns earlier.",
|
||||
"Both modes verify exact task counts, recovered occupied slots and fixture worker shutdown.",
|
||||
"Not a game FPS, terrain completion, whole-game RAM, or hardware compatibility benchmark."]}, indent=2) + "\n")
|
||||
print(f"Applied-hook integration checks passed: {output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user