From 4dc5645ec633ca4d656ed5165277a5b5e5053eef Mon Sep 17 00:00:00 2001 From: Michael Williams Date: Thu, 29 Oct 2026 03:57:48 -0700 Subject: [PATCH] Validate managed lifecycles without retaining benchmark results --- bench/lifecycle.exs | 36 ++++++++++++++++++-------- docs/integration.md | 61 +++++++++++++++++++++++++++++---------------- 2 files changed, 65 insertions(+), 32 deletions(-) diff --git a/bench/lifecycle.exs b/bench/lifecycle.exs index 80479ca..401163a 100644 --- a/bench/lifecycle.exs +++ b/bench/lifecycle.exs @@ -1,7 +1,9 @@ -# MIX_ENV=test ORTEX_BUILD=true mix run bench/lifecycle.exs [runs] [load_dispose_cycles] +# MIX_ENV=test ORTEX_BUILD=true mix run bench/lifecycle.exs [runs] [load_dispose_cycles] [bounded_runs] # RSS includes BEAM/allocator retention. This is a diagnostic, not a leak proof. runs = System.argv() |> Enum.at(0, "100000") |> String.to_integer() cycles = System.argv() |> Enum.at(1, "500") |> String.to_integer() +bounded = System.argv() |> Enum.at(2, "0") |> String.to_integer() +if bounded < 0, do: raise(ArgumentError, "bounded run count must be non-negative") if runs < 10 or cycles < 10, do: raise(ArgumentError, "counts must be at least ten") path = Path.expand("../test/fixtures/xgboost.onnx", __DIR__) bytes = File.read!(path) @@ -9,8 +11,8 @@ bytes = File.read!(path) input = Nx.tensor(rows, type: :f32) |> Nx.backend_transfer(Ortex.Backend) options = [intra_threads: 1, intra_op_spinning: false] -score = fn model -> - {output} = Ortex.run(model, input) +score = fn model, run_options -> + {output} = Ortex.run(model, input, run_options) actual = output |> Nx.backend_transfer(Nx.BinaryBackend) |> Nx.to_flat_list() unless length(actual) == length(expected) and @@ -27,37 +29,51 @@ sample = fn phase, count -> _ -> nil end - IO.inspect(%{phase: phase, count: count, rss_bytes: rss, beam_bytes: :erlang.memory(:total)}) + IO.inspect(%{ + phase: phase, + count: count, + rss_bytes: rss, + beam_bytes: :erlang.memory(:total), + processes: :erlang.system_info(:process_count) + }) end model = Ortex.load(path, options) -for _ <- 1..1000, do: score.(model) +Enum.each(1..1000, fn _ -> score.(model, []) end) sample.(:warmup, 1000) -for i <- 1..runs do - score.(model) +Enum.each(1..runs, fn i -> + score.(model, []) if rem(i, max(div(runs, 10), 1)) == 0, do: sample.(:repeated_run, i) -end +end) # Each load is owned by a fresh short-lived process so resources become eligible # for destruction at a known boundary. Alternate both copying constructors. -for i <- 1..cycles do +Enum.each(1..cycles, fn i -> Task.async(fn -> loaded = if rem(i, 2) == 0, do: Ortex.load(path, options), else: Ortex.load_from_memory(bytes, options) - score.(loaded) + score.(loaded, []) end) |> Task.await(60_000) if rem(i, max(div(cycles, 10), 1)) == 0, do: sample.(:load_dispose, i) +end) + +if bounded > 0 do + Enum.each(1..bounded, fn i -> + score.(model, timeout: 5000) + if rem(i, max(div(bounded, 10), 1)) == 0, do: sample.(:bounded_run, i) + end) end IO.inspect(%{ runs: runs, load_dispose_cycles: cycles, + bounded_runs: bounded, constructors: [:file, :memory], note: "Golden scores checked on every call; RSS is diagnostic, not a leak assertion." }) diff --git a/docs/integration.md b/docs/integration.md index e33e786..ba0d6ac 100644 --- a/docs/integration.md +++ b/docs/integration.md @@ -37,10 +37,13 @@ writes to another repository. The broad original PR is superseded by the stack. | 27 | One-attempt native cancellation controls | | 28 | Managed deadlines and caller-exit cancellation | | 29 | Dirty-scheduler measurement and policy | +| 30 | Final lifecycle/managed-run soak and review ledger | -Each PR targets the preceding branch. Merge or retarget in order; the integration -branch is for validation rather than a replacement broad PR. CI on each branch -checks that its intermediate state works independently. +Each PR targets the preceding branch. Merge in order with merge commits to +preserve ancestry. If using squash/rebase merges, rebase and retarget the remaining +stack and rerun its checks before proceeding. The integration branch combines +validation rather than replacing the small PRs. CI checks each intermediate state. +Task commits are unsigned because the local signing agent cancelled requests. ## Integration gates @@ -87,24 +90,36 @@ redundant native clone changed median latency from 121 to 85 microseconds for a reused resource and 207 to 166 microseconds from BinaryBackend. These are local diagnostics, not cross-platform latency guarantees or a raw-binary zero-copy claim. -`bench/lifecycle.exs` validates golden scores on each repeated run and each model -loaded by a short-lived owner process. A local Apple silicon run exercised 100,000 -runs plus 1,000 alternating file/memory load/disposal cycles per runtime: +`bench/lifecycle.exs` checks golden scores through ordinary runs, alternating +file/memory load-disposal owners, and optional bounded runs. The final Apple +silicon run used 100,000 ordinary runs, 1,000 load/disposal cycles, and 100,000 +managed bounded calls on each runtime: -| Runtime | Warmup RSS | Maximum run RSS | Final load/disposal RSS | -| --- | --- | --- | --- | -| 1.17.3 | 104,005,632 | 104,759,296 | 103,596,032 | -| 1.24.4 | 109,608,960 | 110,346,240 | 109,297,664 | +```sh +ORTEX_BUILD=true MIX_ENV=test mix run bench/lifecycle.exs 100000 1000 100000 +``` + +| Runtime | Warmup RSS | Maximum ordinary-run RSS | Final disposal RSS | Bounded RSS min/max | Final bounded RSS | +| --- | --- | --- | --- | --- | --- | +| 1.17.3 | 105,185,280 | 105,742,336 | 104,677,376 | 102,825,984 / 104,153,088 | 103,038,976 | +| 1.24.4 | 107,577,344 | 108,003,328 | 107,003,904 | 105,299,968 / 106,496,000 | 105,299,968 | -These byte counts include BEAM and allocator retention. The bounded samples do -not show sustained growth for this synthetic CPU workload; they do not prove the -absence of leaks in other graphs, providers, or runtime versions. Run this script -against any production-representative fixtures before changing a consumer. +Sampled BEAM process counts stayed at 124 throughout. The harness uses +`Enum.each`, discarding per-call results; collecting comprehensions in an earlier +diagnostic retained an accumulator and distorted heap/native-memory observations. +Temporary lifetime counters independently showed controls being reclaimed and +were removed before the final run. No production native code changed for this +harness correction. + +Byte counts include BEAM and allocator retention. These samples show no sustained +growth for the exercised synthetic CPU workload, including managed worker/control +ownership; they do not prove leak freedom for other graphs, providers, or runtime +versions. Repeat with production-representative fixtures before consumer rollout. The separate global-pool VM exercises **both** file and memory constructors: inherited pools, rejected ignored settings, explicit independent pools, unchanged scores, and resource survival through application restart. The precompiled check -runs these same gates with cargo/rustc replaced by failing stubs. +runs these same gates with cargo/rustc/protoc replaced by failing stubs. Separate runtime-log VMs configure info and warning levels before the one-time environment initialization. They assert per-session log IDs from both constructors, @@ -122,15 +137,17 @@ application restarts. Legacy positional loading keeps its CPU defaults. CI covers API 17/runtime 1.17.3, API 17/runtime 1.24.4, API 24/runtime 1.24.4, both Alpine architectures, all seven NIF targets, and complete checksum/package assembly. PR checks and workflow artifacts are the current evidence; a test count -alone does not prove those platform or constructor combinations passed. - -## Remaining work +alone does not prove those platform or constructor combinations passed. The final +validation PR records exact integration run links without requiring a release tag. +Runtime cancellation is cooperative and may finish after the caller deadline. +Accelerator runtime execution remains unvalidated without its libraries/hardware. -The initial runtime/build foundation is not the whole capability backlog. Continue -with separate focused PRs for: +## Deferred scope -- Per-run profiling toggles require API 25 and remain deferred beyond the API - 17/24 runtime matrix; session profiling is available now. +The stack covers the selected CPU/API 17–24 modernization, session/environment +controls, I/O values, metadata/graph inspection, observability, cancellation, and +Nx support decisions. Per-run profiling toggles require API 25 and remain outside +the tested runtime matrix; session profiling is available now. GPU/NN primitives remain a later or measurement-dependent stage: shared weights and pooling, AOT compilation, I/O binding, device allocation, and adapters. Keep