Repository navigation
rc-0.1.6 - #50
Open
IsuraManchanayake wants to merge 22 commits into
Open
rc-0.1.6#50IsuraManchanayake wants to merge 22 commits into
IsuraManchanayake wants to merge 22 commits into
Conversation
…gs, and a prompt's first token is no decode round The access log timed decode from when a request's Generated events arrived: tokens after the first event over the time from the first event to the last. A tick hands all of its tokens over at once, and the tick that ends a prompt also decodes, so the rate was missing whenever that tick produced the whole generation, and inflated when a channel change (Qwen3.6's reasoning tags) split one tick into several events: the rest of the first tick then counted against the next tick's time. Qwen3.6-35B-A3B, 128 tokens after a 4096-token prompt, max-batch 1, on strix-point-1 (gfx1150): reps logged "-", 116.7 and 37.6 tok/s at the same 7.6 s wall; the engine decoded at 37.8 throughout. The scheduler now adds up each lane's decode: per tick, the lane's share of the tick's decode time (the engine's own when it times the phases apart, the whole tick when it only decoded) and the tokens it decoded after its first. Done carries it to the request, and the access log uses it when present. A lane whose prefill ends in a tick takes its first token from the prefill's logits, at no decode cost (about 1 ms of sampling against ~26 ms per round on that model). The decode-cost estimate counted that token as a round, so a run of prefill-only requests taught it ~1 ms per round, and the next tick that ended a prompt was granted the 256-round maximum beside it: the whole generation landed in the prefill's tick and the client saw its first token only at the end. The estimate now counts only decode rounds.
…he engine's phase timers The previous commit took each tick's decode time from the engine's own decode_ns. llama.cpp's Vulkan backend runs the prefill asynchronously, so its prefill timer stops before the GPU is done and the rest lands in the decode phase: a prefill-only tick reported 0.45 s of decode for its one sampled token, and the tick that ends a prompt charged 35.8 ms per round against 22.1 ms in pure decode ticks (gemma-4-E2B Q4_K_M, strix-point-1). That put llama.cpp's per-request rate at 38-40 tok/s against 44 by the client's clock. The scheduler now stamps each lane once per tick, after the engine returns, with the tokens it has produced, and the rate is the tokens after the request's first tick over the time from that tick's end to the last one's (the event-based figure's definition, measured once per tick, so a tick split into several events by a channel change no longer counts its rest against the next tick). The first tick also ran the prefill, so it only opens the span. A preempted lane closes its span and the requeued one opens a new one. The decode-cost estimate keeps the previous commit's fix: a prompt's first token is no decode round.
llamacpp: tested with b11925
llamacpp: tested with b11951
… basert's by basert For a copy Homebrew, a global npm, pipx, uv or cargo installed, `self update` runs that manager's update instead of only naming it: the command is built from the manager's name (the installed-by file's text is only shown), the manager binary found from the copy's own path (the brew of its Cellar, the npm of its global folder with that bin first on PATH, the cargo beside it). It asks first on a terminal; --yes runs it without asking and is needed without one; --dry-run says what would run; --version is the manager's. A pinned formula, another user's Homebrew, an npm folder that needs root, a project's npm dependency and pip in an environment are named, not run, and nothing runs while a superfluid server runs from the copy. The superfluid basert installs for `basert serve` is basert's: its installed-by file says basert, or, from a basert before 0.4.3, basert's .release stamp marks it. Run by hand, its `self update` used to take it for the install script's copy: move it under ~/.superfluid/versions, link basert's copy to whatever became current there, and check releases against ~/.superfluid's engine rather than basert's. It now runs `basert superfluid install` (the basert beside it, else on PATH), which picks the newest release that serves basert's engine, and refuses --version.
…ven up A node's connection ran each generation inline, so for its whole length the link did no liveness work: no status, no heartbeat, no lease renewal, nothing read. Renewals went out on whichever lane was idle, where the head never answered them, and those lanes then fenced the session another lane was generating; the head waited on a finish that never came. A node that went silent without closing its socket hung the request for good. - Link F 3: each end sends a keepalive after 5 s with nothing sent and gives the link up after 30 s with nothing heard, kept by pump, so any loop that reads the link keeps it alive. Version 2 is still spoken, and a v2 peer is never given up for silence. - The node runs a generation on a thread of its own; its connection goes on reading cancels, grants and heartbeats and sending status. A session's lease is renewed only over the connection it was placed on, again after 1 s until granted. - The head answers status and renewals on every read, idle connections included, tends each idle one every heartbeat, and stops waiting 30 s past a generation's wall budget. - The node's --idle-timeout-ms defaults to 30 s; `node join` logs the node-agent's warnings to stderr.
…e link, frames carry the version chosen - The head's generation waits read the link one pump at a time and check the deadline between reads; wait_frames looped until a frame came, so a v2 node, never given up for silence, that went quiet held the request past the deadline. A reconnect's reconcile waits are bounded by the silence limit too. The deadline's grace is the head's silence limit, so the tests can shorten it. - The node runs a stage forward on a thread of its own, as it runs a generation, so a stage longer than the silence limit no longer silences the connection. - Endpoint::send frames at the negotiated version, and at the oldest one before the handshake settles it; a v2 peer that checks the envelope reads every frame. - The mock engine's stage forward takes a tick_delay, as a step does.
Fleet: a link stays up while a node generates, and a silent one is given up
`superfluid runtimes` lists an install other than the default as `basert@0.3.0-metal`, and `superfluid runtime remove basert@0.3.0-metal` answered that it was not a runtime id. remove, repair and use now read it as `basert 0.3.0-metal`, and refuse two different installs named both ways (or `use <id>@<install> --auto`). install, update, versions and options say they take the id alone, and install names the --version and --backend that ask for the build named. installs::remove takes the install an id names when no name is given, so `<id>@<install>` can never reach the branch that removes every install of the runtime.
`superfluid runtimes` lists the default install under the bare id, so copying that row into `superfluid runtime remove` removed every install of the runtime. With no install named, remove now takes the runtime's only install, and when it has more it names them, removes nothing and exits 2; `--all` removes every one. `<id>@<install> --all` is refused.
…t, WORKER is short `superfluid runtimes` from 0.1.5 had three columns that misled on a 48 GiB M5 Pro. - INSTALLABLE and `superfluid runtime versions` listed only the recipe's pins, while `runtime install basert` with no version installs the newest release that states this build's engine ABI (0.4.2), without --untested. An adapter now says, with `Runtime::unlisted` (`unlisted` in the worker's `versions` JSON), which versions it does not list install without --untested; basert's rule is ABI-marked releases. With a rule, an install made without --untested of a version newer than every listed one that installs here is installable: `0.4.2 (installed)` in the table, newest first in the listing, in `installable` in --json, and the footer states the rule. No network call. - DEVICE showed basert's budget (85% of Metal's working set), MLX's whole working set and llama.cpp's device total. A Device now carries `budget` beside `memory`: what the runtime plans a model and its cache against, PLANNING_SHARE of the GPU for MLX and llama.cpp (0 on llama.cpp's CPU), baseRT's own for basert. The column shows it, as `DEVICE (MEMORY BUDGET)`; --json keeps `memory`. A worker installed before this states no budget; the daemon derives it from what each shipped adapter's `memory` meant. - WORKER printed whole paths (npm's node_modules, Homebrew's Cellar). An install is now named in the runtimes directory (`installed: mlx/0.32.4-metal`), superfluid's own workers by source alone with the release directory once in the footer, and ~ stands for $HOME there and in the problem lines. llama.cpp's not-found names the install command and SUPERFLUID_LLAMA_LIB instead of three search paths. --json keeps the path.
…ers after the question - basert's `superfluid install` refuses while anything runs from the copy it replaces, and `self update` waiting on it was such a process. It now replaces itself with basert (`Report::hand_off`, exec'd by `superfluid self update`), so basert sees no updater. - npm is told the prefix of the global folder that holds the copy (`--prefix`): npm's own configuration or NPM_CONFIG_PREFIX could otherwise send the update to another one. - The running-server check is made again once the question is answered, as it may have waited long enough for a server to start.
…ad as an option `remove basert@--all` became `remove basert --all` and removed every install, past the guard that a bare id with more than one install needs --all; `use basert@--auto` unpinned. No install's name begins with '-', so such a selector is refused, naming the option it looked like where there is one.
… installs it here, and budgets fall back per device - An install's `tested` says only that the adapter that made it took it, which may have been a build for another engine ABI: INSTALLABLE could promote a version the current adapter refuses. An adapter's rule for unlisted versions (`Unlisted`) now says what it asks of one (basert: `engine ABI <n>`), an install of an unlisted version keeps that in its manifest as `requires`, and `runtime versions` counts it only when this build's rule asks the same. An install made before manifests kept it is not counted, and none is when no listed version installs here, as whether this host could install one cannot then be told. - A report that states a budget for some devices and not others had every device's budget inferred, explicit ones included: each device without one is now inferred on its own.
…e rate, whatever its events say A generation that ends in its first tick is measured as (0, 0.0), which `add_measured` dropped, so `measured()` fell back to the events: a tick committed as two events 9 ms apart logged 111.1 tok/s. Whether the scheduler timed the generation is now kept apart from whether it counted tokens; only a generation it did not time falls back to the events.
…counting scheduler: time each request's decode per tick, not per delivered event
self update: a package manager's copy is updated by that manager, and basert's by basert
runtime: remove, repair and use take an install as <id>@<install>
runtimes: installable agrees with install, device is the memory budget, worker is short
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release candidate for 0.1.6, cut from
mainat1f771e0: v0.1.5 (ce46155) plus #40 and #41.