Repository navigation
UI performance - #664
Closed
mrosseel wants to merge 503 commits into
Closed
UI performance#664mrosseel wants to merge 503 commits into
mrosseel wants to merge 503 commits into
Conversation
…ive aarch64 runners)
…ration URL, artifact fix)
…orrect SSID encoding
…acing bash wifi-fallback
… generations, capture evidence, announce rollback
…not version string
…anifest terminology (ADRs 0004-0006)
…ions protect themselves
…DR 0004), Type=notify readiness - release.yml builds images.pifinder-migration and cuts the tarball from it - SD image seeds /var/lib/pifinder/current-build.json with its own store path; version labels resolve via the update manifest; pifinder-build.json deleted - pifinder.service Type=notify: app sends READY=1 once the UI is constructed and drawing (dev-safe no-op without NOTIFY_SOCKET) — the watchdog health check now means 'the screen is live'
…ery-hold hints on failure screens (ADR 0005/0006)
…ration code lives on main, one implementation (brickbots#521)
…on only and lives on main (brickbots#521)
…V, catalog refactor)
…eroff latch on GPIO14 (ADR 0007)
…6 dev boxes python-libinput/rpi-gpio/rpi-hardware-pwm/python-prctl/dbus-python/picamera2/ spidev/pygobject/pyhotkey/pynput are aarch64-only: on dev machines they either can't build (evdev kernel headers, imp removal, cairo/dbus C deps) or are meaningless off-device (dev uses sys_utils_fake and the pygame keyboard). The device env (uv2nix, aarch64) is unchanged. Dev flow: uv sync -> pytest (CI marker set passes 735/735 under 3.13).
…oadcom wifi/BT blobs
…rop the baked-target existence condition (ADR 0003)
…y the update UI keys on
…ng shifted, gpsd pointed at a nonexistent device
The repeated block in UBX-NAV-SVINFO is chn, svid, flags, quality, cno, elev, azim — but the parser started reading at chn, so every field was shifted one byte: satellite IDs were channel numbers, the used flag was svid bit 0, and the C/N0 was the 0-7 quality indicator. Because "cno" was really the quality indicator (>=1 even for idle SBAS/QZSS channels still searching), the sats-seen count was inflated to near the full channel count, and the per-satellite used flags were random — which is why uSat was previously treated as stale and ignored. With the offsets fixed: - nSat only counts satellites with an actual signal (cno > 0) - uSat from SVINFO is valid, so report it as the used count instead of relying solely on NAV-SOL - elevation/azimuth are decoded as signed values per the spec Also update the used count from NAV-PVT numSV, which was parsed but never surfaced — on protVer >= 15 receivers gpsd enables NAV-PVT instead of NAV-SOL, so the used count stayed 0 forever despite a valid fix. And actually set got_sat_update when NAV-SAT arrives so the SVINFO fallback defers to it as intended. Verified against a live MAX-M8 (PROTVER 14) stream: SVINFO uSat now matches NAV-SOL numSV, and IDs/C-N0 match gpsd's own SKY decode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
Newer u-blox receivers (later M8 firmware, M9, M10) get NAV-PVT + NAV-SAT from gpsd instead of NAV-SOL + NAV-SVINFO, so this path is what recent PiFinder GPS units actually exercise. - take "used" from the dedicated svUsed flag (bit 3) instead of inferring it from the quality indicator - only count satellites with an actual signal: NAV-SAT lists every known satellite, so nSat was inflated by idle entries - decode elevation/azimuth as signed values Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
During cold start the receiver reports an estimated C/N0 for almanac-predicted satellites it is still trying to confirm, so a cno > 0 filter makes the seen count start around 20+ and sink to the real value as candidates fail to confirm. Gate on the quality indicator instead: only code-locked signals (quality >= 4) count as seen, in both NAV-SVINFO and NAV-SAT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
Gating the seen count on quality >= 4 (code locked) made the display flap to zero during marginal re-acquisition — walking out to sky view showed 0-3 satellites bouncing instead of the steady climb to a lock. Satellites at quality 2-3 (signal acquired) have a real signal and belong in the count; only quality-1 search candidates, whose C/N0 is an estimate, still don't. Same threshold for NAV-SVINFO and NAV-SAT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
"Installing patches" stayed at 0/41 while the package names changed:
nix sends a copyPath stop as {"action": "stop", "id": ...}, with no
type (checked with nix build --log-format internal-json). feed() passed
a stop on only when its type was 100, so no copy ever ended in the count.
Every stop now goes to _on_stop, which ignores the ids it does not track.
The tests use the real stop format.
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Software screen reads /run/pifinder/upgrade-status every frame. write_status truncated the file and then wrote it, so a read in between found an empty status and the screen drew one frame of its default. The screen flickered, most during Applying patches, which writes the status once per patch. write_status now writes a temporary file in the same directory and os.replace()s it. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… loop (#97) A py-spy profile on the CM4 while opening the Messier list: the UI thread was busy 91% of the time. About 7 s of 40 went to ImageDraw.text in the text menu, which rasterized every line again on every frame, and about 5 s to the GPS block of the main loop, which made a round trip to the shared state for each GPS message. - ui_utils.draw_text_cached keeps the rendered text of a (text, font) pair as a grey mask (LRU, 512) and pastes it with the colour. Pixel-identical to ImageDraw.text in RGBA mode (tests with the real fonts, 128 and 176 panels). The text menu uses it. - drain_gps_queue drains the GPS queue first and keeps one message per kind: the fix with the smallest error (the result the one-by-one loop gave), the newest time (force if any was time_force), the newest satellites and comms. Resets act first and drop older pending messages. Fix and time go to the shared state at once; satellites and comms at most once a second. Checked: 1747 unit tests, 6 new drain tests and 13 text tests; a headless run on nixtop reaches the event loop and applies the fake GPS fixes with falling error. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ezoneFinder build (#98) A py-spy profile on the CM4 (menus slow after a boot): 36% of the UI thread's time was waiting on the shared state, 21% on one call, set_location. set_location waited for the TimezoneFinder build (10.6 s on the CM4) and then ran timezone_at on every call: 180 ms per call for Belgium on the CM4. While the GPS converges, a fix with a smaller error comes about every second, and the UI thread waited for each lookup. - The zone is looked up again only after a move of more than TZ_MOVE_DEG (0.05 degrees, about 5 km); GPS jitter reuses it. - set_location never waits for the build. Until it is ready a location keeps the last known zone (or UTC), and the build thread fills in the zone of the current location when it ends. - wait_for_timezone_finder() for tests and tools. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rsion (#99) The rollback list named each generation by its store path, which only carries the NixOS version (26.05.20260924.c508844): the same for every build on one nixpkgs. The upgrade now keeps the build label by store path in /var/lib/pifinder/build-labels.json (at most 20, atomic write): the new build, and the build it leaves (from current-build.json), the likely rollback target. The rollback list uses the label when it has one, else the NixOS version as before. The file is for display only; a missing or broken file gives the old list. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The row count came from a fixed MAX_VISIBLE of 4 and the list stopped at y 128. On the 176 and 320 panels the list showed 4 or 5 versions and left the rest of the screen empty. The row height now comes from the base font. The list bottom comes from the screen height, less the status line while it shows. The draw computes the visible rows and key_down scrolls by that number. On the 128 panel the layout stays the same, pixel for pixel. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A profile on the CM4 with the Messier list open showed that most of the list's frame time went to FreeType glyph rendering (getmask2). The list drew each row name, each second column and the two header lines again on every frame. These draws now use draw_text_cached, as the text menus do. The result is the same pixel for pixel. On nixtop a frame of seven rows takes 0.03 ms instead of 1.06 ms. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ng (#101) The checking step ran nix-store --realise --dry-run. The dry run gives no progress, so the screen showed a moving block. The differ's /update-start reply now lists the target closure with NAR sizes. The device checks that list against its store in batches of 100 paths with nix-store --check-validity and writes "checking <done>/<total>" after each batch. The missing paths and the sum of their NAR sizes give the same estimate as the dry run. The screen shows "Checking paths", a % bar and the count. An older differ sends no closure. Then, and when a check fails, the device uses the dry run as before. delta-e2e compares the local check with the dry run when the test store holds the target closure from an earlier run. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…#103) PiFinder.state imported timezonefinder at module level, so every start paid for the import before the UI came up. The module now loads through lazy_module on first use, which is in the TimezoneFinder build thread. A test checks that importing PiFinder.state leaves it unloaded. This is the remaining part of brickbots#482. The pandas part and the restart unit name are already in nixos. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The update-manifest job used a bare actions/checkout. Under
pull_request_target that lands on the repository's DEFAULT branch, not the
PR's base, so the job runs whatever .github/scripts/ the default branch
happens to carry.
Today that works here by coincidence: both main and release carry the
scripts. Two ways it bites anyway.
It is silently the wrong tooling. A PR based on main has its manifest entry
written by release's copy of update_manifest.py. While the two agree that is
invisible; the moment the manifest schema moves on one branch it is a
mismatch nobody is looking for.
And it is fragile against a branch that carries the workflow but not the
scripts -- the workflow file has to be on a branch for pull_request_target to
fire there at all, so registering it somewhere new is exactly the thing that
sets this off. Downstream we hit precisely that: a successful 19-minute build,
then
bash: .github/scripts/publish_manifest.sh: No such file or directory
The image was built and pushed to Attic; no device could ever see it, because
the manifest was never written. A green build and an unreachable artifact is
about the worst shape a failure can take.
Check out the PR's base branch instead. It is the branch this workflow file
itself came from, so its scripts are always beside it, and it stays
maintainer-controlled. The PR head is deliberately not an option here: this
job holds contents:write and the Attic token.
The preview label did nothing testable did not. Both job conditions accepted either, and nothing downstream inspected which one was applied: update_manifest.py never receives a label and writes every PR build into channels["unstable"]. This repo has no preview label, so the condition was dead; the label existed only on the mrosseel fork, where it has now been deleted and testable's description corrected to say it publishes to the unstable channel. Both actions/checkout allow-unsafe-pr-checkout opt-ins are unchanged.
…105) When the multiprocessing Manager that owns SharedStateObj dies, worker proxy calls raise BrokenPipeError/ConnectionResetError/EOFError. The solver's outer restart loop turned this into an infinite ~2s spin that flooded the log (70k+ "Broken pipe" lines), and the integrator/camera loops handled it inconsistently. Add a shared helper in state_utils (DEAD_MANAGER_EXCEPTIONS, the SharedStateLost signal raised by sleep_for_framerate, and is_dead_manager_error) and apply it in the solver, integrator and camera worker loops so a dead manager is logged once and the loop exits cleanly instead of retrying forever. Behaviour is unchanged while the manager is healthy. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The luma color_device packs each pixel to 5-6-5 RGB in a Python loop. On a CM4 a full 176x176 frame takes 58 ms, more than the 33 ms frame budget, so the menu stutters when you scroll. The numpy version takes 1.6 ms on the same CM4 and sends the same bytes. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The FPS counter in the title bar was only available with the -f command-line flag. The NixOS service starts PiFinder with no arguments, so the counter could not be turned on for a device. Tools > Experimental > Dev Tools > Show FPS saves the show_fps config option and turns the counter on or off at once. At startup, the -f flag or the config option turns it on. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…#109) Opening the Messier list filtered every catalog, WDS included (131,303 objects). With an altitude criterion the filter took 2.4 s on a CM4, one object at a time in Python. - CatalogFilter.verdicts evaluates each active criterion for all objects at once with numpy. FastAltAz.radec_to_alt_array is the numpy form of the altitude calculation. WDS now takes 0.5 s on the CM4. - Catalogs.filter_catalogs takes the catalogs to filter. An object list filters only the catalogs it shows, so the Messier list filters 110 objects. The chart filters the selected catalogs before it builds its nearby index. - An object list rebuilds when the filter's dirty_time differs from the one it was built with, so it does not depend on which catalog was filtered last. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each screen kept its own frame counter. After a screen change, the new screen's first value covered only part of a second, and a screen you went back to counted from its old sample time. The title bar showed a low FPS for about one second after each change, also when the real rate did not drop. FrameRate is one counter on UIModule. The menu manager counts each frame it sends to the display, and the rate divides by the real window length. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
While catalogs loaded, the UI ran at 2 FPS on the CM4. The UI thread waits for the shared-state process many times per frame. After each wait it needs the GIL back, and the catalog loader thread kept it for up to the 5 ms switch interval, so each call cost up to 5 ms. - The UI process sets the GIL switch interval to 1 ms, after the worker processes start. - A frame makes about 7 shared-state calls in place of 14. The popup time, the FPS flag and the hardware capabilities are kept in the UI process. The title bar reads SQM, constellation and battery once a second. The power state is read once per frame. The UI state and the frame go to shared state only when they change, through UIModule.publish_screen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The UI thread made a round trip to the shared-state manager for each value it read, many times per frame. During each wait it released the GIL, and a busy thread in the UI process could then keep the GIL for up to the switch interval. The manager process now also writes the most-read values into one shared-memory block, one slot per value with its own lock (PiFinder/state_snapshot.py). The main loop reads a StateSnapshot once per frame into UIModule.snapshot, and the screens read from it. A read is a memory copy: no round trip, and the GIL is not released. The setters and their rules do not change. - StateSnapshot has the getter names of SharedStateObj, so read-only helpers take either one. CurrentSnapshot gives the catalog filter and Nearby the snapshot of the current frame. - A heartbeat slot makes a reader raise SharedStateLost when the manager stops, as the proxy does. - PowerManager keeps the power state it writes. - In a headless run, the UI main thread makes no manager read in a normal frame (was about 7). Design: ui-performance-plan.md, step 1. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The UI process kept one CompositeObject per catalog listing: 151,170 objects, about 276 MB, plus a 46 MB T9 cache. A loader thread built the WDS objects after start and took the GIL from the UI thread for about 17 s. Filters, search, All Filtered and the Nearby index walked every object in Python. - catalog_arrays.py builds one directory per DB catalog: numpy columns and UTF-8 text blobs, opened with mmap. A Nix derivation (nixos/pkgs/catalog-arrays.nix) builds them in CI from the DB and the builder files only. Without Nix, the app builds them into the cache directory once. - ArrayCatalog makes a CompositeObject only for a row that code reads, and keeps the last 5000. logged, the filter verdict and the observing-list descriptions live next to the columns. - ObjectSequence is a read-only list over catalog rows. Filter, sort, Nearby de-duplication, search and the list cursor work on columns. - Planets, comets and observing-list objects stay plain Catalogs. - Gone: the loader threads, priority and deferred catalogs, the pickle cache (and its files), the T9 cache, the Names class and the "Catalogs Fully Loaded" message. On a CM4: filter WDS 150-200 ms (was 2.37 s), all catalogs 170 ms, Nearby index 220 ms, open 0.8-0.9 s with no thread after start. Private RAM of the UI process in a headless PC run: 152 MB (was 542 MB). Every listing that ArrayCatalog makes is tested against the rules of the per-object builder, for the whole DB. Design: ui-performance-plan.md, step 2; ADR 0043. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Opening a catalog list called sort(), which always showed a "Sorting by" popup for 0.1 s. The popup went to the display at once on the empty new screen, and while it showed, update_screen skipped the frames of the 0.1 s menu slide. So the list appeared after a popup, with no slide. sort() shows the popup only with announce=True, which the marking-menu sort choices pass. Opening and refreshing a list show no popup, and the slide is visible. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#114) The early boot splash set the SSD1333 to contrast 0xFF and master current 0x0F: a drive product of 4080. That is 2.3 times the blooming cap in DisplaySSD1333 (1792), so bright pixels smeared to the left. The app splash then showed the same image clean, at set_brightness(125). The boot splash now sets contrast 142 and master 1, the registers of set_brightness(125), so the image also keeps its brightness when the app takes over. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#111) While catalogs loaded, the UI ran at 2 FPS on the CM4. The UI thread waits for the shared-state process many times per frame. After each wait it needs the GIL back, and the catalog loader thread kept it for up to the 5 ms switch interval, so each call cost up to 5 ms. - The UI process sets the GIL switch interval to 1 ms, after the worker processes start. - A frame makes about 7 shared-state calls in place of 14. The popup time, the FPS flag and the hardware capabilities are kept in the UI process. The title bar reads SQM, constellation and battery once a second. The power state is read once per frame. The UI state and the frame go to shared state only when they change, through UIModule.publish_screen. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collaborator
Author
|
Merged into the NixOS branch as mrosseel#113 (mrosseel#113), so the NixOS support build (#379) now contains it. This test build is no longer needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Test build of the UI performance work on the NixOS branch. Not for merge: the review and merge happen in mrosseel#113 (mrosseel#113).
Design and measurements:
ui-performance-plan.mdand the PR above.🤖 Generated with Claude Code