Repository navigation
test(ui): stop the comet retry loop spinning through the UI tests (CI ui_tests hang) - #684
Merged
Merged
Conversation
ui_tests has hung intermittently on CI since 2026-10-24: four of the last six nox runs (main at 0e0b2b1 and c355928, #681, #683) stalled in tests/test_ui_modules.py until the job's 20-minute limit, at different points in the run, while every other session passed. Locally the suite passes in about 80 s, warm or cold cache; CI's passing runs took 150-400 s. The session-scoped catalogs fixture yielded while CatalogBuilder's background CatalogLoader thread was still loading the deferred objects (CI never has a catalog cache). The autouse _fast_sleep fixture no-ops time.sleep, which removes the loader's per-batch yield, so it ran flat out beside every test; and on completion its callback added objects to the live catalogs, re-filtered them and pickled them to the cache, from that thread, while tests were reading the same catalogs. The fixture now joins the loader before yielding, so the tests run on fully loaded catalogs with no thread mutating them. ui_tests also sets pytest's faulthandler_timeout, so a test still running after 180 s dumps every thread's stack to the log instead of the job dying silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… tests The CI thread dump from the previous commit's faulthandler_timeout found the hang. A failed comet download starts CometCatalog's background retry, which loops "sleep 60 s, download" until the comet file exists. In the UI harness the download is stubbed to fail and _fast_sleep no-ops time.sleep, so that loop never slept and never ended: it spun for the whole session, calling and recording the mock and logging on every pass, until CI's tests crawled past the job's 20-minute limit. A machine that already has a comet file in ~/PiFinder_data never starts the loop, which is why it didn't reproduce locally. Run with an empty home, a four-test slice logged 5,012 retries; with the retry stubbed out, none. _no_comet_download now stubs _start_background_retry too: with no network there is nothing to retry. The loader join from the previous commit stays, reworded: it isn't the cause, but it keeps the loader from competing with the first tests and every test sees fully loaded catalogs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Zhuohang-Shen
pushed a commit
to Zhuohang-Shen/PiFinder
that referenced
this pull request
Nov 15, 2026
Brings the merge of brickbots main into deepchart into observable-asteroids. Conflicts and their resolution: - ui/object_list.py: the imports of both sides (re for the names view of upstream, time for the asteroid branch). - tests/test_ui_modules.py: the comet fixture keeps the check_if_comet_download_needed stub of the asteroid branch and adds the retry stub of upstream (brickbots#684). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ui_testshas hung intermittently on CI since 2026-10-24, onmainas well as on PRs. A CI thread dump found the cause: the UI test harness's owntime.sleepstub turns the comet catalog's retry loop into a busy loop. Test-only fix; no runtime code changes.Evidence
ui_testsalone took 150–400 s; locally ~80 sui_teststhen stalled at a different point each time.Cause
This PR's first commit made a hanging test dump every thread's stack (
faulthandler_timeout), and the next CI run hung and dumped them:UILog.updatedraw.comet_catalog.pyretry_task→_download_once→ the mock's_increment_mock_call.CometCatalog._start_background_retryloops "time.sleep(60), download" until the comet file exists. The UI harness:comet_data_downloadto fail, so the file never appears;time.sleepfor the whole session (_fast_sleep).So the loop never slept and never ended. It spun for the whole run, calling the mock (which records every call) and logging on every pass, and starved the tests until CI ran past the 20-minute limit.
A developer machine that already has a comet file in
~/PiFinder_datanever starts the loop. That's why it never reproduced locally. With an emptyHOME, a four-test slice logged 5,012 retries before the fix, and 0 after.Both pieces date from #438 (May), so the loop has spun in every CI run since then. It only recently crossed the time limit, as the suite grew.
The change
_no_comet_downloadalso stubsCometCatalog._start_background_retry. With no network, there's nothing to retry. This is the fix.catalogsfixture waits for the backgroundCatalogLoaderbefore yielding (first commit). It isn't the cause, but on CI (no catalog cache) the loader otherwise runs flat out beside the first tests, again without its per-batch yield, and rewrites the live catalogs when it finishes. Now every test sees fully loaded catalogs. Locally this adds about 6 s, once.nox -s ui_testssets-o faulthandler_timeout=180, so any future hang dumps every thread's stack to the log instead of the job dying silently. This is what found this one.Testing
HOME(as on CI: no comet file, no catalog cache): 223 passed, 2 skipped, in 64 s.ruff checkandruff format --checkare clean. The parenthesizedwithparses on Python 3.9.🤖 Generated with Claude Code