Skip to content

test: Fix metrics-related timeouts in instrumentation tests and stabilize 16 KB emulator job - #228

Merged
BartoszLitwiniuk merged 5 commits into
masterfrom
bugfix/sauce-instrumentation-tests
Oct 21, 2026
Merged

BartoszLitwiniuk merged 5 commits into
masterfrom
bugfix/sauce-instrumentation-tests

Conversation

@BartoszLitwiniuk

@BartoszLitwiniuk BartoszLitwiniuk commented Oct 3, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes the metrics-related timeouts in the instrumentation tests. The library metrics tests and ExampleInstrumentedTest.handledException failed for the same reason.

Root cause

Metrics and report uploads share a single sender thread. When metrics.enable() runs, it sends startup events to the real events endpoint right away. On Sauce devices and CI those calls could stall. Any report submitted afterwards was stuck in the queue behind them, so tests waiting on a report upload timed out.

Example app instrumentation tests (example-app)

  • New MainActivity.EXTRA_DISABLE_METRICS intent extra. ExampleInstrumentedTest sets it, so metrics stay off and can't hold up the reports under test. None of these tests assert on metrics.
  • handledException hardening:
    • waits up to 60 s for the submission result (was 5 s) and asserts its status is Ok, so a failed upload shows its real reason instead of an empty rxId
    • polls Coroner for up to 60 s until the report can be queried, instead of one query 2 s after upload. Server-side ingestion is asynchronous.

Library tests (backtrace-library)

  • BacktraceClientMetricsTest, BacktraceClientSummedEventTest, BacktraceClientUniqueEventTest: route startup metrics events to mock handlers in setUp(), so no test reaches the real events endpoint.

CI: 16 KB qualification job

  • google_apis_ps16k image bumped from API 35 to API 36. The API 35 image crashed system_server mid-run (NullPointerException in ReferenceQueueDaemon, a runtime bug unrelated to our code).
  • Emulator cores raised from 2 to 4, matching the standard GitHub runner.

Ref: BT-7865

@BartoszLitwiniuk
BartoszLitwiniuk marked this pull request as draft October 3, 2026 12:20
@BartoszLitwiniuk
BartoszLitwiniuk marked this pull request as ready for review October 13, 2026 19:31
@BartoszLitwiniuk BartoszLitwiniuk changed the title Fix unit-test ci: Fix flaky Sauce instrumentation tests and stabilize 16 KB emulator job Oct 13, 2026
@BartoszLitwiniuk
BartoszLitwiniuk requested review from KishanPRao and melekr and removed request for KishanPRao October 13, 2026 20:01

@KishanPRao KishanPRao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mocking metrics / turning them off fixes the metrics timeouts and handledException.
The description needs updating, since the issue wasn't the upload taking longer.
Reports were stuck behind stalled metrics calls on the same queue.

The PR only fixes part of the Sauce flakiness. The native crash report tests and dumpWithoutCrash are unchanged and will likely still fail. On the latest run, Android 14 passed only on attempt 3 of 3.
Attempts 1 and 2 failed fatalCrashIsRecoveredAndIngestedExactlyOnce:

"More stable emulator image": can we prove this? We might need more CI runs to confirm.

The description needs to be updated to cover just the metrics fixes, or the PR needs to be extended to fix the remaining flakes.

(And a BT ticket hasn't been linked here)

@BartoszLitwiniuk BartoszLitwiniuk changed the title ci: Fix flaky Sauce instrumentation tests and stabilize 16 KB emulator job test: Fix metrics-related timeouts in instrumentation tests and stabilize 16 KB emulator job Oct 20, 2026
@KishanPRao
KishanPRao self-requested a review October 21, 2026 11:43
@BartoszLitwiniuk
BartoszLitwiniuk merged commit fb639b0 into master Oct 21, 2026
41 of 44 checks passed
@BartoszLitwiniuk
BartoszLitwiniuk deleted the bugfix/sauce-instrumentation-tests branch October 21, 2026 19:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants