Skip to content

fix(test): Count Swift Testing failures per test, not per issue - #541

Open
breken-ai wants to merge 2 commits into
getsentry:mainfrom
breken-ai:fix/swift-testing-failed-test-count
Open

breken-ai wants to merge 2 commits into
getsentry:mainfrom
breken-ai:fix/swift-testing-failed-test-count

Conversation

@breken-ai

Copy link
Copy Markdown

Fixes #533

Problem

When one Swift Testing test records several failed #expect assertions, progress and the test summary count each assertion as a failed test. With the input from #533 (one passing test, one failing test with 3 issues), the stream reports:

  • progress completed: 2, failed: 3
  • summary totalTests: 3, passedTests: 0, failedTests: 3

The reporter's real run of 16 tests (8 passed, 8 failed) came out as totalTests: 25, passedTests: 0, failedTests: 25. A consumer that checks progress against the test-case results rejects that stream as contradictory.

There are two causes:

  1. createXcodebuildEventParser skips native ✘ Test "..." failed lines when counting failures and uses the run summary's with N issues count instead.
  2. createTestSummaryFragment (and createStateTestCounts in the domain results) take Math.max(failedTests, testFailures.length), and testFailures holds one entry per diagnostic, not per test.

Fix

  • The event parser counts native failed result lines per test and records the issue count from each line (with N issues). At the run summary, it subtracts the issues already attributed to reported failures. Any issues left over can mark unreported tests as failed, capped at the number of unreported tests. This keeps the behavior from defers Swift Testing failure progress until the run summary for summary-only failures.
  • A new countFailedTests helper in xcodebuild-run-state.ts counts failure diagnostics by distinct test. The run-state summary and the domain-result counts both use it.
  • Progress is still deferred for native failed lines, as before.

For the #533 input, the stream now reports progress completed: 2, failed: 1 and summary totalTests: 2, passedTests: 1, failedTests: 1. All three failure diagnostics are still emitted.

Tests

Both tests fail on d13ff0c (2 failed / 50 passed in the two files) and pass with the fix (52/52). The full unit suite passes (243 files, 2593 tests). npm run build, npm run typecheck, npm run lint (0 errors) and prettier --check on the changed files also pass. I did not run the snapshot and smoke suites.

I wrote this change with AI assistance (Claude) and checked it against the reproduction in #533 and the tests above.

breken-ai and others added 2 commits September 29, 2026 18:22
A Swift Testing test that records several failed #expect assertions was
counted as several failed tests. The run summary reports issues rather than
failed tests, and the run state counted failure diagnostics as failed tests,
so progress and the final summary could report more failures than tests run
and zero passes.

Count native failed result lines per test, attribute their issue counts, and
only let unattributed summary issues mark unreported tests as failed. Count
failure diagnostics by distinct test when reconciling the summary.

Fixes getsentry#533

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Swift Testing JSONL counts assertion issues as failed tests

1 participant