Skip to content

Android: a system stop counts as a failed attempt only when it cut a request - #85

Open
dmurphy5 wants to merge 3 commits into
masterfrom
dylan/android-stop-not-failure
Open

dmurphy5 wants to merge 3 commits into
masterfrom
dylan/android-stop-not-failure

Conversation

@dmurphy5

Copy link
Copy Markdown

What was wrong

When Android stopped an upload job at its time limit, the library counted the stop as a failed send and added one backoff step. The wait grows with each step, up to 2 hours. It did this for every time-limit stop, also when the run sent nothing:

  • The run waited for the network the whole time.
  • The run waited for a free transmit slot. The library wrote the attempt ahead (attempts + 1) before it took the slot, so this wait also counted one attempt.

Smaller problems in the same code:

  • The foreground-service time limit (Android 15 and later) was not treated as a time limit.
  • Other system stops (quota, standby bucket, and so on) left no record, so takeDiagnostics() could not show why a queue did not move.
  • A stop during a short in-worker backoff lost that failure's backoff step.

What changes

All changes are Android only.

  1. A stop counts only when it cut a request. EntryRun sets a flag when a cancel reaches a request after its write-ahead. On a time-limit stop with the flag set, the run takes one backoff step, as before. Every other stop moves the entry back to queued with no backoff step.
  2. After every time-limit stop, a fresh WorkManager row starts the entry again. A cut request gets a wake row at the end of its backoff. An idle run gets a wake row at once, or at the nextAttemptAt it still had. Both use REPLACE on <id>#wake, so the row's run count starts at 0. The reason: WorkManager 2.12.0 also runs the stopped row again, but only after its own backoff. That backoff doubles with each start of that row: by the 2.12.0 code it is 64 minutes at the 8th start and 128 minutes at the 9th, up to 5 hours. While that row waits, schedule() adds no row. Other stop kinds keep today's behavior: no new row, and the stopped row runs again when the system allows it and after WorkManager's own backoff.
  3. The attempt is written ahead inside the slot. TransferHost.send is split into withSlot and send. attempt() takes the slot first, then calls ops.recordAttempt, then sends. A wait for a slot counts no attempt. TransferRequest.highPriority is removed. The slot argument is now the only source of the priority.
  4. stoppedByTimeout() becomes stopKind(). The table below shows the mapping. On API 29 and 30, getStopReason() is not available (@RequiresApi(31)), so every stop maps to OTHER, which is today's path.
  5. One diagnostic record per stop kind. The message is, for example, run stopped by the system: QUOTA. data holds the latest entryId and a count. The record replaces its untaken copy, as the boot snapshot does, so many stops cannot push older records out of the 50-record log. The state change is saved before the record.
  6. A short in-worker backoff saves its step at once. A stop during that wait keeps the step. A chunked entry keeps the highest step of its parts. A stop clears a nextAttemptAt that is already past.
  7. A lost wake is made again. A run that finds its entry queued with a wait too long to sleep out schedules the wake with KEEP. A wake lost to a failed enqueue or a process death is made again, and a pending one stays. Master had this gap too.
  8. A chunked entry does not complete with a part that was not accepted. Before it settles completed, a chunked run checks every pending part. If a part ended with its own CancellationException while the run was not cancelled, the run throws PartsNotAcceptedException (an IOException). The entry goes back to queued with no outcome, and the worker returns retry. No production path to this case is known.

The plan proposed no wake for a wait of 30 s or less, and the first commit built it. The review found that rule depends on WorkManager's backoff of the stopped row, so the second commit removes it.

WorkManager stop reason Value Kind What the run does
TIMEOUT 3 TIME_LIMIT Backoff step and wake row if a request was cut. Else a wake row at once.
FOREGROUND_SERVICE_TIMEOUT -128 TIME_LIMIT Same as TIMEOUT
CONSTRAINT_CONNECTIVITY 7 NETWORK Back to queued, never a failure
QUOTA, APP_STANDBY, BACKGROUND_RESTRICTION, DEVICE_STATE, USER 10, 12, 11, 4, 13 one kind each Back to queued, then one counted record per kind
All others, and API 29 and 30 OTHER Back to queued
flowchart TD
    A[System stops the run] --> B{API 31 or later?}
    B -- no --> Q[Back to queued, no backoff step]
    B -- yes --> C{Stop reason}
    C -- network or other --> Q
    C -- quota, standby, restriction, device state, user --> Q2[Back to queued, no backoff step] --> W[Update the counted record for that kind]
    C -- time limit or FGS time limit --> D{Did the stop cut a request?}
    D -- yes --> S[One backoff step] --> K[Fresh wake row when the backoff ends]
    D -- no, it only waited --> I[Back to queued, no backoff step] --> N[Fresh wake row now, or at the nextAttemptAt still due]
Loading

Files: EntryRun.kt, TransferHost.kt, EntryWorker.kt, WorkerOps.kt, EntryTransitions.kt, ChunkedUploadWorker.kt, Diag.kt, DiagnosticLog.kt, UploadUtils.kt, tests, CHANGELOG.md. Main-chain rows, the transmit gate's rotation, the journal and iOS are unchanged.

Behavior change

  • attempts, and the attempt field of later attempt events, are lower for uploads that only waited for a slot. The number of attempt events does not change. That is the correct count.
  • An upload that was idle when the system stopped it no longer builds up a backoff of up to 2 hours.
  • Every time-limit stop schedules a fresh wake row, also for a short wait.
  • Quota, standby, restriction, device-state and user stops add one counted record per reason to takeDiagnostics(). A record is written only when the run had already taken the entry.

Tests

New or changed in EntryRunTest, with FakeHost and FakeScheduler (which now records each wake's replace argument):

  • A time-limit stop while it waits for the network: queued, streak 0, 0 attempts, one fresh wake row at once.
  • A time-limit stop while it waits for a slot: 0 attempts, streak unchanged, one fresh wake row at once.
  • The attempt is written ahead inside the slot.
  • A high-priority entry asks for a high-priority slot, and a normal entry for a normal one.
  • a timeout stop takes one more backoff step instead of restarting at once keeps its original numbers: streak 2 to 3 and a fresh wake row at +4 s.
  • A foreground-service timeout that cut a request takes one backoff step.
  • A chunked part cut by a time-limit stop: streak +1, a wake row, the accepted parts, attempts and the cut request's id are kept.
  • A time-limit stop during a short backoff keeps the 503's step and wakes at the time that is still due.
  • A stop after a short backoff ran out clears the past nextAttemptAt.
  • A quota stop writes one counted record with no wake. 50 stops leave one record per kind with the right count, and an older error survives.
  • The stop is saved before the record is written.
  • A network-constraint stop: no backoff step, no record.
  • A chunked part cut by a sibling's park does not make a later idle stop a failure.
  • A run that finds a long wait makes its wake again with KEEP. A wake lost after a time-limit release is made again by the stopped row's re-run, and the error record names the failed wake.
  • A failed wake in an idle time-limit stop leaves the entry queued, ends as a stop, and writes one error record.
  • A chunked part that ends with its own cancel does not let the entry complete.
  • A chunked entry keeps the highest backoff step of its concurrent parts.

New StopKindTest: each WorkManager constant has its documented value, and each maps to its kind.

Each fix was broken on purpose and at least one test failed each time. Local runs: yarn lint:ci, yarn typecheck, yarn test (305 passed), Android testDebugUnitTest --no-daemon (445 passed), swift test in ios/ (383 executed, 1 skipped, 0 failures).

Ships together with the change that makes a run with no foreground mode wait for the network on a constrained wake. Without it, an offline run with no foreground mode starts again at once after each stop.

Two independent reviews were applied. The WorkManager numbers come from reading the 2.12.0 classes.

🤖 Generated with Claude Code

dmurphy5 and others added 3 commits November 17, 2026 16:41
…request

A stop at the job's time limit added one backoff step every time, up to
2 hours, also for a run that only waited for the network or for a transmit
slot and sent nothing.

- EntryRun sets a flag when a cancel reaches a request after its
  write-ahead. Only a time-limit stop with the flag takes a backoff step.
  Every other stop goes back to queued with no step.
- TransferHost.send is split into withSlot and send. The attempt is
  written ahead inside the slot, so a wait for a slot counts no attempt.
- stoppedByTimeout() becomes stopKind(). The foreground-service timeout
  counts as a time limit. Quota, standby, background restriction, device
  state and user stops write one persisted warning. API 29 and 30 cannot
  read stop reasons and keep the plain path.
- A time-limit release with a wait of 30 s or less schedules no wake. The
  stopped row runs again and waits out the rest.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Review fixes for "a system stop counts as a failed attempt only when it cut
a request".

- After every time-limit stop, schedule a fresh wake row (REPLACE, run
  count 0): at once after an idle stop, or at the backoff time after a cut
  request, also for a short wait. WorkManager 2.12.0 runs a stopped row
  again only after its own backoff, which grows with each start of that
  row (up to 5 hours), and schedule() adds no row while it waits. This
  drops the "no wake for a wait of 30 s or less" rule.
- Write one diagnostic record per stop kind, with no entry id in the
  message, the latest entry id and a count in data, and the
  replace-if-untaken rule. Many stops can no longer push older warnings
  and errors out of the 50-record log.
- Save the state change before the diagnostic record.
- Record the slot's highPriority argument in FakeHost and test it. Remove
  TransferRequest.highPriority: nothing reads it now.
- An in-worker backoff saves its step at once, so a stop during the wait
  keeps it.
- A stop clears a nextAttemptAt that is already past.
- Test a chunked part cut by a time-limit stop.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Follow-ups from the delta review.

- A run that finds its entry queued with a wait too long to sleep out
  schedules the wake with KEEP. A wake lost to a failed enqueue or a
  process death is made again; a pending one stays. A release that saved
  the entry but failed to schedule its wake now logs that, not "could not
  release".
- A chunked run checks that every pending part was accepted before it
  settles completed. A part that ended with its own CancellationException
  no longer lets the entry complete: the run throws
  PartsNotAcceptedException (an IOException), the entry goes back to
  queued with no outcome, and the worker returns retry.
- Tests for the highest-part-streak rule and for a failed wake in an
  idle stop.
- CHANGELOG: exact WorkManager backoff numbers, and the other stop kinds
  also wait for WorkManager's own backoff.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant