Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 69 additions & 7 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,16 @@ transmit slot and sent nothing. Now a run that only waited goes back to
takes one backoff step. The foreground-service time limit (Android 15 and
later) now counts as a time-limit stop.

After every time-limit stop, the library schedules a fresh WorkManager row
for the entry: at once after a run that only waited, or when its backoff
ends, also for a short backoff. WorkManager runs the stopped row again too,
but only after its own backoff, which doubles with each start of that row.
By the WorkManager 2.12.0 code that wait is 64 minutes at the 8th start and
128 minutes at the 9th, so it passes an hour after about 9 starts. A fresh
row does not have that wait. Stops for other reasons go back to `queued`
After a time-limit stop, the library schedules a fresh WorkManager row for
the entry: at once after a run that only waited, or when its backoff ends,
also for a short backoff. WorkManager runs the stopped row again too, but
only after its own backoff, which doubles with each start of that row and
counts from the row's last enqueue time. By the WorkManager 2.12.0 code that
wait is 64 minutes at the 8th start and 128 minutes at the 9th, so it passes
an hour after about 9 starts. A fresh row does not have that wait. When that
wait is already over at the stop (for a 10-minute run, up to the 5th start),
the stopped row runs again at once and the library adds no fresh row, so
the entry does not get two rows. Stops for other reasons go back to `queued`
with no new row, as before: the stopped row runs again when the system
allows it and after WorkManager's own backoff.

Expand Down Expand Up @@ -46,6 +49,65 @@ counts no attempt. So `attempts`, and the `attempt` field of later attempt
events, are lower for uploads that only waited. The number of attempt
events does not change. That is the correct count.

**Behavior change:** on Android, the boot sweep, `resume()` and a return to
the foreground now end some backoff waits and start those uploads at once.
A wait is ended only when the failure behind it had no server response (a
transport failure, or a stop that cut the request), or when an older
version saved the entry, and only 30 s or more after that failure. A wait
after a server response (a 5xx, 408, 429, or a status a `transient` rule
retries) keeps its time: the server asked for it, and ending it early
would spend a rule's `maxRetries` faster. Before, an entry on a long
backoff (up to 2 hours) kept its wait when the app opened, and `resume()`
kept it too. `resume({ keys })` affects those keys only. The entry keeps at
most 5 backoff steps, so if the request fails again, the next wait is 32
times `baseMs` (32 s at the default 1 s base), and then it doubles. "A
return to the foreground" is every Activity resume, also after a
permission dialog or a picker, so the library does this at most once every
30 s, measured on a clock that does not change with the wall clock. The
app needs no code for this. Entries now save the cause and time of their
last failure; entries saved by an older version read as unknown.

**Behavior change:** on Android, `pause()` now keeps a backoff wait with
its cause. `resume()` gives the entry back what is left of it, ended only
by the rule above. Before, `pause()` cleared the wait, so a pause and a
resume sent the request again at once, also after a server response. A
paused row still does not show `nextAttemptAt`. An entry whose wait stays
keeps its backoff steps, so the waits after it grow as with no pause. When
the wait ended or is cleared, it keeps at most 5 steps.

Android: the boot sweep, `resume()`, a return to the foreground,
`updateHeaders()` and a same-id `mutate()` now replace a WorkManager row
that WorkManager holds back after a system stop, so the entry can start at
once. This applies to every queued entry with no backoff wait, whatever its
last failure: it only makes sure a row exists. Before, the library added no
row while that stopped row waited, and the entry waited for WorkManager's
own backoff (up to 5 hours). The library replaces the row only when no row
of the entry's main chain runs, and keeps a row that WorkManager starts
within 30 s. It makes the new row before it cancels the old wake, and it
keeps a wake that the stored entry still needs. A return to the foreground
also makes the wake again (KEEP) for a queued entry whose wait stays and
that has no wake row, also when it has a main row that WorkManager holds
back.

Android: a call whose every try failed while it connected now waits at
most 15 minutes. Such a call sent no request. The failures are
`ConnectException`, `UnknownHostException`, `NoRouteToHostException`,
`PortUnreachableException`, `SSLHandshakeException` and
`SSLPeerUnverifiedException`. OkHttp attaches the earlier tries of a call
as suppressed exceptions; each of them must be one of these too, as when
every address of a host refuses. Any other earlier try may have sent the
body, and then the cap does not apply. Every attempt
starts on a network that Android reports as validated. Every other
failure, a timeout included, keeps the retry policy's `maxMs` (2 hours by
default), because the server may have the request.

Android: in a chunked entry, the part that releases a run for a long wait
takes the stored backoff with its own count. It goes on from it in the
next run, and only that part's acceptance resets it. The other parts count
their own failures from 0. Before, every part started at the stored
backoff, so one failed part after a long backoff sent the whole entry back
for the next, longer wait.

## 11.3.0

Added `useRequests(filter?)` and `useRequestsWithProgress(filter?)`, React
Expand Down
24 changes: 23 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,21 @@ at the next run, after a growing backoff. A body that needs more than
10 minutes headless cannot finish that way. Use `parts` for large bodies:
accepted parts are kept across runs. iOS has no equal limit.

**Backoff and the app.** When the app starts, calls `resume()`, or comes
to the foreground (every Activity resume, at most once every 30 s), the
library ends the retry backoff of a queued entry and runs it at once, when
two things hold: the failure behind the wait had no server response (or an
older version saved the entry), and it happened 30 s or more ago. A wait
after a server response, such as a 503 or a status a `transient` rule
retries, keeps its time. `resume({ keys })` affects those keys only. The
app needs no code for this. The entry keeps at most 5 backoff steps, so if
it fails again the next wait is 32 times `baseMs` (32 s at the default
1 s), then it doubles. For every queued entry with no wait, the library
also replaces a WorkManager row that WorkManager holds back after a system
stop, so the entry can start at once. In a chunked entry, the part whose
failures set the backoff goes on from it; the other parts count their own
failures from zero.

**Backups.** The queue store (`files/rnbgupload-chunked/`) and the journal
(`files/rnbgupload-settled/`) hold request headers, including auth tokens,
and staged bodies. Set `android:allowBackup="false"` in the host app, or
Expand Down Expand Up @@ -343,7 +358,7 @@ Retry classes:
| Attempt outcome | Action |
| --- | --- |
| 2xx, or an `accept` rule matches | settle `completed` |
| network failure, 5xx, 408, 429 | exponential backoff (base 1 s, max 2 h, jitter 0.2) until `expiresAt` |
| network failure, 5xx, 408, 429 | exponential backoff (base 1 s, max 2 h, jitter 0.2) until `expiresAt`. On Android a call whose every try failed while it connected waits at most 15 minutes |
| 401, 403 | park as `awaiting-auth`; resume on `updateHeaders()` |
| other 4xx matching a `retry.transient` rule (`status`, optional `bodyIncludes`), under the rule's `maxRetries` | as transient. The first matching rule in the list wins. `maxRetries: 3` retries three matching responses; the fourth settles. No `maxRetries` means no cap. See "Transient rule counts" below. A rule for 401, 403, 408 or 429 is rejected at the call |
| other 4xx in `retry.terminalHttp.exempt` (default `[404]`), when no rule matched | as transient, with no cap |
Expand Down Expand Up @@ -474,6 +489,13 @@ event (`paused`, then `queued`). No outcome is produced and the bytes are
kept. A paused entry past `expiresAt` settles `error` with
`errorKind: 'expired'` when it is resumed.

On Android, `resume()` also ends some retry backoff waits of the queued
entries in its scope, and they run at once. `pause()` keeps an entry's
wait, and `resume()` gives back what is left of it under the same rule. A
kept wait keeps its backoff steps; when the wait ended or is cleared, the
entry keeps at most 5. See "Backoff and the app" under Android platform
notes.

Neither call rejects because the settings cannot be saved. On Android,
either call can still reject `E_STORAGE` when an entry file cannot be saved.
See [Settings on a full disk](#settings-on-a-full-disk).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@ internal class ChunkedTransfer(private val run: EntryRun) {
run.progressStarted(total, acceptedBytes.get())

try {
ChunkedEngine.run(pending) { index -> executePart(index, blob, start.backoffStreak) }
ChunkedEngine.run(pending) { index ->
executePart(index, blob, start.backoffStreak, start.backoffPartStart, parts[index].start)
}
} catch (failed: PartFailed) {
return failed.settlement
}
Expand All @@ -68,8 +70,20 @@ internal class ChunkedTransfer(private val run: EntryRun) {
return Settlement.Completed(null, lastAcceptedUrl ?: d0.reportUrl, d0.method)
}

internal suspend fun executePart(index: Int, blob: File, initialStreak: Int) {
var streak = initialStreak
/**
* [initialStreak] is the entry's stored streak, and [streakPart] the start
* of the part whose failures set it. That part goes on from the stored
* streak, so a part that keeps failing keeps backing off across runs. The
* other parts count their own failures from 0, so an old streak does not
* slow the parts that work. When no part is named (an older version, or a
* stop that cut the run), every part uses the stored streak until a part
* is accepted in this run. [partStart] is the part's start from the
* entry the run began with: within one generation the parts keep their
* ranges and order.
*/
internal suspend fun executePart(index: Int, blob: File, initialStreak: Int, streakPart: Long?, partStart: Long) {
var base = if (streakPart == null || streakPart == partStart) initialStreak else 0
var failures = 0
while (true) {
if (index in acceptedHere) return
val latest = run.ops.latest(run.entryId, run.generation)
Expand Down Expand Up @@ -104,12 +118,14 @@ internal class ChunkedTransfer(private val run: EntryRun) {
}
is EntryRun.AttemptResult.Auth -> {
if (!r.reissue) throw EntryRun.ParkException(r.headerGeneration)
streak = 0
base = 0
failures = 0
}
EntryRun.AttemptResult.Transient -> {
onPartProgress(index, 0L)
streak++
run.backoffOrRelease(a.policy, streak, a.entry.expiresAt)
if (streakPart == null && acceptedHere.isNotEmpty()) base = 0
failures++
run.backoffOrRelease(a.policy, base + failures, a.entry.expiresAt, a.cause, partStart)
}
is EntryRun.AttemptResult.Terminal -> {
onPartProgress(index, 0L)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@ object EnqueueRules {
updatedAt = now,
nextAttemptAt = null,
backoffStreak = 0,
backoffPartStart = null,
headerGeneration = headerGeneration,
parkedGeneration = null,
generation = if (reopen) existing.generation + 1 else existing.generation,
Expand Down
68 changes: 51 additions & 17 deletions android/src/main/java/ai/openspace/backgroundupload/EntryRun.kt
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,14 @@ internal class EntryRun(
class ParkException(val headerGeneration: Int) : Exception("awaiting auth")

/** A backoff too long to wait here. */
class BackoffException(val nextAttemptAt: Long, val streak: Int) : Exception("released for backoff")
class BackoffException(
val nextAttemptAt: Long,
val streak: Int,
/** [WaitCause] of the failure. */
val waitCause: String,
/** The chunked part whose failure it was; null for a simple entry. */
val partStart: Long?,
) : Exception("released for backoff")

class ExpiredException : Exception("expired before completion")

Expand Down Expand Up @@ -61,6 +68,8 @@ internal class EntryRun(
val url: String,
val policy: RetryClassifier.Policy,
val result: AttemptResult,
/** [WaitCause]: whether the server answered. */
val cause: String,
) {
val method: String get() = entry.descriptor!!.method
}
Expand Down Expand Up @@ -132,7 +141,7 @@ internal class EntryRun(
if (ops.park(entryId, generation, park.headerGeneration) != WorkerOps.ParkResult.REISSUE) return
} catch (backoff: BackoffException) {
endProgress(completed = false)
ops.release(entryId, generation, backoff.nextAttemptAt, backoff.streak)
ops.release(entryId, generation, backoff.nextAttemptAt, backoff.streak, cause = backoff.waitCause, partStart = backoff.partStart)
return
} catch (error: ExpiredException) {
endProgress(completed = false)
Expand Down Expand Up @@ -202,38 +211,53 @@ internal class EntryRun(
* attempt. A run that only waited (for the network, a slot, or a short
* backoff) sent nothing, so it goes back to queued with no backoff step.
*
* After any time-limit stop a fresh wake row starts the entry again.
* WorkManager also runs the stopped row again, but only after its own
* backoff, which grows with every start of that row. The other stop
* reasons say nothing about the server, and the system decides when their
* row runs again.
* After a time-limit stop a fresh wake row starts the entry again, unless
* WorkManager starts the stopped row again at once. WorkManager runs the
* stopped row again after its own backoff, which grows with every start
* of that row ([WorkManagerBackoff]). For the first few starts that wait
* is already over at the stop; a wake then would be a second row for the
* entry, and one of the two would wait at the per-id gate for the whole
* upload. The other stop reasons say nothing about the server, and the
* system decides when their row runs again.
*
* The state change is saved first, then the diagnostic record.
*/
private fun afterStop(kind: StopKind) {
when {
kind == StopKind.TIME_LIMIT && stopCutRequest -> releaseAfterTimeout()
kind == StopKind.TIME_LIMIT -> ops.stopped(entryId, generation, wake = true)
else -> ops.stopped(entryId, generation)
kind != StopKind.TIME_LIMIT -> ops.stopped(entryId, generation)
stopCutRequest -> releaseAfterTimeout(wake = !stoppedRowRestartsNow())
else -> ops.stopped(entryId, generation, wake = !stoppedRowRestartsNow())
}
if (kind.logged) Diag.countStop(kind.name, entryId)
}

/** WorkManager starts the stopped row again at once. False when unknown. */
private fun stoppedRowRestartsNow(): Boolean {
val by = host.rowRestartsBy() ?: return false
return by <= host.now()
}

/**
* The system stopped the run at its time limit while a request was on its
* way. That happens to a run with no foreground mode after about 10
* minutes, or to a foreground run whose service type used up its time.
* Starting again at once would send the body from byte 0 on every run, so
* this is one more transient failure: the next backoff step, then a wake.
* With [wake] false the stopped row, which WorkManager starts again at
* once, sleeps out a short wait or makes the wake itself ([waitUntilDue]).
*/
private fun releaseAfterTimeout() {
private fun releaseAfterTimeout(wake: Boolean) {
runCatching {
val e = store.load(entryId)
if (!EntryTransitions.isOwnedRun(e, generation)) return
val streak = e!!.backoffStreak + 1
val now = host.now()
val backoff = RetryClassifier.backoffMs(policy(e), streak)
ops.release(entryId, generation, RetryClassifier.nextAttemptAt(now, backoff, e.expiresAt), streak)
// A cut request has no response. Which part it cut is not known.
ops.release(
entryId, generation, RetryClassifier.nextAttemptAt(now, backoff, e.expiresAt), streak, wake,
cause = WaitCause.NO_RESPONSE, partStart = null,
)
}.onFailure {
if (it is WorkerOps.WakeNotScheduledException) {
Diag.error("released '$entryId' after a timeout stop, but could not schedule its wake", it.cause)
Expand Down Expand Up @@ -348,7 +372,11 @@ internal class EntryRun(
val result = if (verdict is RetryClassifier.Verdict.Terminal) {
AttemptResult.Terminal(verdict.errorKind, verdict.message, null)
} else AttemptResult.Transient
return Attempt(entry, url, policy, result)
// The attempt started on a validated network (waitForNetwork). A call
// whose every try failed while it connected waits at most 15 minutes;
// any other failure keeps the entry's maximum.
val waitPolicy = if (RetryClassifier.reachedNoServer(error)) RetryClassifier.transportFailurePolicy(policy) else policy
return Attempt(entry, url, waitPolicy, result, WaitCause.NO_RESPONSE)
}

// A part counts on its own, so each part gets a rule's full maxRetries.
Expand All @@ -375,23 +403,29 @@ internal class EntryRun(
is RetryClassifier.Verdict.Transient -> AttemptResult.Transient
is RetryClassifier.Verdict.Terminal -> AttemptResult.Terminal("http", verdict.message, response)
}
return Attempt(entry, url, policy, result)
return Attempt(entry, url, policy, result, WaitCause.RESPONSE)
}

/**
* A short backoff waits here, with the row still running and showing
* nextAttemptAt; a long one throws [BackoffException] to release the run.
*/
suspend fun backoffOrRelease(policy: RetryClassifier.Policy, streak: Int, expiresAt: Long) {
suspend fun backoffOrRelease(
policy: RetryClassifier.Policy,
streak: Int,
expiresAt: Long,
cause: String,
partStart: Long? = null,
) {
val backoff = RetryClassifier.backoffMs(policy, streak)
val now = host.now()
if (backoff <= RetryClassifier.IN_WORKER_BACKOFF_MAX_MS) {
val wait = min(backoff, max(0L, expiresAt - now))
ops.backingOff(entryId, generation, now + wait, streak)
ops.backingOff(entryId, generation, now + wait, streak, cause, partStart)
host.sleep(wait)
return
}
throw BackoffException(RetryClassifier.nextAttemptAt(now, backoff, expiresAt), streak)
throw BackoffException(RetryClassifier.nextAttemptAt(now, backoff, expiresAt), streak, cause, partStart)
}

fun progressStarted(total: Long, sent: Long) {
Expand Down
Loading
Loading