Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,51 @@
## Unreleased

Android: a system stop now counts as a failed attempt only when it cut a
request. Before, every stop at the job's time limit added one backoff step,
up to 2 hours, also for a run that only waited for the network or for a free
transmit slot and sent nothing. Now a run that only waited goes back to
`queued` with no backoff step. A time-limit stop that cut a request still
takes one backoff step. The foreground-service time limit (Android 15 and
later) now counts as a time-limit stop.

After every time-limit stop, the library schedules a fresh WorkManager row
for the entry: at once after a run that only waited, or when its backoff
ends, also for a short backoff. WorkManager runs the stopped row again too,
but only after its own backoff, which doubles with each start of that row.
By the WorkManager 2.12.0 code that wait is 64 minutes at the 8th start and
128 minutes at the 9th, so it passes an hour after about 9 starts. A fresh
row does not have that wait. Stops for other reasons go back to `queued`
with no new row, as before: the stopped row runs again when the system
allows it and after WorkManager's own backoff.

A run that finds its entry waiting longer than it can sleep out now makes
sure the entry's wake row exists, and keeps one that does. Before, a wake
lost to a failed enqueue or a process death left the entry waiting for the
next boot sweep, `resume()` or `mutate()` of the same id.

A chunked run no longer settles `completed` while a part is not accepted.
If a part ends without an answer and without a failure, the run goes back
to `queued` with no outcome and the worker is retried.

Stops for quota, standby bucket, background restriction, device state and
the user write a warning to `takeDiagnostics()`, only when the run had
already taken the entry. There is one record per reason, for example
`run stopped by the system: QUOTA`. Its `data` holds the latest `entryId`
and a `count` of such stops since the last `takeDiagnostics()`. So many
stops cannot push older warnings and errors out of the log. Android 10 and
11 (API 29 and 30) cannot read stop reasons, so every stop there goes back
to `queued` with no backoff step and no warning, as before.

A short backoff that a run waits out in place now saves its backoff step at
once. A stop during that wait keeps the step. A stopped entry whose
`nextAttemptAt` is already past no longer shows it.

**Behavior change:** the attempt is now written ahead after the request
gets a transmit slot, not before. A run that waits for a slot and is stopped
counts no attempt. So `attempts`, and the `attempt` field of later attempt
events, are lower for uploads that only waited. The number of attempt
events does not change. That is the correct count.

## 11.3.0

Added `useRequests(filter?)` and `useRequestsWithProgress(filter?)`, React
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,16 @@ internal class ChunkedTransfer(private val run: EntryRun) {
return failed.settlement
}
// Every executor returned, and an executor returns only when its part was
// accepted: the server's auto-publish condition.
// accepted: the server's auto-publish condition. A part that ended with a
// CancellationException while the run was not cancelled also returns, so
// check before completing.
var missing = pending.filterNot { it in acceptedHere }
if (missing.isNotEmpty()) {
// A part can also return when the store already shows it accepted.
val stored = run.ops.latest(run.entryId, run.generation).descriptor!!.parts!!
missing = missing.filterNot { stored[it].accepted }
if (missing.isNotEmpty()) throw EntryRun.PartsNotAcceptedException(missing)
}
return Settlement.Completed(null, lastAcceptedUrl ?: d0.reportUrl, d0.method)
}

Expand Down
12 changes: 12 additions & 0 deletions android/src/main/java/ai/openspace/backgroundupload/Diag.kt
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,18 @@ internal object Diag {
record(DiagnosticLog.INFO, message, null, data, replaceUntaken)
}

/**
* A system stop of a run, as a warning. One record per [kind]: it holds
* the latest entry id and how many such stops happened since JS last
* took the log.
*/
fun countStop(kind: String, entryId: String) {
val message = "run stopped by the system: $kind"
runCatching { Log.w(TAG, "$message ($entryId)") }
val target = log ?: return
runCatching { target.count(DiagnosticLog.WARN, message, clock(), mapOf("entryId" to entryId)) }
}

private fun record(
level: String,
message: String,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ class DiagnosticLog(private val file: File) {
const val INFO = "info"
const val WARN = "warn"
const val ERROR = "error"
const val COUNT_KEY = "count"

private val gson = Gson()
private val listType = object : TypeToken<List<Record>>() {}.type
Expand Down Expand Up @@ -88,6 +89,21 @@ class DiagnosticLog(private val file: File) {
}
}

/**
* A repeating event in one slot, as [replace] does, with a count. The new
* record carries [data], plus `count`: 1 more than the untaken record of
* the same level and message, or 1. So many events cannot push older
* records out of the log.
*/
@Synchronized
fun count(level: String, message: String, at: Long, data: Map<String, Any?>) {
runCatching {
val earlier = records().lastOrNull { it.level == level && it.message == message }
val count = ((earlier?.data?.get(COUNT_KEY) as? Number)?.toInt() ?: 0) + 1
replace(Record(at, level, message, null, data + (COUNT_KEY to count)))
}
}

/** Every record, oldest first, then an empty log. */
@Synchronized
fun drain(): List<Record> {
Expand Down
121 changes: 96 additions & 25 deletions android/src/main/java/ai/openspace/backgroundupload/EntryRun.kt
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,14 @@ internal class EntryRun(
/** The entry was paused (whole queue or its key) between the module's pause and the work cancel reaching us. */
class PausedException : Exception("paused")

/**
* A chunked run ended with [indexes] not accepted, and no part failed.
* An IOException, so the run takes the store-failure path: back to queued
* with no outcome, and the worker returns retry.
*/
class PartsNotAcceptedException(val indexes: List<Int>) :
IOException("chunked run ended with parts $indexes not accepted")

/** How one attempt ended, after the retry table. */
sealed class AttemptResult {
data class Accepted(val response: UploadResponse) : AttemptResult()
Expand Down Expand Up @@ -79,6 +87,10 @@ internal class EntryRun(
var liveBytes = 0L
private set

/** A cancel reached a request after its write-ahead. Chunked parts set it from their own coroutines. */
@Volatile
private var stopCutRequest = false

/**
* Returns when the run is over. Throws a CancellationException after it
* handled a stop, or an IOException after a store write failed (the
Expand All @@ -105,6 +117,8 @@ internal class EntryRun(
var first = true
while (true) {
try {
// A part cancelled by a sibling's park in the last pass does not count.
stopCutRequest = false
if (!first) current = ops.latest(entryId, generation)
first = false
host.foreground(current)
Expand Down Expand Up @@ -134,13 +148,14 @@ internal class EntryRun(
// A system stop moves a running entry back to queued. A pause or a
// cancel already moved it; then this does nothing.
endProgress(completed = false)
if (host.stoppedByTimeout()) releaseAfterTimeout() else ops.stopped(entryId, generation)
afterStop(host.stopKind())
throw error
} catch (error: IOException) {
// A store write failed (disk full, directory briefly unwritable).
// The attempt step classifies every network IOException itself, so
// one that lands here is storage: transient, no outcome. Back to
// queued; the caller returns retry.
// A store write failed (disk full, directory briefly unwritable), or
// a chunked run ended with a part not accepted. The attempt step
// classifies every network IOException itself, so one that lands here
// is not a network failure: transient, no outcome. Back to queued;
// the caller returns retry.
endProgress(completed = false)
ops.stopped(entryId, generation)
throw error
Expand Down Expand Up @@ -183,11 +198,33 @@ internal class EntryRun(
)

/**
* The system stopped the run at its time limit. That happens to a
* headless run whose foreground start was denied (API 31+), after about
* 10 minutes. Starting again at once would send the body from byte 0 on
* every run, so this is one more transient failure: the next backoff
* step, then a wake.
* A system stop. Only a time-limit stop that cut a request is a failed
* attempt. A run that only waited (for the network, a slot, or a short
* backoff) sent nothing, so it goes back to queued with no backoff step.
*
* After any time-limit stop a fresh wake row starts the entry again.
* WorkManager also runs the stopped row again, but only after its own
* backoff, which grows with every start of that row. The other stop
* reasons say nothing about the server, and the system decides when their
* row runs again.
*
* The state change is saved first, then the diagnostic record.
*/
private fun afterStop(kind: StopKind) {
when {
kind == StopKind.TIME_LIMIT && stopCutRequest -> releaseAfterTimeout()
kind == StopKind.TIME_LIMIT -> ops.stopped(entryId, generation, wake = true)
else -> ops.stopped(entryId, generation)
}
if (kind.logged) Diag.countStop(kind.name, entryId)
}

/**
* The system stopped the run at its time limit while a request was on its
* way. That happens to a run with no foreground mode after about 10
* minutes, or to a foreground run whose service type used up its time.
* Starting again at once would send the body from byte 0 on every run, so
* this is one more transient failure: the next backoff step, then a wake.
*/
private fun releaseAfterTimeout() {
runCatching {
Expand All @@ -197,20 +234,31 @@ internal class EntryRun(
val now = host.now()
val backoff = RetryClassifier.backoffMs(policy(e), streak)
ops.release(entryId, generation, RetryClassifier.nextAttemptAt(now, backoff, e.expiresAt), streak)
}.onFailure { Diag.error("could not release '$entryId' after a timeout stop", it) }
}.onFailure {
if (it is WorkerOps.WakeNotScheduledException) {
Diag.error("released '$entryId' after a timeout stop, but could not schedule its wake", it.cause)
} else {
Diag.error("could not release '$entryId' after a timeout stop", it)
}
}
}

/**
* Sleeps out a short backoff remainder. False when the wait is long (the
* wake run comes back for it) or the entry is no longer queued.
* wake run comes back for it) or the entry is no longer queued. For a long
* wait it first makes sure a wake is pending, so a lost wake does not
* leave the entry waiting for the next boot sweep.
*/
private suspend fun waitUntilDue(initial: QueueEntry): Boolean {
var e = initial
while (true) {
val at = e.nextAttemptAt ?: return true
val remaining = at - host.now()
if (remaining <= 0) return true
if (remaining > RetryClassifier.IN_WORKER_BACKOFF_MAX_MS) return false
if (remaining > RetryClassifier.IN_WORKER_BACKOFF_MAX_MS) {
if (e.state == EntryState.QUEUED) ops.ensureWake(e, at)
return false
}
host.sleep(min(remaining, WAIT_POLL_MS))
e = store.load(entryId) ?: return false
if (e.state != EntryState.QUEUED && e.state != EntryState.RUNNING) return false
Expand Down Expand Up @@ -239,11 +287,15 @@ internal class EntryRun(
RetryClassifier.policy(ops.settings().retry, entry.descriptor?.retry)

/**
* One HTTP attempt, the step both transfers share: write the attempt
* ahead (attempts + 1, its X-Request-Id), send, emit the attempt event,
* and classify. [partIndex] is null for a simple entry. [body] builds the
* request body from the entry the attempt runs under. [fileExists] tells a
* missing payload from a network failure.
* One HTTP attempt, the step both transfers share: take a transmit slot,
* write the attempt ahead (attempts + 1, its X-Request-Id), send, emit the
* attempt event, and classify. [partIndex] is null for a simple entry.
* [body] builds the request body from the entry the attempt runs under.
* [fileExists] tells a missing payload from a network failure.
*
* The write-ahead happens inside the slot. A run that only waits for a
* slot sends nothing, so it counts no attempt, and the request takes the
* headers that are current when it starts to send.
*/
suspend fun attempt(
partIndex: Int?,
Expand All @@ -252,19 +304,38 @@ internal class EntryRun(
fileExists: () -> Boolean,
): Attempt {
val requestId = UUID.randomUUID().toString()
val entry = ops.recordAttempt(entryId, generation, requestId)
val highPriority = ops.latest(entryId, generation).descriptor!!.highPriority
var written: QueueEntry? = null
var failure: Throwable? = null
val response = try {
host.withSlot(highPriority) {
val e = ops.recordAttempt(entryId, generation, requestId)
written = e
val d = e.descriptor!!
val part = partIndex?.let { d.parts!![it] }
host.send(TransferRequest(part?.url ?: d.url!!, d.method, headersFor(d, part, requestId), body(d, part)), onProgress)
}
} catch (error: CancellationException) {
// A stop after the write-ahead cut a request that was on its way.
if (written != null) stopCutRequest = true
throw error
} catch (error: Throwable) {
// With no write-ahead, the store write failed: not a network attempt.
if (written == null) throw error
failure = error
null
}

val entry = written!!
// The generation of the headers this attempt sends: both come from the same entry.
val headerGeneration = entry.headerGeneration
val d = entry.descriptor!!
val part = partIndex?.let { d.parts!![it] }
val url = part?.url ?: d.url!!
val policy = policy(entry)

val response = try {
host.send(TransferRequest(url, d.method, headersFor(d, part, requestId), body(d, part), d.highPriority), onProgress)
} catch (error: CancellationException) {
throw error
} catch (error: Throwable) {
if (response == null) {
val error = failure!!
// A failed probe reads as present, so a flaky check is a retryable
// network error and not a terminal "file gone".
val verdict = RetryClassifier.classifyFailure(error, runCatching(fileExists).getOrDefault(true))
Expand Down Expand Up @@ -316,7 +387,7 @@ internal class EntryRun(
val now = host.now()
if (backoff <= RetryClassifier.IN_WORKER_BACKOFF_MAX_MS) {
val wait = min(backoff, max(0L, expiresAt - now))
ops.backingOff(entryId, generation, now + wait)
ops.backingOff(entryId, generation, now + wait, streak)
host.sleep(wait)
return
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package ai.openspace.backgroundupload

import kotlin.math.max

/**
* The state changes of one entry, as pure functions. [QueueController] and
* [WorkerOps] apply them inside `QueueStore.compute`, so each one is atomic
Expand All @@ -21,10 +23,15 @@ object EntryTransitions {

/**
* A short backoff that the worker waits out itself. The row stays running
* and shows when the next attempt is due.
* and shows when the next attempt is due. The failure's step is saved now,
* not at the next release. Chunked parts keep their own streaks, so the
* entry keeps the highest; an accepted part resets it to 0.
*/
fun toBackingOff(e: QueueEntry, nextAttemptAt: Long, now: Long) =
e.copy(nextAttemptAt = nextAttemptAt, updatedAt = now)
fun toBackingOff(e: QueueEntry, nextAttemptAt: Long, streak: Int, now: Long) = e.copy(
nextAttemptAt = nextAttemptAt,
backoffStreak = max(e.backoffStreak, streak),
updatedAt = now,
)

/** One attempt starts: attempts + 1, its X-Request-Id, and no pending backoff. */
fun toAttempt(e: QueueEntry, requestId: String, now: Long) = e.copy(
Expand Down Expand Up @@ -71,8 +78,15 @@ object EntryTransitions {
e.copy(state = EntryState.QUEUED, parkedGeneration = null, updatedAt = now)
}

/** A system stop of a running worker. WorkManager runs the row again. No outcome. */
fun toStopped(e: QueueEntry, now: Long) = e.copy(state = EntryState.QUEUED, updatedAt = now)
/**
* A system stop of a running worker. No outcome. A nextAttemptAt that is
* already past is cleared: a queued row does not show a due time behind it.
*/
fun toStopped(e: QueueEntry, now: Long) = e.copy(
state = EntryState.QUEUED,
nextAttemptAt = e.nextAttemptAt?.takeIf { it > now },
updatedAt = now,
)

/** updateHeaders() on a parked entry. */
fun toUnparked(e: QueueEntry, paused: Boolean, now: Long) = e.copy(
Expand Down
Loading
Loading