Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,23 @@
## Unreleased

Android never expires a request. Its native code ignores the descriptor's
`expiresAt` and `configure()`'s `lifetimeMs`. The native module accepts an
entry with or without `expiresAt`, also one in the past, and a
`configure()` call with or without `lifetimeMs`. A request waits and retries
until the server accepts it, it settles as an error (for example a 4xx that
the retry rules do not cover, or a missing file), or the app cancels it.
Before, Android settled an entry `error` with `errorKind: 'expired'` once
`expiresAt` passed, and gave each parked (`awaiting-auth`) entry a
WorkManager wake at its `expiresAt` when that was less than 24 hours away.
Those wakes could fill the app's 20 JobScheduler slots, so other uploads had
to wait. A parked entry now has no WorkManager row and waits for
`updateHeaders()`. An app that needs a limit calls `cancel(id)` itself.
Entries an earlier version queued keep going and are sent, also when their
`expiresAt` has passed. An `expired` outcome that an earlier version stored
and the app has not yet acknowledged is still delivered, unchanged. New and
re-queued entries store an `expiresAt` that 11.0.2 to 11.3.0 never reach, so
an app that goes back to one of those versions does not expire them.

Android: a system stop now counts as a failed attempt only when it cut a
request. Before, every stop at the job's time limit added one backoff step,
up to 2 hours, also for a run that only waited for the network or for a free
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,17 @@ package ai.openspace.backgroundupload
* setting. entriesByWake counts the queued entries by what starts them:
* their backoff wake, or the network or Wi-Fi doorbell; doorbells says
* which doorbell rows exist. Both describe the rows and entries before the
* sweep changed them.
* sweep changed them. delayedPastHorizon counts the delayed runs due more
* than [FAR_DELAY_MS] ahead. This version makes none with the default retry
* settings (a backoff waits 2 hours at most), so such a run is a wake an
* earlier version left, or a backoff whose maxMs is over a day.
*/
internal object BootSnapshot {
const val MESSAGE = "boot sweep"

/** A delayed run due further ahead than this holds a JobScheduler slot for long. */
const val FAR_DELAY_MS = 24L * 60 * 60 * 1000

/**
* [entries]: the entries the sweep schedules, as it found them, so an
* entry that was running when its process died still reads as running
Expand Down Expand Up @@ -49,7 +55,7 @@ internal object BootSnapshot {
"runsByState" to runs.groupingBy { it.state }.eachCount(),
"delayedRuns" to runs.count { it.delayed },
"delayedPastHorizon" to runs.count { run ->
run.delayed && run.nextScheduleAt != null && run.nextScheduleAt - now > EXPIRY_WAKE_HORIZON_MS
run.delayed && run.nextScheduleAt != null && run.nextScheduleAt - now > FAR_DELAY_MS
},
"maxRunAttemptCount" to runs.maxOfOrNull { it.runAttemptCount },
"entriesByWake" to entriesByWake(entries, runs, now, settings),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,6 @@ internal class ChunkedTransfer(private val run: EntryRun) {
val latest = run.ops.latest(run.entryId, run.generation)
val stored = latest.descriptor!!.parts!![index]
if (stored.accepted) return
if (RetryClassifier.isExpired(run.host.now(), latest.expiresAt)) throw EntryRun.ExpiredException()

// A range past EOF can never be sent. length() is 0 for a missing file;
// that case falls through to the attempt, which classifies it as file.
Expand Down Expand Up @@ -125,7 +124,7 @@ internal class ChunkedTransfer(private val run: EntryRun) {
onPartProgress(index, 0L)
if (streakPart == null && acceptedHere.isNotEmpty()) base = 0
failures++
run.backoffOrRelease(a.policy, base + failures, a.entry.expiresAt, a.cause, partStart)
run.backoffOrRelease(a.policy, base + failures, a.cause, partStart)
}
is EntryRun.AttemptResult.Terminal -> {
onPartProgress(index, 0L)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@ object EnqueueRules {
attempts = 0,
bytesSent = runParts?.let { ChunkedParts.acceptedBytes(it) } ?: 0L,
totalBytes = staged.body.totalBytes,
expiresAt = p.expiresAt,
createdAt = now,
updatedAt = now,
headerGeneration = headerGeneration,
Expand All @@ -121,7 +120,7 @@ object EnqueueRules {
)

/**
* Same body. New headers, expiresAt, vars, accept, retry, wifiOnly,
* Same body. New headers, vars, accept, retry, wifiOnly,
* priority, and notification flag replace the stored ones; the body and
* accepted parts stay. A settled entry reopens with a fresh generation,
* attempts 0 (attempts count the current generation), and no transient
Expand Down Expand Up @@ -155,7 +154,8 @@ object EnqueueRules {
state = if (running) EntryState.RUNNING else initialState(paused),
attempts = if (reopen) 0 else existing.attempts,
bytesSent = parts?.let { ChunkedParts.acceptedBytes(it) } ?: if (running) existing.bytesSent else 0L,
expiresAt = p.expiresAt,
// Not read here. Written so that a downgrade does not expire the entry.
expiresAt = QueueEntry.NO_DEADLINE,
updatedAt = now,
nextAttemptAt = null,
backoffStreak = 0,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ import java.net.URI
* because React Native on iOS drops object keys whose value is null. Native
* keeps the text as it came: it is the body that goes out. A dataJson of
* "null" is the JSON body null, a real body.
*
* A descriptor may still carry `expiresAt`. Native ignores it: the library
* never expires a request.
*/
object EntryParsing {
class InvalidEntryException(message: String) : IllegalArgumentException(message)
Expand All @@ -25,7 +28,6 @@ object EntryParsing {
val key: String,
val varsJson: String,
val descriptor: Descriptor,
val expiresAt: Long,
)

private val METHODS = setOf("POST", "PUT", "PATCH", "DELETE", "GET")
Expand All @@ -36,8 +38,7 @@ object EntryParsing {
val varsJson = entry.string("varsJson") ?: invalid("varsJson is required")
if (!JsonBridge.isJson(varsJson)) invalid("varsJson is not JSON text")
val d = entry.map("descriptor") ?: invalid("descriptor is required")
val expiresAt = d.number("expiresAt")?.toLong() ?: invalid("descriptor.expiresAt is required")
return Parsed(id, key, varsJson, descriptor(d), expiresAt)
return Parsed(id, key, varsJson, descriptor(d))
}

fun descriptor(d: ReadableMap): Descriptor {
Expand Down
39 changes: 5 additions & 34 deletions android/src/main/java/ai/openspace/backgroundupload/EntryRun.kt
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import kotlinx.coroutines.CancellationException
import okhttp3.RequestBody
import java.io.IOException
import java.util.UUID
import kotlin.math.max
import kotlin.math.min

/**
Expand All @@ -16,7 +15,7 @@ import kotlin.math.min
* settle, park, or release. The body kind at run time picks the transfer,
* [SimpleTransfer] or [ChunkedTransfer], so a kind change under a queued run
* is safe. The entry is read from the store at start and again before every
* attempt, so fresh headers and a new expiresAt reach a running run.
* attempt, so fresh headers reach a running run.
*/
internal class EntryRun(
val entryId: String,
Expand All @@ -37,8 +36,6 @@ internal class EntryRun(
val partStart: Long?,
) : Exception("released for backoff")

class ExpiredException : Exception("expired before completion")

/**
* The run waited long enough for the network and hands the wait to a
* doorbell row ([WorkerOps.awaitNetwork]).
Expand Down Expand Up @@ -176,14 +173,6 @@ internal class EntryRun(
suspend fun run() {
val initial = store.load(entryId) ?: return // forgotten while queued
if (initial.legacy) return
if (initial.state == EntryState.AWAITING_AUTH) {
// The expiry wake of a parked entry. No attempt ran here, so the
// stored bytes stand.
if (RetryClassifier.isExpired(host.now(), initial.expiresAt)) {
ops.settle(entryId, initial.generation, expired(initial).copy(bytesSent = null))
}
return
}
if (initial.state != EntryState.QUEUED && initial.state != EntryState.RUNNING) return
if (ops.settings().isPaused(initial.key)) return
if (!waitUntilDue(initial)) return
Expand Down Expand Up @@ -212,10 +201,6 @@ internal class EntryRun(
endProgress(completed = false)
ops.release(entryId, generation, backoff.nextAttemptAt, backoff.streak, cause = backoff.waitCause, partStart = backoff.partStart)
return
} catch (error: ExpiredException) {
endProgress(completed = false)
ops.settle(entryId, generation, expired(current))
return
} catch (error: NotOwnedException) {
endProgress(completed = false)
return
Expand Down Expand Up @@ -271,16 +256,6 @@ internal class EntryRun(
private fun liveBytesOf(entry: QueueEntry): Long? =
if (entry.body?.kind == StagedBody.CHUNKED) null else liveBytes

private fun expired(entry: QueueEntry) = Settlement.Failed(
errorKind = "expired",
message = "expired before completion",
response = null,
partIndex = null,
url = entry.descriptor?.reportUrl ?: "",
method = entry.descriptor?.method ?: "POST",
bytesSent = liveBytesOf(entry),
)

/**
* A system stop. Only a time-limit stop that cut a request is a failed
* attempt. A run that only waited (for the network, a slot, or a short
Expand Down Expand Up @@ -326,11 +301,10 @@ internal class EntryRun(
val e = store.load(entryId)
if (!EntryTransitions.isOwnedRun(e, generation)) return
val streak = e!!.backoffStreak + 1
val now = host.now()
val backoff = RetryClassifier.backoffMs(policy(e), streak)
// A cut request has no response. Which part it cut is not known.
ops.release(
entryId, generation, RetryClassifier.nextAttemptAt(now, backoff, e.expiresAt), streak, wake,
entryId, generation, host.now() + backoff, streak, wake,
cause = WaitCause.NO_RESPONSE, partStart = null,
)
}.onFailure {
Expand Down Expand Up @@ -402,7 +376,6 @@ internal class EntryRun(
val s = ops.settings()
val entry = ops.latest(entryId, generation)
if (s.isPaused(entry.key)) throw PausedException()
if (RetryClassifier.isExpired(host.now(), entry.expiresAt)) throw ExpiredException()
val wifiOnly = s.wifiOnlyFor(entry)
val network = safeConnectivity(wifiOnly) { host.connectivity(wifiOnly) }
if (network == Connectivity.Ok) return
Expand Down Expand Up @@ -542,19 +515,17 @@ internal class EntryRun(
suspend fun backoffOrRelease(
policy: RetryClassifier.Policy,
streak: Int,
expiresAt: Long,
cause: String,
partStart: Long? = null,
) {
val backoff = RetryClassifier.backoffMs(policy, streak)
val now = host.now()
if (backoff <= RetryClassifier.IN_WORKER_BACKOFF_MAX_MS) {
val wait = min(backoff, max(0L, expiresAt - now))
ops.backingOff(entryId, generation, now + wait, streak, cause, partStart)
host.sleep(wait)
ops.backingOff(entryId, generation, now + backoff, streak, cause, partStart)
host.sleep(backoff)
return
}
throw BackoffException(RetryClassifier.nextAttemptAt(now, backoff, expiresAt), streak, cause, partStart)
throw BackoffException(now + backoff, streak, cause, partStart)
}

fun progressStarted(total: Long, sent: Long) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,6 @@ object LegacyImport {
attempts = 0,
bytesSent = 0,
totalBytes = 0,
expiresAt = at,
createdAt = at,
updatedAt = at,
generation = 1,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -636,8 +636,9 @@ class QueueController(
* orphans from a cancel race. Ack them.
* 4. A completed or cancelled entry whose own record is gone: the ack
* landed but the forget did not. Forget it.
* 5. Schedule every queued entry, and the expiry wake of every parked one.
* A row that WorkManager holds back after a stop is replaced
* 5. Schedule every queued entry. A parked entry gets no row, and a wake
* an earlier version made for it is cancelled. A row that WorkManager
* holds back after a stop is replaced
* ([WorkScheduler.schedule]), so each queued entry can start now.
*
* 6. Delete the orphan directories that hold only copied bodies and are
Expand Down Expand Up @@ -708,7 +709,7 @@ class QueueController(
}
if (cur !== e) changed += cur
}
if (!paused || cur.state == EntryState.AWAITING_AUTH) {
if (!paused) {
toSchedule += cur
found += e
}
Expand Down Expand Up @@ -770,9 +771,8 @@ class QueueController(

/**
* Runs [entry] when it is queued. A backoff longer than a worker waits goes
* to the wake; a parked entry gets a wake at its expiry, so it settles
* 'expired' on time, once that expiry is near (see [EXPIRY_WAKE_HORIZON_MS]).
* Any other wake of the entry is stale, so it is cancelled.
* to the wake. A parked entry gets no row: updateHeaders() starts it. Any
* other wake of the entry is stale, so it is cancelled.
*
* With a [probe] (a repair: the boot sweep, resume(), a return to the
* foreground, a same-id enqueue, setWifiOnly(), a doorbell), an entry
Expand All @@ -799,12 +799,7 @@ class QueueController(
cancelStaleWake(entry.id)
}
}
EntryState.AWAITING_AUTH ->
if (expiryWakeIsNear(entry.expiresAt, now)) {
scheduler.scheduleWake(entry, entry.expiresAt, replace = !keepWake)
} else {
cancelStaleWake(entry.id)
}
EntryState.AWAITING_AUTH -> cancelStaleWake(entry.id)
else -> Unit
}
}
Expand Down Expand Up @@ -888,13 +883,13 @@ class QueueController(
* whose WorkManager calls failed. One WorkManager query. An entry with
* no long wait and no row that can start now is scheduled as a repair
* (its main row when the network fits, else its doorbell); an entry with
* a long wait and no wake gets its wake (KEEP); a parked entry gets its
* expiry wake. It does not end a backoff wait. When it fails too, it
* a long wait and no wake gets its wake (KEEP). A parked entry needs no
* row. It does not end a backoff wait. When it fails too, it
* asks for itself again, later each time, up to [REPAIR_MAX_MS].
*/
fun repairRows() {
val now = clock()
val entries = store.all().filter { !it.legacy && (it.state == EntryState.QUEUED || it.state == EntryState.AWAITING_AUTH) }
val entries = store.all().filter { !it.legacy && it.state == EntryState.QUEUED }
val pass = SchedulePass("repair")
if (entries.isNotEmpty()) {
val rows = try {
Expand All @@ -909,8 +904,7 @@ class QueueController(
val at = e.nextAttemptAt
val longWait = at != null && at - now > RetryClassifier.IN_WORKER_BACKOFF_MAX_MS
when {
e.state == EntryState.AWAITING_AUTH || longWait ->
if (e.id !in rows.withWake) pass.schedule { scheduleRun(e, keepWake = true) }
longWait -> if (e.id !in rows.withWake) pass.schedule { scheduleRun(e, keepWake = true) }
e.id !in rows.startable -> pass.schedule { scheduleRun(e, probe = probe) }
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,15 @@ data class QueueEntry(
val attempts: Int,
val bytesSent: Long,
val totalBytes: Long,
val expiresAt: Long,
/**
* Not read. This version never expires an entry. 11.0.2 to 11.3.0 read
* this field as the time to settle the entry 'expired', and read a
* missing one as 0, which is already past. So a new or re-queued entry
* stores [NO_DEADLINE], and an app that goes back to such a version does
* not expire it. An entry from an earlier version that is not queued
* again keeps the value it had.
*/
val expiresAt: Long = NO_DEADLINE,
val createdAt: Long,
val updatedAt: Long,
/** Set while the entry waits out a backoff: queued for a long one, running for a short one. */
Expand Down Expand Up @@ -153,6 +161,11 @@ data class QueueEntry(
*/
val backoffPartStart: Long? = null,
) {
companion object {
/** The [expiresAt] a new or re-queued entry stores. 11.0.2 to 11.3.0 never reach it, and do no sum with it that overflows. */
const val NO_DEADLINE = Long.MAX_VALUE
}

val isLive get() = state.isLive
val isSettled get() = !state.isLive

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -150,8 +150,6 @@ object RetryClassifier {
*/
fun failureKind(verdict: Verdict): String = (verdict as? Verdict.Terminal)?.errorKind ?: "network"

fun isExpired(now: Long, expiresAt: Long) = now >= expiresAt

/**
* base * 2^(streak-1), capped at maxMs, then spread by ± jitter and capped
* again. streak 1 is baseMs.
Expand All @@ -163,7 +161,4 @@ object RetryClassifier {
val spread = raw * (1.0 + jitter * (2.0 * random.nextDouble() - 1.0))
return min(spread, policy.maxMs.toDouble()).toLong().coerceAtLeast(0L)
}

/** The wake time, never later than expiresAt, so an entry expires on time. */
fun nextAttemptAt(now: Long, backoffMs: Long, expiresAt: Long): Long = min(now + backoffMs, expiresAt)
}
20 changes: 3 additions & 17 deletions android/src/main/java/ai/openspace/backgroundupload/Scheduler.kt
Original file line number Diff line number Diff line change
Expand Up @@ -15,20 +15,6 @@ import androidx.work.workDataOf
import java.util.concurrent.TimeUnit
import kotlin.math.min

/**
* How far ahead a parked entry's expiry wake may be scheduled. A delayed
* WorkManager row holds one of the app's JobScheduler slots (20 by default)
* until it runs, and WorkManager's in-process scheduler reads only the 200
* oldest enqueued rows. Wakes years ahead, from long-lived requests that
* parked, filled both, so no other work started. An entry that expires past
* this horizon gets its wake from a later boot sweep, and settles at the
* first launch after it expires at the latest.
*/
const val EXPIRY_WAKE_HORIZON_MS = 24L * 60 * 60 * 1000

/** Whether an expiry wake at [expiresAt] is near enough to schedule now. */
fun expiryWakeIsNear(expiresAt: Long, now: Long): Boolean = expiresAt - now <= EXPIRY_WAKE_HORIZON_MS

/**
* When WorkManager 2.12.0 starts a row again after a system stop. Read from
* the 2.12.0 classes: a stop sets the row back to ENQUEUED and keeps its
Expand Down Expand Up @@ -141,9 +127,9 @@ interface WorkScheduler {
fun scheduleAndWait(entry: QueueEntry) = schedule(entry)

/**
* A run at [at] under a second unique name. Used for long backoffs and for
* the expiry of a parked entry. [replace] false keeps a wake that already
* exists (the boot sweep).
* A run at [at] under a second unique name. Used for long backoffs and
* after a stop. [replace] false keeps a wake that already exists (the boot
* sweep).
*/
fun scheduleWake(entry: QueueEntry, at: Long, replace: Boolean)

Expand Down
Loading
Loading