Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 107 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,113 @@ their own failures from 0. Before, every part started at the stored
backoff, so one failed part after a long backoff sent the whole entry back
for the next, longer wait.

**Behavior change:** on Android 11 and later (API 30), an upload that finds
no usable network no longer polls inside a running job until the network
returns. A run with no foreground mode checks every 10 s for at most 60 s.
Then the entry goes back to `queued` with no WorkManager row of its own,
and waits on a doorbell: one row for the whole queue that waits for a
validated network, and a second one that waits for a validated Wi-Fi
network, for the entries that Wi-Fi only applies to. When Android starts a
doorbell, it sends nothing: it starts each waiting entry whose network
fits on an ordinary row, and keeps a doorbell for the others. No row that
runs an upload has a network rule, so WorkManager never stops an upload
because the network rule is lost. However many entries wait, at most two
rows sleep for them. The hand-over is not a failure: `attempts`, the
backoff step and the last failure's cause stay as they were, and it sets
no `nextAttemptAt`. A run hands over only while none of its requests is on
its way: a chunked part that waits for the network waits for a sibling
part to finish sending, and once it hands over, no part starts a request.
Before, the job used the app's job quota while it waited (in the `rare`
standby bucket, 10 minutes in 24 hours) and Android stopped it at its time
limit. Android 10 (API 29) keeps the poll and has no doorbell.

In an app that targets API 35, on Android 15 and later, a run that holds
foreground mode polls for at most 30 minutes, then hands over the same
way: Android documents the `dataSync` limit of 6 hours in 24 for apps that
target Android 15, and each minute of the poll uses it. In other apps, and
below Android 15, such a run polls as before.

A ring that the library's own check refuses (WorkManager sees the
network, the library does not) is followed by the next ring at least 30 s
later. Two bounded exceptions can add one ring with no delay: a hand-over
that lands while such a ring runs, and a repair that replaces a doorbell
WorkManager holds back. Each attempt now checks the network again once it has a
transmit slot, just before it is written ahead. When the check fails, the
request leaves the slot with no attempt counted and waits for the network
again. Before, a Wi-Fi-only request that waited for a slot could go out
over another network.

Known limits:

- An app start with no network starts no run, so no "waiting for the
network" notification shows until a run starts.
- On API 31 and later a run that a doorbell starts in the background
cannot take foreground mode, so a single body (`data`, `form`, `file`)
that needs more than the 10-minute job limit cannot finish until the app
is opened. Use `parts` for large bodies.
- WorkManager hands JobScheduler at most 20 rows, the oldest first. With
20 or more older sleeping rows (backoff or expiry wakes), the doorbell is
not handed to JobScheduler until some of them have run, while the app
process is not alive. That wait is bounded by the backoff maximum and
the 24-hour expiry-wake horizon. The app's start, `resume()` and return
to the foreground start the entry sooner when its network fits.

**Behavior change:** on Android 11 and later, the boot sweep, `resume()`,
a return to the foreground, a same-id `mutate()` of a queued entry and
`setWifiOnly()` start a queued entry with no backoff wait only when the
network fits it now. Otherwise they add no row and make sure its doorbell
exists. So an app start with no network starts no run. A new entry, and an
entry that `updateHeaders()` takes out of `awaiting-auth`, start a run as
before. Backoff wakes and a parked entry's expiry wake have no network
rule, as before. An entry whose `expiresAt` passes while it waits for a
doorbell settles `expired` when its next run starts: when the network
returns, or at the next start, resume or foreground with a network.

Android: `setWifiOnly(false)` now moves the queued entries that follow the
setting from the Wi-Fi rule to any network: each one with no long backoff
wait starts at once when a network fits, else it waits on the network
doorbell. The same value, or `setWifiOnly(true)`, schedules
nothing: a run that starts on another network hands over to the Wi-Fi
doorbell. A run that hands over uses the setting at that moment.

Android: a Wi-Fi-only upload now also waits while the app binds its
process to a network that is not Wi-Fi (`bindProcessToNetwork`), because
its sockets would use that network. The network check reads the system
default network, which a bind does not change. Such a run keeps waiting
and does not hand over, because the Wi-Fi doorbell would start it again at
once. A network check that throws now reads as no network, so the upload
waits. Before, the error settled the upload as failed.

Android: the boot record (`takeDiagnostics()`, message `boot sweep`) now
has `wifiOnly`, the queue's setting; `entriesByWake`
(`{ backoff, network, wifi }`), the queued entries a backoff wake, the
network doorbell or the Wi-Fi doorbell starts; and `doorbells`
(`{ network, wifi }`), whether each doorbell row exists. Both describe the
state the start found, before the boot sweep changed it, and are left out
when WorkManager could not be read. `entriesWithoutRun` now also counts the
entries that wait for a doorbell, so by itself it no longer means an entry
is stuck: compare it with `entriesByWake`. The `Diagnostic` type has the
three new fields.

Android: a WorkManager call that fails or times out no longer makes
`mutate()`, `resume()` or `updateHeaders()` reject, and no longer stops
the boot sweep. WorkManager queries and awaited inserts now time out after
10 s. A pass stops scheduling at the first timeout, because each further
call would hold the queue thread (which also serves every JS call) for
10 s more. It writes one diagnostics record and goes on with the rest of
its work: `mutate()` resolves once the entry is saved, and the boot sweep
still writes its boot record. Then the library itself runs a repair on its
queue thread, 30 s later at first, and later each time it fails again (up
to 5 minutes): it starts every queued entry that has no row that can start
now (its main row when the network fits, else its doorbell) and makes the
wakes that are missing. No app call is needed while the process lives:
the repair timer is in memory, so if the process ends first, the next app
start (the boot sweep) or a doorbell ring starts the entry. While that
repair is pending, a new pass makes no WorkManager call and leaves its
entries to the repair, so each `mutate()` does not wait 10 s again. A
doorbell ring that meets a timeout stops, writes one record and rings
again later.

## 11.3.0

Added `useRequests(filter?)` and `useRequestsWithProgress(filter?)`, React
Expand Down
34 changes: 31 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,23 @@ at the next run, after a growing backoff. A body that needs more than
10 minutes headless cannot finish that way. Use `parts` for large bodies:
accepted parts are kept across runs. iOS has no equal limit.

**No network.** On API 30 and later a run that finds no usable network
checks every 10 s for at most 60 s; with foreground mode, in an app that
targets API 35 on Android 15 and later, for at most 30 minutes. Then the entry goes back to `queued` with no
row of its own and waits on a doorbell: one WorkManager row for the whole
queue that waits for a validated network, and one that waits for a
validated Wi-Fi network. When Android starts a doorbell, it starts the
waiting entries whose network fits. The doorbell sends nothing, and no row
that sends has a network rule. The app's start, `resume()` and return to
the foreground also start a waiting entry when its network fits, and
otherwise make sure its doorbell exists. This is not a failed attempt and
adds no backoff. A Wi-Fi-only upload also waits while the app binds its
process to a network that is not Wi-Fi, and every attempt checks the
network again once it has a transmit slot. API 29, and any other
foreground run, keep the 10 s check for as long as the run lasts. A run that a
doorbell starts in the background cannot take foreground mode on API 31
and later, so the headless time limit above applies to it.

**Backoff and the app.** When the app starts, calls `resume()`, or comes
to the foreground (every Activity resume, at most once every 30 s), the
library ends the retry backoff of a queued entry and runs it at once, when
Expand Down Expand Up @@ -519,8 +536,10 @@ again.
The queue's Wi-Fi setting. Persisted natively. Applies to queued and future
entries whose descriptor does not set `wifiOnly`. An entry that sets it keeps
its own value. Both are checked before each attempt, so a toggle moves the
queued entries that follow the setting. A call with the current value
resolves without a write.
queued entries that follow the setting. On Android, turning it off also
starts at once the queued entries that follow it and have no backoff wait,
when a network fits. A call with the current value resolves without a
write.

It does not reject. The new value applies at once, also when the disk is
full. See [Settings on a full disk](#settings-on-a-full-disk).
Expand Down Expand Up @@ -774,7 +793,16 @@ entry that was running when its process died counts as running. It also
describes every unfinished WorkManager run the library had before the start
changed them: the count by state, how many are delayed wakes, how many of
those are due more than 24 hours out, the highest run attempt count, and how
many queued or running entries had no run at all.
many queued or running entries had no run at all. `entriesByWake`
(`{ backoff, network, wifi }`) counts the queued entries by what starts
them: a backoff wake, the network doorbell, or the Wi-Fi doorbell.
`doorbells` (`{ network, wifi }`) says whether each doorbell row exists.
Both describe the state before the boot sweep changed it, and are left out
when WorkManager could not be read. `entriesWithoutRun` also counts the
entries that wait for a doorbell, so by itself it does not mean an entry
is stuck. `wifiOnly` is the
queue's Wi-Fi setting. Together they tell "waits for the network or for
Wi-Fi" from "waits for a backoff".

On iOS the boot record has only `v9Leftovers` and `orphans`. The other iOS
records are the warnings of the count and of the orphan cleanup, and a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,16 @@ package ai.openspace.backgroundupload
* What the boot sweep found, as one diagnostic record: the entries it
* schedules, and every unfinished WorkManager run of the library before the
* sweep changes them. Stale wakes show as delayed runs far in the future.
* A queued or running entry with no run at all shows in entriesWithoutRun.
* A queued or running entry with no run at all shows in entriesWithoutRun;
* that includes an entry that waits for a doorbell, so by itself it does
* not mean a stuck entry.
* v9Leftovers counts the v9 work still on disk, and orphans the
* directories with no entry file that are not v9: what the boot cleanup
* left, and what it deleted. Zeros included.
* left, and what it deleted. Zeros included. wifiOnly is the queue's
* setting. entriesByWake counts the queued entries by what starts them:
* their backoff wake, or the network or Wi-Fi doorbell; doorbells says
* which doorbell rows exist. Both describe the rows and entries before the
* sweep changed them.
*/
internal object BootSnapshot {
const val MESSAGE = "boot sweep"
Expand All @@ -22,6 +28,7 @@ internal object BootSnapshot {
runs: List<PendingRun>?,
now: Long,
leftovers: Leftovers,
settings: QueueSettings,
): Map<String, Any?> {
val base = mapOf(
"entries" to entries.size,
Expand All @@ -30,6 +37,7 @@ internal object BootSnapshot {
"oldestUpdatedAt" to entries.minOfOrNull { it.updatedAt },
"v9Leftovers" to leftovers.v9.toMap(),
"orphans" to leftovers.orphans.toMap(),
"wifiOnly" to settings.wifiOnly,
)
if (runs == null) return base + ("runsUnavailable" to true)
val withRun = runs.mapNotNull { it.entryId }.toSet()
Expand All @@ -44,6 +52,37 @@ internal object BootSnapshot {
run.delayed && run.nextScheduleAt != null && run.nextScheduleAt - now > EXPIRY_WAKE_HORIZON_MS
},
"maxRunAttemptCount" to runs.maxOfOrNull { it.runAttemptCount },
"entriesByWake" to entriesByWake(entries, runs, now, settings),
"doorbells" to mapOf(
"network" to runs.any { it.doorbell == RequiredNetwork.CONNECTED },
"wifi" to runs.any { it.doorbell == RequiredNetwork.WIFI },
),
)
}

/**
* Each queued entry once. "backoff": it has a delayed wake (a backoff
* wait). "network" and "wifi": it has no run of its own and no long
* wait, so the doorbell of that rule starts it ([requiredNetwork]). An
* entry with a main-chain row is not counted.
*/
private fun entriesByWake(entries: List<QueueEntry>, runs: List<PendingRun>, now: Long, settings: QueueSettings): Map<String, Int> {
val byEntry = runs.filter { it.entryId != null }.groupBy { it.entryId }
val counts = linkedMapOf("backoff" to 0, "network" to 0, "wifi" to 0)
for (e in entries) {
if (e.state != EntryState.QUEUED) continue
val own = byEntry[e.id].orEmpty()
val kind = when {
own.any { it.delayed } -> "backoff"
own.isNotEmpty() -> continue
else -> when (requiredNetwork(e, settings, now)) {
RequiredNetwork.CONNECTED -> "network"
RequiredNetwork.WIFI -> "wifi"
RequiredNetwork.NONE -> continue
}
}
counts[kind] = counts.getValue(kind) + 1
}
return counts
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
package ai.openspace.backgroundupload

import android.content.Context
import androidx.work.CoroutineWorker
import androidx.work.WorkerParameters
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext

/**
* The worker of a doorbell row ([WorkScheduler.ensureDoorbell]). WorkManager
* runs it when the row's network is up. It sends nothing: on the queue
* executor it starts every queued entry that waits for a network that fits
* now ([QueueController.answerDoorbell]), and makes sure a doorbell stays for
* the entries whose network does not fit yet. It also runs with no JS and
* no module, so it builds its own controller over the shared stores.
*
* Returns retry when a WorkManager call failed or anything threw, so the
* row runs again ([DoorbellRing]); it never fails.
*/
class DoorbellWorker(private val context: Context, params: WorkerParameters) : CoroutineWorker(context, params) {

override suspend fun doWork(): Result = withContext(Dispatchers.IO) {
val rung = inputData.getString(WorkManagerScheduler.DOORBELL_NETWORK_KEY)
?.let { name -> RequiredNetwork.values().firstOrNull { it.name == name } }
val answered = DoorbellRing.answered {
// A headless run has no module, so it opens the log that its Diag calls write to.
DiagnosticLog.get(context)
val controller = QueueController(
QueueStore.get(context),
EventJournal.get(context),
QueueSettingsStore.get(context),
EventReporter,
WorkManagerScheduler(context),
networkFits = { wifiOnly -> networkFitsNow(context, wifiOnly) },
)
// Every queue change runs on this one thread; a blocking wait here is off the main thread.
UploaderModule.queueExecutor.submit<Boolean> { controller.answerDoorbell(rung) }.get()
}
if (answered) Result.success() else Result.retry()
}
}

/**
* The doorbell's result, from its answer. Any error is a retry, never a
* failure: WorkManager fails a FAILED row's successors without a run, so a
* doorbell queued behind it would be lost. A cancellation (a stop) goes on.
*/
internal object DoorbellRing {
fun answered(answer: () -> Boolean): Boolean =
try {
answer()
} catch (stop: java.util.concurrent.CancellationException) {
throw stop
} catch (error: Throwable) {
Diag.error("doorbell: the answer failed; it rings again", error)
false
}
}
Loading
Loading