Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@ its row is forgotten (the ack of a `completed` or `cancelled` outcome, or
body. Before, the end of the 10 s wait deleted every key of the waited
attempt, including keys of tasks the library had cancelled (superseded or
paused). Now such a key stays until its task's cancel callback removes it,
or until the row is gone and the next launch prunes it.
or until a later launch prunes it. A launch prunes a key whose task is not
running when the row is gone, or when the id now belongs to a newer request
(see the next fix).

On iOS, a simple request's outcome now carries the `requestId` of the
attempt whose result settled it, so its response and `requestId` belong to
Expand All @@ -34,6 +36,32 @@ this. On Android a simple request's last attempt is always the one that
settled it; a chunked request's `requestId` is the last part attempt that
started. Android and iOS chunked parts are otherwise unchanged.

Fixed (iOS): a late result from an earlier use of a request id could change
a later request with the same id. An app may `mutate()` an id again after
its row is gone (the outcome was acknowledged, or the row was cancelled). The
new request starts again at attempt 1, so a result that arrived late from
the earlier request could match it. A late 2xx then completed the new
request with the old response and cancelled its task, so the new body was
not sent. A late 4xx failed it, and a late 5xx counted as one of its failed
attempts. Now each use of an id has its own value (the entry's
`incarnation`, a random value made when the entry is created or its body is
replaced). Every simple upload task carries it, and a result, progress
event or relaunch match from another use is ignored. iOS chunked parts
already carried it.

A task started by 11.3.0 or earlier carries no value. A request created
before the upgrade keeps the old behavior until it settles: it accepts any
task with no value that has its id, generation and attempt. So its own task
in flight during the upgrade is not sent again. A late result from an even
earlier use of the id, also started by the old version, can still change
it, as before. A request created after the upgrade ignores every task with
no value: such a task can only come from an earlier use.

Also on iOS: when the 10 s relaunch wait gave up on a task and the next
attempt could not be saved, the next launch cancelled that task if it was
still running and sent the request again. It now keeps the task, which may
already have sent the body.

## 11.3.0

Added `useRequests(filter?)` and `useRequestsWithProgress(filter?)`, React
Expand Down
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,6 +203,12 @@ id that exists:
- A completed entry whose outcome is not yet acknowledged: the journaled
outcome is emitted again. The request does not run again.

When the row is gone, the same id starts a new request. The ack of a
`completed` or `cancelled` outcome removes the row, and so does `cancel()`
on a settled row. The ack of an `error` outcome keeps it. On iOS a result, progress event or
cancel that arrives late from the earlier request is ignored: it never
completes, fails or retries the new one.

### Silent uploads (Android)

`android: { noNotification: true }` runs the request without a progress
Expand Down
16 changes: 11 additions & 5 deletions ios/ChunkedEngine.swift
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,14 @@ enum ChunkedEngine {
var inc: String?
}

/// `inc` is the entry's incarnation: one value per use of the id (see
/// QueueEntry.incarnation). nil in a task from a build before it. An
/// older build decodes a description with `inc` and ignores the field.
private struct RequestRef: Codable {
let id: String
let gen: Int
let att: Int
var inc: String?
}

/// A chunked part task: (id, part index, incarnation).
Expand All @@ -62,14 +66,16 @@ enum ChunkedEngine {
return (ref.id, ref.part, ref.inc)
}

/// A simple attempt: (id, entry generation, attempt ordinal).
static func taskDescription(id: String, attempt: Int, generation: Int) -> String {
requestPrefix + encode(RequestRef(id: id, gen: generation, att: attempt))
/// A simple attempt: (id, entry generation, attempt ordinal, the
/// entry's incarnation).
static func taskDescription(id: String, attempt: Int, generation: Int, incarnation: String) -> String {
requestPrefix + encode(RequestRef(id: id, gen: generation, att: attempt, inc: incarnation))
}

static func parseRequestDescription(_ description: String?) -> (id: String, generation: Int, attempt: Int)? {
static func parseRequestDescription(_ description: String?)
-> (id: String, generation: Int, attempt: Int, incarnation: String?)? {
guard let ref: RequestRef = decode(description, requestPrefix) else { return nil }
return (ref.id, ref.gen, ref.att)
return (ref.id, ref.gen, ref.att, ref.inc)
}

private static func encode<T: Encodable>(_ value: T) -> String {
Expand Down
52 changes: 39 additions & 13 deletions ios/QueueCoordinator+Reconcile.swift
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,18 @@ extension QueueCoordinator {
let liveKeys = Set(tasks.filter(\.isLive).map(\.key))

for task in tasks where task.isLive {
let meta = taskMap.meta(forKey: task.key)
var meta = taskMap.meta(forKey: task.key)
let owner = TaskOwner.resolve(description: task.taskDescription, meta: meta)
let entry = owner.flatMap { index.entry($0.id) }
// A late-result key whose task is live and is the entry's current
// attempt. The wait gave up on this attempt, but the re-issue after it
// was never saved (a failed write-ahead), so the entry is still on this
// attempt. Adopt the task: it may have sent the body already.
if meta?.purpose == .lateResult, case .request(let id, let generation, let attempt, let incarnation)? = owner,
let e = entry, isCurrentSimpleTask(e, generation: generation, attempt: attempt, incarnation: incarnation) {
taskMap.setPurpose(.attempt, forKey: task.key, id: id)
meta?.purpose = .attempt
}
// A task the library replaced or dropped, still live because the
// process died before the daemon applied the cancel. Never adopt it:
// its replacement, or the re-issue below, drives the entry. A kept
Expand All @@ -120,27 +131,33 @@ extension QueueCoordinator {
task.cancel()
continue
}
let owner = TaskOwner.resolve(description: task.taskDescription, meta: meta)
let entry = owner.flatMap { index.entry($0.id) }
let runnable = entry.map { !$0.legacy && ($0.state == .queued || $0.state == .running) } ?? false
switch owner {
case .part(let id, let part, let incarnation)? where runnable && entry?.isChunked == true:
partTasks[id, default: []].append(.init(task: task, part: part, incarnation: incarnation))
case .request(let id, let generation, let attempt)?
where runnable && entry?.isChunked == false && entry?.generation == generation
&& entry?.attempts == attempt && !simpleLive.contains(id):
case .request(let id, let generation, let attempt, let incarnation)?
where entry.map { isCurrentSimpleTask($0, generation: generation, attempt: attempt, incarnation: incarnation) }
== true && !simpleLive.contains(id):
simpleLive.insert(id)
liveTasks[task.key] = (id, task)
default:
// No v10 owner (a v9 task), an older generation or attempt, a
// duplicate, or an entry that must have no task. Drop its callback.
// No v10 owner (a v9 task), an older generation or attempt, an
// earlier use of the id, a duplicate, or an entry that must have no
// task. Drop its callback.
taskMap.setPurpose(.superseded, forKey: task.key, id: owner?.id ?? "")
task.cancel()
}
}
// A key whose id has no entry belongs to nothing. A live task keeps its
// key until its cancel callback, which reads the purpose.
taskMap.removeAll { key, meta in !liveKeys.contains(key) && index.entry(meta.id) == nil }
// A key whose id has no entry belongs to nothing, and a simple key of an
// earlier use of the id belongs to nothing now. A live task keeps its key
// until its cancel callback, which reads the purpose. A task listed as
// finished may still report: its description names its use, so the
// report is dropped without the key.
taskMap.removeAll { key, meta in
guard !liveKeys.contains(key) else { return false }
guard let e = index.entry(meta.id) else { return true }
return meta.partIndex == nil && meta.generation != nil && !e.ownsSimpleTask(incarnation: meta.incarnation)
}

for e in index.entries() where e.isLive && !e.legacy {
armExpiry(e)
Expand Down Expand Up @@ -174,7 +191,8 @@ extension QueueCoordinator {
/// completion is handled.
private func withoutTask(_ e: QueueEntry) {
let pending = taskMap.keys(where: {
$0.id == e.id && $0.generation == e.generation && $0.attempt == e.attempts
$0.id == e.id && e.ownsSimpleTask(incarnation: $0.incarnation)
&& $0.generation == e.generation && $0.attempt == e.attempts
})
guard !pending.isEmpty else {
// No task was ever made for a waiting attempt: it never ran, so it
Expand All @@ -196,7 +214,8 @@ extension QueueCoordinator {
// already marked superseded or pause keeps its purpose: its task may
// still owe a cancel callback.
let stale = self.taskMap.keys(where: {
$0.id == e.id && $0.generation == e.generation && $0.attempt == e.attempts
$0.id == e.id && e.ownsSimpleTask(incarnation: $0.incarnation)
&& $0.generation == e.generation && $0.attempt == e.attempts
&& ($0.purpose ?? .attempt) == .attempt
})
self.taskMap.setPurpose(.lateResult, for: stale.map { (key: $0, id: e.id) })
Expand All @@ -205,6 +224,13 @@ extension QueueCoordinator {
}
}

/// Whether a simple task with this identity is the current attempt of `e`,
/// a simple entry that may have a task.
private func isCurrentSimpleTask(_ e: QueueEntry, generation: Int, attempt: Int, incarnation: String?) -> Bool {
!e.legacy && !e.isChunked && (e.state == .queued || e.state == .running)
&& e.ownsSimpleTask(incarnation: incarnation) && e.generation == generation && e.attempts == attempt
}

/// Issues again, keeping what is left of a wait.
private func reissue(_ e: QueueEntry, advanceAttempt: Bool) {
var n = e
Expand Down
27 changes: 17 additions & 10 deletions ios/QueueCoordinator+Simple.swift
Original file line number Diff line number Diff line change
Expand Up @@ -85,15 +85,16 @@ extension QueueCoordinator {
}

let meta = TaskMap.Meta(
id: id, attempt: e.attempts, requestId: requestId,
id: id, incarnation: e.incarnation, attempt: e.attempts, requestId: requestId,
headerGeneration: settings.headerGeneration, generation: e.generation, purpose: .attempt,
foreground: activity.isActive)
let task: UploadTask
do {
task = try transport.upload(
buildRequest(e, url: url, requestId: requestId), fromFile: body,
wifiOnly: settings.wifiOnly(e.wifiOnly),
description: ChunkedEngine.taskDescription(id: id, attempt: e.attempts, generation: e.generation),
description: ChunkedEngine.taskDescription(
id: id, attempt: e.attempts, generation: e.generation, incarnation: e.incarnation),
beginAt: beginAt.map { Date(timeIntervalSince1970: $0 / 1000) })
} catch {
// The session could not open the body. Gone since the check above: it
Expand Down Expand Up @@ -169,9 +170,12 @@ extension QueueCoordinator {
return
}
// Only the current attempt of the current generation may drive the
// entry, or an older one the library replaced (see below).
guard case .request(let id, let generation, let attempt) = owner,
var e = index.entry(id), !e.legacy, !e.isChunked, e.generation == generation,
// entry, or an older one the library replaced (see below). The task must
// belong to this use of the id: an earlier use (acked, cancelled, or
// replaced, then used again) also starts at generation 1, attempt 1.
guard case .request(let id, let generation, let attempt, let incarnation) = owner,
var e = index.entry(id), !e.legacy, !e.isChunked, e.ownsSimpleTask(incarnation: incarnation),
e.generation == generation,
e.attempts == attempt || (meta?.purpose?.isOutdated == true && attempt < e.attempts) else { return }

let cancelled = RetryClassifier.isCancellation(c.error)
Expand Down Expand Up @@ -275,8 +279,9 @@ extension QueueCoordinator {
// one the relaunch grace gave up on. Cancel it: its key becomes
// superseded below, because it now owes a cancel callback.
guard meta?.purpose?.isOutdated != true,
case .request(let id, let generation, let attempt) = owner,
var e = index.entry(id), !e.isChunked, e.generation == generation,
case .request(let id, let generation, let attempt, let incarnation) = owner,
var e = index.entry(id), !e.isChunked, e.ownsSimpleTask(incarnation: incarnation),
e.generation == generation,
e.attempts == attempt, e.state == .queued || e.state == .running, !settings.isPaused(e.key),
let url = e.url.flatMap(URL.init(string:)) else {
taskMap.setPurpose(.superseded, forKey: key, id: owner.id)
Expand Down Expand Up @@ -304,12 +309,14 @@ extension QueueCoordinator {
switch owner {
case .part(let id, let part, let incarnation):
self.chunked.partProgress(id: id, part: part, incarnation: incarnation, key: key, sent: sent)
case .request(let id, let generation, let attempt):
case .request(let id, let generation, let attempt, let incarnation):
// A replacement shares its attempt with the task it replaced, so
// only the task this process holds may report. Before reconcile no
// task is held yet, and none was replaced.
// task is held yet, and none was replaced: then the task's own
// identity, use of the id included, must match.
guard !self.ready || self.liveTasks[key] != nil,
var e = self.index.entry(id), e.generation == generation, e.attempts == attempt,
var e = self.index.entry(id), e.ownsSimpleTask(incarnation: incarnation),
e.generation == generation, e.attempts == attempt,
e.state == .queued || e.state == .running else { return }
if e.state == .queued {
e.state = .running
Expand Down
12 changes: 8 additions & 4 deletions ios/QueueCoordinator.swift
Original file line number Diff line number Diff line change
Expand Up @@ -460,8 +460,8 @@ final class QueueCoordinator {
switch resolved {
case .part(let id, let part, let incarnation):
parts[id, default: []].append(.init(part: part, incarnation: incarnation, key: key))
case .request(let id, let generation, let attempt):
replaceRequestTask(id, generation: generation, attempt: attempt, key: key)
case .request(let id, let generation, let attempt, let incarnation):
replaceRequestTask(id, generation: generation, attempt: attempt, incarnation: incarnation, key: key)
}
}
for (id, replaced) in parts {
Expand All @@ -472,8 +472,12 @@ final class QueueCoordinator {
/// A simple entry's current attempt, sent again from a new task. The
/// attempt never sent its body, so it keeps its ordinal and request id.
/// An empty body is left alone: 0 bytes sent cannot tell whether it went.
private func replaceRequestTask(_ id: String, generation: Int, attempt: Int, key: String) {
guard var e = index.entry(id), !e.legacy, !e.isChunked, e.generation == generation,
/// The task must belong to this use of the id, as in every other reader
/// of a task's owner. A forget already cancels and drops the held tasks,
/// so this check is a guard, not the only one.
private func replaceRequestTask(_ id: String, generation: Int, attempt: Int, incarnation: String?, key: String) {
guard var e = index.entry(id), !e.legacy, !e.isChunked, e.ownsSimpleTask(incarnation: incarnation),
e.generation == generation,
e.attempts == attempt, e.state == .queued || e.state == .running, e.totalBytes > 0,
!settings.isPaused(e.key) else { return }
let remaining = e.nextAttemptAt.map { Int($0 - now()) }.flatMap { $0 > 0 ? $0 : nil }
Expand Down
30 changes: 27 additions & 3 deletions ios/QueueEntry.swift
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,13 @@ struct QueueEntry: Codable, Equatable {
/// The body identity for the same-id rules. See EnqueueParser.fingerprint.
var bodyFingerprint: String
var parts: [Part]
/// The parts-plan identity. Rotates when the body is replaced. Part tasks
/// carry it, so a late callback from a replaced plan is dropped.
/// The parts-plan identity, and the identity of this use of the id. A new
/// random value at create and when the body is replaced, so no two uses of
/// an id share one. Part tasks and simple tasks carry it, so a late
/// callback from a replaced plan or from an earlier use of the id is
/// dropped. A same-body reopen keeps it: the generation tells those apart.
/// A v9 chunked upload that create() resumes keeps its v9 value; its
/// manifest is removed then, so a later use gets a new one.
var incarnation: String
/// The merged request headers. updateHeaders() patches them.
var headers: [String: String]
Expand Down Expand Up @@ -90,6 +95,12 @@ struct QueueEntry: Codable, Equatable {
var legacy: Bool
let createdAt: Double
var updatedAt: Double
/// true when this use of the id started in a build that writes
/// `incarnation` into every simple task. Then a simple task with no
/// incarnation comes from an earlier use. nil in an entry that an older
/// build created or saved: its tasks may carry no incarnation. An older
/// build ignores the field and drops it at its next save of the entry.
var tasksCarryIncarnation: Bool? = nil
}

extension QueueEntry {
Expand All @@ -101,6 +112,17 @@ extension QueueEntry {
}

var isSettled: Bool { !isLive }

/// Whether a simple task that carries `incarnation` (nil: none) belongs
/// to this use of the id. A task with a value must carry this entry's. A
/// task with none is from a build before the value. It matches only an
/// entry such a build created or saved, as every task did before, so a
/// task in flight across the upgrade still drives its entry and is not
/// sent again.
func ownsSimpleTask(incarnation task: String?) -> Bool {
guard let task else { return tasksCarryIncarnation != true }
return task == incarnation
}
var isChunked: Bool { bodyKind == .parts }

var acceptedBytes: Int64 { parts.filter(\.accepted).reduce(0) { $0 + $1.size } }
Expand Down Expand Up @@ -158,7 +180,7 @@ extension QueueEntry {
/// Rule 2: a new entry.
static func created(from p: ParsedEnqueue, staged: StagedBody, headerGeneration: Int,
paused: Bool, now: Double, createdAt: Double? = nil) -> QueueEntry {
QueueEntry(
var e = QueueEntry(
id: p.id, key: p.key, varsJSON: p.varsJSON, url: p.url, method: p.method, accept: p.accept, retry: p.retry,
wifiOnly: p.wifiOnly, bodyKind: staged.kind, bodyPath: staged.relativePath,
bodyContentType: staged.contentType, forceContentType: staged.forceContentType,
Expand All @@ -168,6 +190,8 @@ extension QueueEntry {
bytesSent: 0, totalBytes: staged.totalBytes, expiresAt: p.expiresAt,
nextAttemptAt: nil, settledEventId: nil, lastRequestId: nil, lastUrl: nil,
lastPartIndex: nil, legacy: false, createdAt: createdAt ?? now, updatedAt: now)
e.tasksCarryIncarnation = true
return e
}

/// Rule 3, same body: the new vars, headers, expiresAt and descriptor
Expand Down
4 changes: 2 additions & 2 deletions ios/Tests/CoordinatorLateResultTests.swift
Original file line number Diff line number Diff line change
Expand Up @@ -310,8 +310,8 @@ final class CoordinatorLateResultTests: XCTestCase {
/// a new attempt is added: the kept key has the entry's current attempt.
/// It still goes, so a later use of the id cannot take its late result.
/// The later use is on attempt 2 here. On attempt 1 the task description
/// alone matches it, with or without a key: that needs a per-use value,
/// which this change does not add.
/// alone would match it by id, generation and attempt; the incarnation in
/// the description drops it (CoordinatorPerUseTests).
func testAnEntryThatSettlesInsideTheGraceReissueDropsItsKeptKey() throws {
h.boot()
_ = try h.enqueue(h.dataRaw(id: "a", extra: ["expiresAt": h.clock + 60_000])).get()
Expand Down
Loading
Loading