Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,26 @@
## Unreleased

Android now sets its HTTP timeouts: connect 15 s, read 60 s, write 60 s.
Before, only the 24 h call cap was set, so OkHttp's 10 s defaults applied.
A server that took more than 10 s to answer after the body was sent made the
phone give up while the server still stored the request. The phone then sent
it again, and the server could store it twice. The 24 h call cap is
unchanged.

Behavior change: the read timeout now allows 60 s of silence after the
body, before the attempt fails as a transport failure. A dead connection is
found after 60 s, not 10 s, and holds its transfer slot for that time. The
connect timeout is now 15 s per address, not 10 s.

Android now keeps a response it finished reading before a stop (a system
stop, or `pause()`) took effect. Before, the response was dropped and the
request was sent again (at once after a system stop, at `resume()` after a
pause). Now the response goes through the usual rules: an accepted response
settles the entry, and an accepted chunked part keeps its accepted flag. A
response that has not been read when the stop lands is still lost, and the
request is still sent again. `cancel()` is unchanged: a cancelled entry
still settles `cancelled`.

## 11.3.0

Added `useRequests(filter?)` and `useRequestsWithProgress(filter?)`, React
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ internal class ChunkedTransfer(private val run: EntryRun) {
)
when (val r = a.result) {
is EntryRun.AttemptResult.Accepted -> {
// No suspension before this save: the part may have landed as the
// run was stopped (see awaitLanded).
markAccepted(index, a.entry.descriptor!!.parts!![index])
return
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,11 @@ internal class EntryRun(
return Attempt(entry, url, policy, result)
}

// The response can arrive on a stopped job (see awaitLanded). From here to
// ops.settle in run(), or to markAccepted for a part, nothing may suspend:
// a suspension on a stopped job throws, and the response is lost. The
// "lands as the run is stopped" tests in EntryRunTest pin this.

// A part counts on its own, so each part gets a rule's full maxRetries.
val hits = (if (part != null) part.transientHitsByRule else entry.transientHitsByRule) ?: emptyMap()
var verdict = RetryClassifier.classifyResponse(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,32 @@ import java.util.concurrent.TimeUnit
// chunked workers share them. Thus "requests transmitting at one time" means
// one thing across all uploads.

// Max total time for a single request to complete
// This is 24hrs so plenty of time for large uploads
// Worst case is the time maxes out and the upload gets restarted.
// Not using unlimited time to prevent unexpected behaviors.
private const val REQUEST_TIMEOUT = 24L
private val REQUEST_TIMEOUT_UNIT = TimeUnit.HOURS
// OkHttp's defaults for the three timeouts below are 10 s each. 10 s of
// silence is too short: a server that takes longer to answer still stores
// the request, the phone sends it again, and the server can store it twice.

// The time to open a TCP connection to one address. OkHttp tries the next
// address of the host when one does not answer only after this time, so a
// host with several addresses that do not answer (for example a broken IPv6
// path) holds a transfer slot for that many times this value. Nothing was sent yet, so a connect timeout cannot make a duplicate.
// That is why it is shorter than the read and write timeouts.
private const val CONNECT_TIMEOUT_SECONDS = 15L

// The longest silence while reading. This includes the wait for the server's
// answer after the last body byte. 60 s matches iOS's default request timeout.
// A dead connection is found after this time, and holds its transfer slot
// until then.
private const val READ_TIMEOUT_SECONDS = 60L

// The longest a single write of the body may block.
private const val WRITE_TIMEOUT_SECONDS = 60L

// A cap on the whole call: DNS, connect, the body, the server's work, and the
// response. OkHttp counts any retries it makes on its own inside the same cap.
// It is not a silence limit; the read and write timeouts above are. 24 h lets
// a large body finish on a slow network and still ends a call that never
// finishes. When it expires, the attempt fails and the retry rules apply.
private const val CALL_TIMEOUT_HOURS = 24L

// The number of requests transmitting at one time across ALL uploads, chunked
// parts included. This is the design's library-wide cap of 4. Every request,
Expand Down Expand Up @@ -53,5 +73,8 @@ internal val transferGate = TransferGate(MAX_TRANSFER_CONCURRENCY, HIGH_PRIORITY

// Use Okhttp as it provides the most standard behaviors even though it's not coroutine friendly
internal val uploadHttpClient = OkHttpClient.Builder()
.callTimeout(REQUEST_TIMEOUT, REQUEST_TIMEOUT_UNIT)
.connectTimeout(CONNECT_TIMEOUT_SECONDS, TimeUnit.SECONDS)
.readTimeout(READ_TIMEOUT_SECONDS, TimeUnit.SECONDS)
.writeTimeout(WRITE_TIMEOUT_SECONDS, TimeUnit.SECONDS)
.callTimeout(CALL_TIMEOUT_HOURS, TimeUnit.HOURS)
.build()
103 changes: 75 additions & 28 deletions android/src/main/java/ai/openspace/backgroundupload/UploadUtils.kt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package ai.openspace.backgroundupload

import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.suspendCancellableCoroutine
import okhttp3.Call
import okhttp3.Callback
Expand All @@ -18,7 +19,7 @@ import okio.buffer
import java.io.File
import java.io.IOException
import java.io.RandomAccessFile
import kotlin.coroutines.resumeWithException
import java.util.concurrent.atomic.AtomicReference

// Throttling interval of the raw progress callback. ProgressThrottle limits
// the JS events above this.
Expand Down Expand Up @@ -97,38 +98,84 @@ fun rangeRequestBody(file: File, start: Long, end: Long): RequestBody = object :
}
}

private suspend fun awaitResponse(client: OkHttpClient, request: Request): UploadResponse =
suspendCancellableCoroutine { continuation ->
val call = client.newCall(request)
continuation.invokeOnCancellation { call.cancel() }
call.enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) =
continuation.resumeWithException(e)

override fun onResponse(call: Call, response: Response) {
val result = try {
response.use { res -> // close the response asap
// The body unchanged: an empty body stays empty. A substituted
// reason phrase would make accept `bodyIncludes` rules match text
// the server never sent. The cap applies while it streams in.
val body = res.body?.let {
BodyCap.read(it.source(), BodyCap.SETTLED_MAX_BYTES, it.contentType()?.charset() ?: Charsets.UTF_8)
private suspend fun awaitResponse(client: OkHttpClient, request: Request): UploadResponse {
val call = client.newCall(request)
return awaitLanded(
register = { onResult ->
call.enqueue(object : Callback {
override fun onFailure(call: Call, e: IOException) = onResult(Result.failure(e))

override fun onResponse(call: Call, response: Response) {
val result = try {
response.use { res -> // close the response asap
// The body unchanged: an empty body stays empty. A substituted
// reason phrase would make accept `bodyIncludes` rules match text
// the server never sent. The cap applies while it streams in.
val body = res.body?.let {
BodyCap.read(it.source(), BodyCap.SETTLED_MAX_BYTES, it.contentType()?.charset() ?: Charsets.UTF_8)
}
UploadResponse(
res.code,
body?.text.orEmpty(),
res.headers.toMultimap().mapValues { it.value.joinToString(", ") },
body?.truncated ?: false,
)
}
UploadResponse(
res.code,
body?.text.orEmpty(),
res.headers.toMultimap().mapValues { it.value.joinToString(", ") },
body?.truncated ?: false,
)
} catch (e: IOException) {
onResult(Result.failure(e))
return
}
} catch (e: IOException) {
continuation.resumeWithException(e)
return
onResult(Result.success(result))
}
continuation.resumeWith(Result.success(result))
})
},
cancel = { call.cancel() },
)
}

/**
* Waits for one callback result and keeps a result that landed before a stop.
*
* [register] starts the work and gets the callback for its result. [cancel]
* stops the work. A stop (WorkManager stops the worker, pause() or cancel()
* cancels its work) with no result cancels the work and throws.
*
* Why: a plain suspendCancellableCoroutine drops a value when the stop comes
* after the resume and before the coroutine runs again. Its docs say it "will
* not resume successfully, even if CancellableContinuation.resume was already
* invoked". For an upload, that throws away a response OkHttp already read,
* and the request is sent again (at once after a system stop, at resume()
* after a pause). So the result is saved before the resume, and a stop
* returns it when it is there. A response not yet read when the stop lands
* is still lost: the stop cancels the call.
*
* A failure that landed first is thrown as it is (kotlinx delivers it).
*
* The caller's job is then cancelled, but plain code still runs. Each caller
* must reach its save (ops.settle for a simple entry, markAccepted for a part)
* with no suspension in between, because a suspension on a cancelled job
* throws. EntryRunTest pins this for both callers.
*/
internal suspend fun <T : Any> awaitLanded(
register: (onResult: (Result<T>) -> Unit) -> Unit,
cancel: () -> Unit,
): T {
val landed = AtomicReference<T?>(null)
try {
return suspendCancellableCoroutine { continuation ->
// Runs at a stop, also after a resume that waits for dispatch. A cancel
// after the response was read and closed does nothing.
continuation.invokeOnCancellation { cancel() }
register { result ->
result.getOrNull()?.let { landed.set(it) }
continuation.resumeWith(result)
}
})
}
} catch (stop: CancellationException) {
landed.get()?.let { return it }
throw stop
}
}

private fun throttled(onProgress: (Long) -> Unit): (Long) -> Unit {
var lastProgressReport = 0L
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ internal class SimpleTransfer(private val run: EntryRun) {
fileExists = { file == null || file.exists() },
)
when (val r = a.result) {
// No suspension between here and ops.settle: the response may have
// landed as the run was stopped (see awaitLanded).
is EntryRun.AttemptResult.Accepted -> return Settlement.Completed(r.response, a.url, a.method)
is EntryRun.AttemptResult.Auth -> {
if (!r.reissue) throw EntryRun.ParkException(r.headerGeneration)
Expand Down
Loading
Loading