Add sixteen sdkharness resilience checks: retry, backoff, reauthorization

.sdkharness/README.md

@@ -5,7 +5,7 @@
 assertions in `tests/`; the harness builds this checkout, starts a fresh local B2 simulator,
 and retains and reports the evidence.
 
-Scenarios so far: `health/golden-path` and eight `conformance` scenarios (below). `tests/run-health` runs
+Scenarios so far: `health/golden-path` and eight `conformance` and sixteen `resilience` scenarios (below). `tests/run-health` runs
 `tests/health/GoldenPath.java` as a single-file Java program that authorizes, uploads a small
 file, downloads it and compares the bytes, lists it, deletes it and confirms it is gone, then
 prints one result line:
@@ -41,18 +41,58 @@ formatted to one capped printable line from exception class names, status codes
 assertions; that is formatting, not redaction, and other lines the check prints are forwarded as they
 are. The only credential in play is the simulator's fixed test one.
 
+## Resilience
+
+`tests/run-resilience` runs one scenario per call, named by `SDKHARNESS_SCENARIO`; every
+`resilience` row in `tests.tsv` points at it. Each scenario is a Java program in
+`tests/resilience/` that arms **one** documented fault through the simulator's control listener
+(`SDKHARNESS_SIMULATOR_CONTROL_URL`), makes one SDK call with the SDK's own default retry policy,
+and asserts two things: the end state (the bytes read back intact, the listing succeeds, or the
+error reaches the caller) and the recovery path, read from the simulator's request journal
+(`GET /journal`: endpoint, status, fault, an opaque upload URL id; never a header, token or body).
+Each file cites the B2 pages it asserts. They share `tests/resilience/Support.java`; as for
+conformance, the wrapper compiles `Support.java` and the check with `javac` against the
+harness-built jars, then runs the check class. Each check creates its own `sdkharness-res-*`
+bucket, clears every fault it armed, and tries to delete everything it made, on failure too; it attempts
+every step, and one that fails turns the verdict into a `FAIL` (`cleanup -- N step(s) failed`).
+
+| Scenario | Fault, and what it asserts |
+|---|---|
+| `upload.retry_503`, `upload.retry_500`, `upload.retry_408` | status on `b2_upload_file`: a fresh `b2_get_upload_url`, the retried upload on a different upload URL id, one stored version, bytes intact |
+| `upload.expired_token_401` | 401 `expired_auth_token` on `b2_upload_file`: a fresh upload URL, then the file lands intact |
+| `upload.get_url_503` | 503 on `b2_get_upload_url`, armed before the first upload in the bucket (the SDK pools upload URLs, so after any earlier upload the fault would never fire): retried, then the file lands |
+| `upload.cap_exceeded_403` | 403 `cap_exceeded`: the error reaches the caller, exactly one `b2_upload_file`, nothing stored |
+| `upload.reset_mid_request`, `upload.reset_before_response` | connection reset: a fresh upload URL, the file reads back intact (a second version is accepted after a reset before the response) |
+| `upload.stall` | the server holds the connection for 60 s: the check sets a 2 s socket timeout (`HttpClientFactoryImpl.Builder.setSocketTimeoutSeconds`, the default is 20 s), the stalled request is abandoned and retried on a new upload URL |
+| `part.retry_503` | 503 on `b2_upload_part`: a successful part on a different URL id, `b2_finish_large_file`, bytes intact |
+| `auth.expired_401` | 401 `expired_auth_token` on `b2_list_file_names`: `b2_authorize_account` again, then the listing succeeds |
+| `auth.clock_expiry` | the simulator's clock moves 24 h and 1 ms: the real 401, a new authorization, the listing succeeds, all on one long-lived client |
+| `api.retry_after_429`, `api.retry_after_503` | `Retry-After: 2`: one retry, the whole call took at least 2 s |
+| `api.backoff_503` | no `Retry-After`: one retry, the whole call took at least 1 s |
+| `download.retry_503` | 503 on `b2_download_file_by_id`: retried, the bytes are intact |
+
+Timing is a lower bound only, never an upper bound (a busy runner would make that flaky). A check
+that is still running after 60 s is killed and reported as a failure: the slowest scenario takes
+about 3 s, so a minute means the SDK never gave up on something.
+
+A failing resilience scenario prints one `SDKHARNESS_RESULT ... FAIL` line with the step and the reason,
+formatted and capped as described under Conformance (formatting, not redaction).
+
 ## What the check can and cannot reach
 
-- **Only the loopback simulator.** `run-health` and `run-conformance` refuse any
-  `SDKHARNESS_SIMULATOR_URL` that is not `http://127.0.0.1:<port>`. In conformance the SDK is also
-  held to it at request time: every check builds its client from one guarded HTTP client factory
-  (`Support.guardedFactory`, plain HTTP only) that refuses any request whose target host is not
-  `127.0.0.1`, redirect hops included, and a refused request fails the verdict
+- **Only the loopback simulator.** `run-health`, `run-conformance` and `run-resilience` refuse any
+  `SDKHARNESS_SIMULATOR_URL` (and, for resilience, `SDKHARNESS_SIMULATOR_CONTROL_URL`) that is not
+  `http://127.0.0.1:<port>`. In conformance and resilience the SDK is also held to it at request time:
+  every check builds its client from one guarded HTTP client factory (`Support.guardedFactory`, plain
+  HTTP only; in resilience it carries the SDK's request settings including the per-check socket
+  timeout, and the control-listener requests are held to the same host) that refuses any request whose
+  target host is not `127.0.0.1`, redirect hops included, and a refused request fails the verdict
   (`loopback guard -- blocked a request to non-loopback host ...`). `health` is guarded by its origin
-  check only. `conformance/GuardSelfTest.java` proves the guard (see below).
+  check only. `conformance/GuardSelfTest.java` and `resilience/GuardSelfTest.java` prove the guard (see
+  below).
 - **Only the simulator's fixed test credential** (`test-key-id` / `test-key`, in the source).
-  Every ambient `B2_*` variable except `B2_BUCKET_NAME` is dropped first (conformance drops all of
-  them and makes its own buckets).
+  Every ambient `B2_*` variable except `B2_BUCKET_NAME` is dropped first (conformance and resilience drop all of
+  them and make their own buckets).
 - **No build.** The harness builds the `core` and `httpclient` jars from this checkout and
   passes them, with their runtime dependencies, as `SDKHARNESS_JAVA_CLASSPATH`. The build
   files here are not changed. A missing `java` on `PATH` reports `SKIP`, which the harness
@@ -63,13 +103,20 @@ are. The only credential in play is the simulator's fixed test one.
 For conformance, set `SDKHARNESS_TEST_LEVEL=conformance`, `SDKHARNESS_SCENARIO=<capability>` and run
 `.sdkharness/tests/run-conformance` (needs `javac`, and no `B2_BUCKET_NAME`).
 
-To re-prove the loopback guard (no simulator needed; `$CP` is the same class path):
+To re-prove the loopback guard (no simulator needed; `$CP` is the same class path; use `resilience`
+in place of `conformance` for the resilience copy):
 
 ```bash
 javac --release 11 -d /tmp/guard -cp "$CP" .sdkharness/tests/conformance/Support.java .sdkharness/tests/conformance/GuardSelfTest.java
 java -cp "/tmp/guard:$CP" GuardSelfTest   # GUARD-SELFTEST b2-sdk-java: PASS
 ```
 
+For resilience, start the simulator with its control listener (`serve.mjs --control`), set
+`SDKHARNESS_TEST_LEVEL=resilience`, `SDKHARNESS_SCENARIO=<scenario>`,
+`SDKHARNESS_SIMULATOR_URL`, `SDKHARNESS_SIMULATOR_CONTROL_URL` and `SDKHARNESS_JAVA_CLASSPATH`
+(as below), and run `.sdkharness/tests/run-resilience` (needs `javac`, and a fresh simulator per
+scenario: a fault is armed once and a stalled connection is held for a minute).
+
 Requires JDK 11 or later and a simulator (`backblaze-labs/b2-simulator`) with a bucket named
 `sdkharness-healthcheck`:
 

.sdkharness/tests.tsv

@@ -8,3 +8,19 @@ conformance	files.delete_version	simulator	./.sdkharness/tests/run-conformance
 conformance	files.hide	simulator	./.sdkharness/tests/run-conformance
 conformance	bucket.crud	simulator	./.sdkharness/tests/run-conformance
 conformance	files.metadata	simulator	./.sdkharness/tests/run-conformance
+resilience	upload.retry_503	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.retry_500	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.retry_408	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.expired_token_401	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.get_url_503	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.cap_exceeded_403	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.reset_mid_request	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.reset_before_response	simulator	./.sdkharness/tests/run-resilience
+resilience	upload.stall	simulator	./.sdkharness/tests/run-resilience
+resilience	part.retry_503	simulator	./.sdkharness/tests/run-resilience
+resilience	auth.expired_401	simulator	./.sdkharness/tests/run-resilience
+resilience	auth.clock_expiry	simulator	./.sdkharness/tests/run-resilience
+resilience	api.retry_after_429	simulator	./.sdkharness/tests/run-resilience
+resilience	api.retry_after_503	simulator	./.sdkharness/tests/run-resilience
+resilience	api.backoff_503	simulator	./.sdkharness/tests/run-resilience
+resilience	download.retry_503	simulator	./.sdkharness/tests/run-resilience

.sdkharness/tests/resilience/ApiBackoff503.java

@@ -0,0 +1,45 @@
+// Copyright 2026, Backblaze Inc. All Rights Reserved.
+// License https://www.backblaze.com/using_b2_code.html
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * api.backoff_503. Run by ../run-resilience (see ../../README.md).
+ *
+ * <p>A 503 with no Retry-After is retried after the documented first backoff step, 1 second (a lower bound
+ * on the whole call), and the listing then succeeds.
+ * https://www.backblaze.com/docs/cloud-storage-integration-checklist
+ */
+public class ApiBackoff503 {
+    public static void main(String[] args) {
+        Support.run("api.backoff_503", () -> {
+            Support.step("setup");
+            Support.bucket();
+            final String name = "res/api-backoff-503.bin";
+            Support.upload(name, Support.payload(1024, 13));
+
+            Support.step("arm the fault");
+            Support.armFault("b2_list_file_names", 503, "service_unavailable", null);
+
+            Support.step("list");
+            Support.markJournal();
+            final List<String> names = new ArrayList<>();
+            final long elapsedMs = Support.timed(() -> names.addAll(Support.listNames()));
+
+            Support.step("recovery path");
+            final List<Support.Entry> window = Support.only(Support.sinceMark(), "b2_list_file_names");
+            Support.check(window.size() >= 2 && window.get(0).status == 503 && window.get(0).fault.equals("injected"),
+                    "the first b2_list_file_names did not carry the 503 fault: " + Support.show(window));
+            Support.check(window.get(window.size() - 1).status == 200, "the listing was never retried to a 200: " + Support.show(window));
+
+            Support.step("backoff");
+            // Lower bound only, for the same reason as the Retry-After checks.
+            Support.check(elapsedMs >= 1000, "the retry came after " + elapsedMs + " ms, sooner than the 1 s first backoff step");
+            Support.note("the call took " + elapsedMs + " ms; only the 1 s lower bound is asserted, which is weak evidence");
+
+            Support.step("listing");
+            Support.check(names.contains(name), "the listing did not contain " + name + ": " + names);
+        });
+    }
+}

.sdkharness/tests/resilience/ApiRetryAfter429.java

@@ -0,0 +1,45 @@
+// Copyright 2026, Backblaze Inc. All Rights Reserved.
+// License https://www.backblaze.com/using_b2_code.html
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * api.retry_after_429. Run by ../run-resilience (see ../../README.md).
+ *
+ * <p>A 429 with Retry-After: 2 is not retried sooner than 2 seconds later (a lower bound on the whole
+ * call), and the listing then succeeds.
+ * https://www.backblaze.com/docs/cloud-storage-integration-checklist
+ */
+public class ApiRetryAfter429 {
+    public static void main(String[] args) {
+        Support.run("api.retry_after_429", () -> {
+            Support.step("setup");
+            Support.bucket();
+            final String name = "res/api-retry_after_429.bin";
+            Support.upload(name, Support.payload(1024, 429));
+
+            Support.step("arm the fault");
+            Support.armFault("b2_list_file_names", 429, "too_many_requests", 2);
+
+            Support.step("list");
+            Support.markJournal();
+            final List<String> names = new ArrayList<>();
+            final long elapsedMs = Support.timed(() -> names.addAll(Support.listNames()));
+
+            Support.step("recovery path");
+            final List<Support.Entry> window = Support.only(Support.sinceMark(), "b2_list_file_names");
+            Support.check(window.size() >= 2 && window.get(0).status == 429 && window.get(0).fault.equals("injected"),
+                    "the first b2_list_file_names did not carry the 429 fault: " + Support.show(window));
+            Support.check(window.get(window.size() - 1).status == 200, "the listing was never retried to a 200: " + Support.show(window));
+
+            Support.step("retry-after");
+            // Lower bound only: the journal has no timestamps, so the whole call is timed, and an upper bound would be flaky.
+            Support.check(elapsedMs >= 2000, "the retry came after " + elapsedMs + " ms, sooner than the 2 s Retry-After");
+            Support.note("the call took " + elapsedMs + " ms; only the 2 s lower bound is asserted, which is weak evidence of honouring Retry-After");
+
+            Support.step("listing");
+            Support.check(names.contains(name), "the listing did not contain " + name + ": " + names);
+        });
+    }
+}

.sdkharness/tests/resilience/ApiRetryAfter503.java

@@ -0,0 +1,45 @@
+// Copyright 2026, Backblaze Inc. All Rights Reserved.
+// License https://www.backblaze.com/using_b2_code.html
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * api.retry_after_503. Run by ../run-resilience (see ../../README.md).
+ *
+ * <p>A 503 with Retry-After: 2 is not retried sooner than 2 seconds later (a lower bound on the whole
+ * call), and the listing then succeeds.
+ * https://www.backblaze.com/docs/cloud-storage-integration-checklist
+ */
+public class ApiRetryAfter503 {
+    public static void main(String[] args) {
+        Support.run("api.retry_after_503", () -> {
+            Support.step("setup");
+            Support.bucket();
+            final String name = "res/api-retry_after_503.bin";
+            Support.upload(name, Support.payload(1024, 503));
+
+            Support.step("arm the fault");
+            Support.armFault("b2_list_file_names", 503, "service_unavailable", 2);
+
+            Support.step("list");
+            Support.markJournal();
+            final List<String> names = new ArrayList<>();
+            final long elapsedMs = Support.timed(() -> names.addAll(Support.listNames()));
+
+            Support.step("recovery path");
+            final List<Support.Entry> window = Support.only(Support.sinceMark(), "b2_list_file_names");
+            Support.check(window.size() >= 2 && window.get(0).status == 503 && window.get(0).fault.equals("injected"),
+                    "the first b2_list_file_names did not carry the 503 fault: " + Support.show(window));
+            Support.check(window.get(window.size() - 1).status == 200, "the listing was never retried to a 200: " + Support.show(window));
+
+            Support.step("retry-after");
+            // Lower bound only: the journal has no timestamps, so the whole call is timed, and an upper bound would be flaky.
+            Support.check(elapsedMs >= 2000, "the retry came after " + elapsedMs + " ms, sooner than the 2 s Retry-After");
+            Support.note("the call took " + elapsedMs + " ms; only the 2 s lower bound is asserted, which is weak evidence of honouring Retry-After");
+
+            Support.step("listing");
+            Support.check(names.contains(name), "the listing did not contain " + name + ": " + names);
+        });
+    }
+}

.sdkharness/tests/resilience/AuthClockExpiry.java

@@ -0,0 +1,47 @@
+// Copyright 2026, Backblaze Inc. All Rights Reserved.
+// License https://www.backblaze.com/using_b2_code.html
+
+import java.util.List;
+
+/**
+ * auth.clock_expiry. Run by ../run-resilience (see ../../README.md).
+ *
+ * <p>One long-lived client (B2StorageClient is one) keeps working past the 24 h life of its account token:
+ * the simulator's clock is moved a day and a millisecond forward, the real 401 comes back, and the
+ * client calls b2_authorize_account again and the listing succeeds. No fault is injected; the 401 is the
+ * simulator's own.
+ * https://www.backblaze.com/apidocs/b2-authorize-account
+ * https://www.backblaze.com/docs/cloud-storage-integration-checklist
+ */
+public class AuthClockExpiry {
+    public static void main(String[] args) {
+        Support.run("auth.clock_expiry", () -> {
+            Support.step("setup");
+            Support.bucket();
+            final String name = "res/auth-clock-expiry.bin";
+            Support.upload(name, Support.payload(1024, 12));
+            // One call first, so the client is holding the token that is about to expire.
+            Support.check(Support.listNames().contains(name), "the listing before the clock moved did not contain " + name);
+
+            Support.step("move the clock");
+            Support.advanceClock(86400001L);
+
+            Support.step("list");
+            Support.markJournal();
+            final List<String> names = Support.listNames();
+
+            Support.step("recovery path");
+            final List<Support.Entry> window = Support.sinceMark();
+            final int hit = Support.indexOf(window, 0, "b2_list_file_names");
+            Support.check(hit >= 0 && window.get(hit).status == 401 && window.get(hit).fault.isEmpty(),
+                    "the first b2_list_file_names was not the simulator's own 401: " + Support.show(window));
+            final int auth = Support.indexOf(window, hit + 1, "b2_authorize_account");
+            Support.check(auth >= 0 && window.get(auth).status == 200, "no b2_authorize_account after the 401: " + Support.show(window));
+            final int retry = Support.indexOf(window, auth + 1, "b2_list_file_names");
+            Support.check(retry >= 0 && window.get(retry).status == 200, "no successful b2_list_file_names after reauthorizing: " + Support.show(window));
+
+            Support.step("listing");
+            Support.check(names.contains(name), "the listing did not contain " + name + ": " + names);
+        });
+    }
+}