From b81f00e673d0e0e14c99a49a21b77467a32a3599 Mon Sep 17 00:00:00 2001 From: andreolf Date: Mon, 17 Aug 2026 21:12:17 +0200 Subject: [PATCH 1/9] docs: document the two-tier storage and pinning model --- docs/STORAGE-AND-PINNING.md | 68 +++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 docs/STORAGE-AND-PINNING.md diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md new file mode 100644 index 000000000..235108b96 --- /dev/null +++ b/docs/STORAGE-AND-PINNING.md @@ -0,0 +1,68 @@ +# Storage and pinning + +How a gitlawb node stores git objects and keeps them available. This documents +the behavior implemented in `crates/gitlawb-node/src/ipfs_pin.rs` and +`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. + +## Two-tier model + +After a push lands, new git objects are pinned to up to two independent sinks. +Both are **opt-in and independent** — a node runs fine with neither, either, or +both configured. + +| Tier | Sink | Module | Enabled by | Purpose | +|------|------|--------|-----------|---------| +| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is itself an IPFS peer | +| Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | + +If a sink's config value is empty, every call into that sink is a no-op — so +leaving `GITLAWB_PINATA_JWT` unset simply disables the warm tier. + +## Configuration + +| Env var | Default | Meaning | +|---------|---------|---------| +| `GITLAWB_IPFS_API` | `""` (disabled) | Base URL of the local Kubo HTTP API, e.g. `http://127.0.0.1:5001` | +| `GITLAWB_PINATA_JWT` | `""` (disabled) | Pinata bearer JWT enabling the warm tier | +| `GITLAWB_PINATA_UPLOAD_URL` | `https://uploads.pinata.cloud/v3/files` | Pinata v3 upload endpoint | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | `8` | Cap on concurrent post-push pin loops across all repos | + +## How pinning runs + +Pinning happens **after** a push is accepted, not on the push's critical path: + +- The **hot** tier pins inline in the post-push encrypt/pin task. +- The **warm** (Pinata) tier runs in a spawned replication tail, so a slow or + unreachable Pinata never blocks the pusher. + +Both tiers share a single global **pin admission semaphore** +(`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is +saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is +bounded by `PIN_BATCH_BUDGET` (120s) so a single large or slow push cannot hold a +slot indefinitely. The Pinata tail re-derives its object list only *after* +acquiring a slot, which bounds outstanding memory to O(refs) rather than +O(pushes × objects). + +De-duplication is per sink: the `pinned_cids` and `pinata_cids` tables record +what each sink already holds, so re-pushing unchanged objects does no redundant +upload. + +## Durability notes + +- With only the **hot** tier, availability depends on the node (and any IPFS + peers that have fetched the CIDs). If the node is down and no peer holds the + objects, they are unreachable until it returns. +- The **warm** tier adds off-node durability via Pinata's Filecoin-backed + storage and makes objects reachable through the public IPFS gateway. +- Running **both** gives a node-local hot copy plus an off-node warm copy. + +> The two sinks are currently invoked as separate call paths. Unifying them +> behind a single pluggable backend interface (to add providers such as direct +> Filecoin deals or self-hosted clusters without touching push logic) is tracked +> separately. + +## See also + +- [RUN-A-NODE.md](RUN-A-NODE.md) — provisioning and running a node +- `crates/gitlawb-node/src/ipfs_pin.rs` — hot-tier implementation +- `crates/gitlawb-node/src/pinata.rs` — warm-tier implementation From 706db57c6528b76f33572ff0a17d87a8ac0d321c Mon Sep 17 00:00:00 2001 From: andreolf Date: Mon, 17 Aug 2026 22:26:22 +0200 Subject: [PATCH 2/9] docs: clarify pin de-duplication is best-effort, not atomic --- docs/STORAGE-AND-PINNING.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 235108b96..31e5ea5dc 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -43,9 +43,11 @@ slot indefinitely. The Pinata tail re-derives its object list only *after* acquiring a slot, which bounds outstanding memory to O(refs) rather than O(pushes × objects). -De-duplication is per sink: the `pinned_cids` and `pinata_cids` tables record -what each sink already holds, so re-pushing unchanged objects does no redundant -upload. +De-duplication is per sink and best-effort: the `pinned_cids` and `pinata_cids` +tables record what each sink already holds, so later pushes normally skip objects +whose successful pin is already recorded. The check-upload-record sequence is not +atomic, so concurrent post-push tasks for the same object, or a failure to record +after a successful upload, can still cause a repeat upload attempt. ## Durability notes From 8a577d3aa320b840f6952641a165047d08c557a7 Mon Sep 17 00:00:00 2001 From: andreolf Date: Tue, 18 Aug 2026 12:39:11 +0200 Subject: [PATCH 3/9] =?UTF-8?q?docs:=20address=20review=20=E2=80=94=20fix?= =?UTF-8?q?=20table/no-op/peer/budget=20claims,=20link=20from=20README?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 1 + docs/STORAGE-AND-PINNING.md | 32 +++++++++++++++++++------------- 2 files changed, 20 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 643992c29..b8f355091 100644 --- a/README.md +++ b/README.md @@ -392,6 +392,7 @@ Read: - [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md) - [`docs/ECONOMICS.md`](docs/ECONOMICS.md) +- [`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md) Use a dedicated low-balance operator wallet. Do not use a treasury wallet as the heartbeat key. diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 31e5ea5dc..8c2321f78 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -12,11 +12,14 @@ both configured. | Tier | Sink | Module | Enabled by | Purpose | |------|------|--------|-----------|---------| -| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is itself an IPFS peer | +| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon | | Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | -If a sink's config value is empty, every call into that sink is a no-op — so -leaving `GITLAWB_PINATA_JWT` unset simply disables the warm tier. +If a sink's config value is empty, its **pin** paths are no-ops — so leaving +`GITLAWB_PINATA_JWT` unset simply disables warm-tier pinning. Note this applies to +pinning only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error +rather than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the +encrypted-blob read endpoint needs Kubo configured. ## Configuration @@ -38,16 +41,19 @@ Pinning happens **after** a push is accepted, not on the push's critical path: Both tiers share a single global **pin admission semaphore** (`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is -bounded by `PIN_BATCH_BUDGET` (120s) so a single large or slow push cannot hold a -slot indefinitely. The Pinata tail re-derives its object list only *after* -acquiring a slot, which bounds outstanding memory to O(refs) rather than -O(pushes × objects). - -De-duplication is per sink and best-effort: the `pinned_cids` and `pinata_cids` -tables record what each sink already holds, so later pushes normally skip objects -whose successful pin is already recorded. The check-upload-record sequence is not -atomic, so concurrent post-push tasks for the same object, or a failure to record -after a successful upload, can still cause a repeat upload attempt. +bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot +indefinitely. The Pinata tail re-derives its object list only *after* acquiring a +slot, which bounds outstanding memory to O(refs) rather than O(pushes × objects). +Note the budget covers the pin batch, not the preceding object-list re-derivation +walk: that walk holds the slot too, and bounds only each child git process +individually (no aggregate deadline). + +De-duplication is per sink and best-effort, backed by the single `pinned_cids` +table: the hot tier keys on its `cid`/`sha256_hex` rows and the warm tier on the +nullable `pinata_cid` column, so later pushes normally skip objects whose +successful pin is already recorded. The check-upload-record sequence is not atomic, +so concurrent post-push tasks for the same object, or a failure to record after a +successful upload, can still cause a repeat upload attempt. ## Durability notes From bc69549ff857c0657503d70c6a61093cbd551100 Mon Sep 17 00:00:00 2001 From: Kevin Codex Date: Mon, 31 Aug 2026 14:20:01 +0800 Subject: [PATCH 4/9] docs: name the hot tier as the unbounded side of the parked-list memory Also record the two wording residuals from the same round: dedup isolation is one-directional (warm row presence suppresses a later hot pin), and an unset Pinata JWT disables uploads but not the tail's permit + walk cost. --- docs/STORAGE-AND-PINNING.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 8c2321f78..16e205fcc 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -16,9 +16,12 @@ both configured. | Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | If a sink's config value is empty, its **pin** paths are no-ops — so leaving -`GITLAWB_PINATA_JWT` unset simply disables warm-tier pinning. Note this applies to -pinning only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error -rather than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the +`GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*, though not the tail's +cost: the post-push Pinata tail still takes a global pin permit and runs the full +object-list re-derivation walk before the upload step returns empty, since its +gate is the announce/walk state, not the JWT. Note the no-op applies to pinning +only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error rather +than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the encrypted-blob read endpoint needs Kubo configured. ## Configuration @@ -43,7 +46,13 @@ Both tiers share a single global **pin admission semaphore** saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot indefinitely. The Pinata tail re-derives its object list only *after* acquiring a -slot, which bounds outstanding memory to O(refs) rather than O(pushes × objects). +slot, which bounds *its* outstanding memory to O(refs) rather than +O(pushes × objects) — but that bound is the warm tail's alone. A **hot-tier** pin +loop takes an owned object list materialized by its caller *before* it waits for +a slot, so every hot loop parked on a saturated pool is holding its full list: +the hot side of the parked-list memory is O(pushes × objects), capped per repo by +`EncryptInflight` but not across repos. Do not size a node's memory from the +semaphore knob (README says the same). Note the budget covers the pin batch, not the preceding object-list re-derivation walk: that walk holds the slot too, and bounds only each child git process individually (no aggregate deadline). @@ -51,7 +60,11 @@ individually (no aggregate deadline). De-duplication is per sink and best-effort, backed by the single `pinned_cids` table: the hot tier keys on its `cid`/`sha256_hex` rows and the warm tier on the nullable `pinata_cid` column, so later pushes normally skip objects whose -successful pin is already recorded. The check-upload-record sequence is not atomic, +successful pin is already recorded. The isolation is one-directional: a hot-only +pin does not suppress the warm tier (`pinata_cid` stays null), but the hot tier's +skip check is row *presence*, and a warm-tier pin inserts the row — so an object +warm-pinned while `GITLAWB_IPFS_API` was unset is skipped by the hot tier after +Kubo is configured, until something re-pins it. The check-upload-record sequence is not atomic, so concurrent post-push tasks for the same object, or a failure to record after a successful upload, can still cause a repeat upload attempt. From 49a9ade2cdfdd75fadbc39c34b0fb5552757ec91 Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Tue, 10 Nov 2026 15:41:06 -0600 Subject: [PATCH 5/9] docs(node): narrow the storage doc to its pinning charter, fix the Durability bullet - opening now says what the page is: optional post-push pinning tiers in ipfs_pin.rs and pinata.rs, with one pointer to bare-repo storage so the page does not read as the answer to "where is my data?" - the "Running both" Durability bullet now carries the warm-first caveat the dedup paragraph already documents: an object warm-pinned before Kubo was configured keeps only the warm copy - README links the doc from the pin configuration row, where operators configuring pins actually look; the original staking-list link stays --- README.md | 5 +++++ docs/STORAGE-AND-PINNING.md | 12 +++++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9589033ec..2c690c95b 100644 --- a/README.md +++ b/README.md @@ -412,6 +412,11 @@ Important node settings: | `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | | `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. | + +For the two-tier pinning model (hot Kubo + warm Pinata), de-duplication +semantics, and parked-list memory behavior see +[`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md). | `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. | | `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. | | `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. | diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 16e205fcc..06aed9b0c 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -1,8 +1,11 @@ # Storage and pinning -How a gitlawb node stores git objects and keeps them available. This documents +How a gitlawb node's **optional post-push pinning tiers** work. This documents the behavior implemented in `crates/gitlawb-node/src/ipfs_pin.rs` and -`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. +`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. Primary +git storage for clone/fetch lives in bare repos under `GITLAWB_REPOS_DIR` +(with optional `GITLAWB_TIGRIS_BUCKET` backing), covered in the README +Configuration table. ## Two-tier model @@ -75,7 +78,10 @@ successful upload, can still cause a repeat upload attempt. objects, they are unreachable until it returns. - The **warm** tier adds off-node durability via Pinata's Filecoin-backed storage and makes objects reachable through the public IPFS gateway. -- Running **both** gives a node-local hot copy plus an off-node warm copy. +- Running **both** gives a node-local hot copy plus an off-node warm copy for + objects pinned while both sinks are enabled. An object warm-pinned before + Kubo was configured keeps only the warm copy (the dedup paragraph above) + unless something re-pins it. > The two sinks are currently invoked as separate call paths. Unifying them > behind a single pluggable backend interface (to add providers such as direct From d3ca3b857ae94c1ba8c34fe4e313a617ed8f3af8 Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Tue, 10 Nov 2026 21:26:46 -0600 Subject: [PATCH 6/9] docs(pinning): keep the config table contiguous, document local_ipfs_provenance - README: the duplicate GITLAWB_MAX_CONCURRENT_PIN_TASKS row and the free-standing paragraph it introduced broke the table; the single row now carries the STORAGE-AND-PINNING cross-link inline - dedup: the hot tier skips on has_ipfs_cid (local_ipfs_provenance = TRUE), not row presence; a Pinata-only row is local_ipfs_provenance = FALSE, so it enters the hot path on later pushes once Kubo is configured instead of being permanently suppressed - the "Running both" durability bullet now agrees: enabling Kubo backfills local copies for warm-only objects --- README.md | 7 +------ docs/STORAGE-AND-PINNING.md | 23 ++++++++++++++--------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 2c690c95b..c579d4cae 100644 --- a/README.md +++ b/README.md @@ -411,12 +411,7 @@ Important node settings: | `GITLAWB_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | -| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. | -| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. | - -For the two-tier pinning model (hot Kubo + warm Pinata), de-duplication -semantics, and parked-list memory behavior see -[`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md). +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. See [docs/STORAGE-AND-PINNING.md](docs/STORAGE-AND-PINNING.md) for the two-tier model, dedup, and parked-list memory. Default 8. | | `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. | | `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. | | `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. | diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 06aed9b0c..67ad3130d 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -61,13 +61,17 @@ walk: that walk holds the slot too, and bounds only each child git process individually (no aggregate deadline). De-duplication is per sink and best-effort, backed by the single `pinned_cids` -table: the hot tier keys on its `cid`/`sha256_hex` rows and the warm tier on the -nullable `pinata_cid` column, so later pushes normally skip objects whose -successful pin is already recorded. The isolation is one-directional: a hot-only -pin does not suppress the warm tier (`pinata_cid` stays null), but the hot tier's -skip check is row *presence*, and a warm-tier pin inserts the row — so an object -warm-pinned while `GITLAWB_IPFS_API` was unset is skipped by the hot tier after -Kubo is configured, until something re-pins it. The check-upload-record sequence is not atomic, +table: the hot tier skips an object when `has_ipfs_cid` reports true, which is +`local_ipfs_provenance = TRUE` — a row that records a real local Kubo CID — and +the warm tier on the nullable `pinata_cid` column, so later pushes normally skip +objects whose successful pin is already recorded for that sink. The isolation is +one-directional: a hot-only pin does not suppress the warm tier (`pinata_cid` +stays null), and a warm-tier pin does not suppress the hot tier either: a row +with only `pinata_cid` set has `local_ipfs_provenance = FALSE`, so an object +warm-pinned while `GITLAWB_IPFS_API` was unset still enters the hot upload path +on later pushes once Kubo is configured, subject to the normal budgets and +failure handling (enabling Kubo can backfill local copies rather than leaving +the object warm-only). The check-upload-record sequence is not atomic, so concurrent post-push tasks for the same object, or a failure to record after a successful upload, can still cause a repeat upload attempt. @@ -80,8 +84,9 @@ successful upload, can still cause a repeat upload attempt. storage and makes objects reachable through the public IPFS gateway. - Running **both** gives a node-local hot copy plus an off-node warm copy for objects pinned while both sinks are enabled. An object warm-pinned before - Kubo was configured keeps only the warm copy (the dedup paragraph above) - unless something re-pins it. + Kubo was configured still enters the hot path on later pushes (the dedup + paragraph above), so enabling Kubo can backfill local copies instead of + leaving it warm-only. > The two sinks are currently invoked as separate call paths. Unifying them > behind a single pluggable backend interface (to add providers such as direct From ca06869051eae52653b0759d0a4e4923cf1ccad9 Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Tue, 10 Nov 2026 21:56:45 -0600 Subject: [PATCH 7/9] docs(pinning): qualify pin admission, per-copy durability, and loop sizing - the opening now names the admission gates: publish-durability confirmation plus root replication; private and non-replicating repos and withheld objects are out of "every push pins" wording - warm-table and durability copy drop the Filecoin backing claim the v3 API does not document - the JWT-unset paragraph now says the permit and re-derivation walk only run when the push was admitted, and an empty IPFS_API can still briefly touch the pin pool on a non-empty list - the semaphore section and README row state the two-loops-per-push fan-out so the default of eight reads as roughly four dual-tier pushes - the backfill and running-both wording is conditional on object selection and per-sink success, naming the reconciliation sweep --- README.md | 2 +- docs/STORAGE-AND-PINNING.md | 60 ++++++++++++++++++++++++------------- 2 files changed, 40 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 1082d5d4f..2e5635185 100644 --- a/README.md +++ b/README.md @@ -411,7 +411,7 @@ Important node settings: | `GITLAWB_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | -| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. See [docs/STORAGE-AND-PINNING.md](docs/STORAGE-AND-PINNING.md) for the two-tier model, dedup, and parked-list memory. Default 8. | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Up to two loops run per push when both IPFS and Pinata are configured. See [docs/STORAGE-AND-PINNING.md](docs/STORAGE-AND-PINNING.md) for the two-tier model, dedup, and parked-list memory. Default 8. | | `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. | | `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. | | `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. | diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 67ad3130d..8b21ea195 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -9,23 +9,36 @@ Configuration table. ## Two-tier model -After a push lands, new git objects are pinned to up to two independent sinks. -Both are **opt-in and independent** — a node runs fine with neither, either, or -both configured. +After a push lands, eligible new git objects can be pinned to up to two +independent sinks. Both are **opt-in and independent** — a node runs fine with +neither, either, or both configured. + +Pinning does not run on every push. The post-push tail only reaches the pin +steps when the push passed publish-durability confirmation and the repository +replicates at root (`announce_at_root`): private repositories and other repos +that are not anonymously readable at the root skip both public pin paths, and +path-scoped rules exclude withheld objects. The warm path additionally requires +this push's own withheld walk to have succeeded. A separate best-effort path can +pin encrypted withheld blobs to Kubo for authorized readers; it is not a +two-tier backup and is outside this page's scope. | Tier | Sink | Module | Enabled by | Purpose | |------|------|--------|-----------|---------| | Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon | -| Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | - -If a sink's config value is empty, its **pin** paths are no-ops — so leaving -`GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*, though not the tail's -cost: the post-push Pinata tail still takes a global pin permit and runs the full -object-list re-derivation walk before the upload step returns empty, since its -gate is the announce/walk state, not the JWT. Note the no-op applies to pinning +| Warm | Pinata (IPFS) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | + +If a sink's config value is empty, its **upload** step is a no-op — so leaving +`GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*. The tail's cost is +conditional on the same admission gates above: only when the push was admitted +to replication does the warm worker still take a global pin permit and run the +full object-list re-derivation walk before the upload step returns empty, +because its gate is the announce/walk state, not the JWT. On a push that was +not admitted, no warm worker runs at all. Note the no-op applies to pinning only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error rather than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the -encrypted-blob read endpoint needs Kubo configured. +encrypted-blob read endpoint needs Kubo configured. An empty `GITLAWB_IPFS_API` +can still briefly touch the pin pool: the hot worker may take a permit when the +built object list is non-empty even though the upload call returns immediately. ## Configuration @@ -45,7 +58,10 @@ Pinning happens **after** a push is accepted, not on the push's critical path: unreachable Pinata never blocks the pusher. Both tiers share a single global **pin admission semaphore** -(`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is +(`max_concurrent_pin_tasks`). One admitted push with both backends configured +can hold up to **two** loops on this pool at once, so the default of eight caps +loops, not pushes: roughly four concurrent all-dual-tier pushes fit before new +loops wait. The pool **defers rather than sheds**: when it is saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot indefinitely. The Pinata tail re-derives its object list only *after* acquiring a @@ -68,10 +84,11 @@ objects whose successful pin is already recorded for that sink. The isolation is one-directional: a hot-only pin does not suppress the warm tier (`pinata_cid` stays null), and a warm-tier pin does not suppress the hot tier either: a row with only `pinata_cid` set has `local_ipfs_provenance = FALSE`, so an object -warm-pinned while `GITLAWB_IPFS_API` was unset still enters the hot upload path -on later pushes once Kubo is configured, subject to the normal budgets and -failure handling (enabling Kubo can backfill local copies rather than leaving -the object warm-only). The check-upload-record sequence is not atomic, +warm-pinned while `GITLAWB_IPFS_API` was unset can still enter the hot upload +path once Kubo is configured — provided a later push selects it (the hot walk +covers objects new to that push) or the enabled reconciliation sweep picks it +up among eligible public objects — subject to visibility checks, budgets, and +failure handling. The check-upload-record sequence is not atomic, so concurrent post-push tasks for the same object, or a failure to record after a successful upload, can still cause a repeat upload attempt. @@ -82,11 +99,12 @@ successful upload, can still cause a repeat upload attempt. objects, they are unreachable until it returns. - The **warm** tier adds off-node durability via Pinata's Filecoin-backed storage and makes objects reachable through the public IPFS gateway. -- Running **both** gives a node-local hot copy plus an off-node warm copy for - objects pinned while both sinks are enabled. An object warm-pinned before - Kubo was configured still enters the hot path on later pushes (the dedup - paragraph above), so enabling Kubo can backfill local copies instead of - leaving it warm-only. +- Running **both** can give a node-local hot copy plus an off-node warm copy: + each copy exists only when its own sink's pin succeeded, so a sink failure or + a truncated batch can leave an object single-copied (the enabled + reconciliation sweep can fill gaps for public objects later). An object + warm-pinned before Kubo was configured can still enter the hot path when a + later push or the sweep selects it (the dedup paragraph above). > The two sinks are currently invoked as separate call paths. Unifying them > behind a single pluggable backend interface (to add providers such as direct From 17e87709401b2904dae9da3fe1465acf27fe948a Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Thu, 12 Nov 2026 14:33:29 -0600 Subject: [PATCH 8/9] docs(pinning): drop unsupported Filecoin claim, normalize punctuation (#576) --- docs/STORAGE-AND-PINNING.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 8b21ea195..18ba38db5 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -10,7 +10,7 @@ Configuration table. ## Two-tier model After a push lands, eligible new git objects can be pinned to up to two -independent sinks. Both are **opt-in and independent** — a node runs fine with +independent sinks. Both are **opt-in and independent**: a node runs fine with neither, either, or both configured. Pinning does not run on every push. The post-push tail only reaches the pin @@ -27,7 +27,7 @@ two-tier backup and is outside this page's scope. | Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon | | Warm | Pinata (IPFS) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | -If a sink's config value is empty, its **upload** step is a no-op — so leaving +If a sink's config value is empty, its **upload** step is a no-op, so leaving `GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*. The tail's cost is conditional on the same admission gates above: only when the push was admitted to replication does the warm worker still take a global pin permit and run the @@ -66,7 +66,7 @@ saturated, a pin loop waits for a slot instead of dropping the pin. Each batch i bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot indefinitely. The Pinata tail re-derives its object list only *after* acquiring a slot, which bounds *its* outstanding memory to O(refs) rather than -O(pushes × objects) — but that bound is the warm tail's alone. A **hot-tier** pin +O(pushes × objects), but that bound is the warm tail's alone. A **hot-tier** pin loop takes an owned object list materialized by its caller *before* it waits for a slot, so every hot loop parked on a saturated pool is holding its full list: the hot side of the parked-list memory is O(pushes × objects), capped per repo by @@ -78,16 +78,16 @@ individually (no aggregate deadline). De-duplication is per sink and best-effort, backed by the single `pinned_cids` table: the hot tier skips an object when `has_ipfs_cid` reports true, which is -`local_ipfs_provenance = TRUE` — a row that records a real local Kubo CID — and +`local_ipfs_provenance = TRUE` (a row that records a real local Kubo CID), and the warm tier on the nullable `pinata_cid` column, so later pushes normally skip objects whose successful pin is already recorded for that sink. The isolation is one-directional: a hot-only pin does not suppress the warm tier (`pinata_cid` stays null), and a warm-tier pin does not suppress the hot tier either: a row with only `pinata_cid` set has `local_ipfs_provenance = FALSE`, so an object warm-pinned while `GITLAWB_IPFS_API` was unset can still enter the hot upload -path once Kubo is configured — provided a later push selects it (the hot walk +path once Kubo is configured, provided a later push selects it (the hot walk covers objects new to that push) or the enabled reconciliation sweep picks it -up among eligible public objects — subject to visibility checks, budgets, and +up among eligible public objects, subject to visibility checks, budgets, and failure handling. The check-upload-record sequence is not atomic, so concurrent post-push tasks for the same object, or a failure to record after a successful upload, can still cause a repeat upload attempt. @@ -97,8 +97,7 @@ successful upload, can still cause a repeat upload attempt. - With only the **hot** tier, availability depends on the node (and any IPFS peers that have fetched the CIDs). If the node is down and no peer holds the objects, they are unreachable until it returns. -- The **warm** tier adds off-node durability via Pinata's Filecoin-backed - storage and makes objects reachable through the public IPFS gateway. +- The **warm** tier adds off-node durability via Pinata's IPFS storage and makes objects reachable through the public IPFS gateway. - Running **both** can give a node-local hot copy plus an off-node warm copy: each copy exists only when its own sink's pin succeeded, so a sink failure or a truncated batch can leave an object single-copied (the enabled @@ -113,6 +112,6 @@ successful upload, can still cause a repeat upload attempt. ## See also -- [RUN-A-NODE.md](RUN-A-NODE.md) — provisioning and running a node -- `crates/gitlawb-node/src/ipfs_pin.rs` — hot-tier implementation -- `crates/gitlawb-node/src/pinata.rs` — warm-tier implementation +- [RUN-A-NODE.md](RUN-A-NODE.md): provisioning and running a node +- `crates/gitlawb-node/src/ipfs_pin.rs`: hot-tier implementation +- `crates/gitlawb-node/src/pinata.rs`: warm-tier implementation From b8435bd0e1e68026a84861d3f5d90125a8bf5eec Mon Sep 17 00:00:00 2001 From: beardthelion <56458543+beardthelion@users.noreply.github.com> Date: Thu, 12 Nov 2026 15:52:06 -0600 Subject: [PATCH 9/9] docs(pinning): qualify warm admission and loop count for coalesced pushes (#576) A push that coalesces onto an in-flight encrypt/pin task never runs its own withheld walk, yet still spawns the warm Pinata worker, which re-derives the withheld set inside its pin permit. Scope "up to two loops per push" to pushes that start their own hot task; a coalesced push adds at most one warm loop. --- README.md | 2 +- docs/STORAGE-AND-PINNING.md | 23 +++++++++++++++-------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 2e5635185..b11e3b93e 100644 --- a/README.md +++ b/README.md @@ -411,7 +411,7 @@ Important node settings: | `GITLAWB_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | -| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Up to two loops run per push when both IPFS and Pinata are configured. See [docs/STORAGE-AND-PINNING.md](docs/STORAGE-AND-PINNING.md) for the two-tier model, dedup, and parked-list memory. Default 8. | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Up to two loops run per push that starts its own encrypt/pin task when both IPFS and Pinata are configured; a push coalesced onto an in-flight task adds at most one warm loop. See [docs/STORAGE-AND-PINNING.md](docs/STORAGE-AND-PINNING.md) for the two-tier model, dedup, and parked-list memory. Default 8. | | `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. | | `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. | | `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. | diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 18ba38db5..04e96e6c4 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -17,10 +17,15 @@ Pinning does not run on every push. The post-push tail only reaches the pin steps when the push passed publish-durability confirmation and the repository replicates at root (`announce_at_root`): private repositories and other repos that are not anonymously readable at the root skip both public pin paths, and -path-scoped rules exclude withheld objects. The warm path additionally requires -this push's own withheld walk to have succeeded. A separate best-effort path can -pin encrypted withheld blobs to Kubo for authorized readers; it is not a -two-tier backup and is outside this page's scope. +path-scoped rules exclude withheld objects. The warm path additionally checks +whether this push ran its own withheld walk: an admitted, non-coalesced push +whose tail walk failed spawns no warm worker. A push that coalesced onto an +in-flight encrypt/pin task has no walk of its own, so it still runs warm +pinning; the detached worker re-derives the withheld set inside its pin permit +(`pinata_object_list_for_refs`) and fails closed on that result instead. A +separate best-effort path can pin encrypted withheld blobs to Kubo for +authorized readers; it is not a two-tier backup and is outside this page's +scope. | Tier | Sink | Module | Enabled by | Purpose | |------|------|--------|-----------|---------| @@ -58,10 +63,12 @@ Pinning happens **after** a push is accepted, not on the push's critical path: unreachable Pinata never blocks the pusher. Both tiers share a single global **pin admission semaphore** -(`max_concurrent_pin_tasks`). One admitted push with both backends configured -can hold up to **two** loops on this pool at once, so the default of eight caps -loops, not pushes: roughly four concurrent all-dual-tier pushes fit before new -loops wait. The pool **defers rather than sheds**: when it is +(`max_concurrent_pin_tasks`). One admitted push that starts its own hot +encrypt/pin task with both backends configured can hold up to **two** loops on +this pool at once, so the default of eight caps loops, not pushes: roughly four +concurrent all-dual-tier pushes fit before new loops wait. A push that coalesced +onto an in-flight task starts no hot task of its own, so it adds at most one new +loop (the warm worker) to that count. The pool **defers rather than sheds**: when it is saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot indefinitely. The Pinata tail re-derives its object list only *after* acquiring a