diff --git a/README.md b/README.md index 00ed9730..b11e3b93 100644 --- a/README.md +++ b/README.md @@ -411,7 +411,7 @@ Important node settings: | `GITLAWB_MAX_CONCURRENT_GIT_OPS` | Max concurrent served git READ ops (upload-pack and its `info/refs` advertisement) across all callers; over-cap sheds a 503 + Retry-After. Anonymous reads draw from this pool, so pair it with `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER`. Pushes and the receive-pack advertisement have their own pools, so a read flood cannot shed an authenticated push. Default 128. | | `GITLAWB_MAX_CONCURRENT_GIT_PUSHES` | Max concurrent `git-receive-pack` POST operations, in a pool separate from the read pool. The anon receive-pack `info/refs` advertisement runs in a third pool of the same size, disjoint from both, so an advertisement flood cannot shed a push either. Two per-source push caps are derived from this value (`/8`, floor 1) and have no env var of their own. Over-cap sheds a 503 + Retry-After. Default 32. | | `GITLAWB_MAX_CONCURRENT_READS_PER_CALLER` | Max concurrent read ops a single caller may hold, so one caller cannot monopolize the read pool. Keyed on the resolved source IP, never the DID, and only as granular as `GITLAWB_TRUSTED_PROXY`: left unset, a node behind an edge or NAT keys every caller on the edge IP and this collapses to one global cap. Default 16. | -| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Default 8. | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | Max post-push pin loops (IPFS + Pinata) running concurrently across all repos. This caps how many loops RUN at once, not how much object-id list memory the node retains: on the local IPFS path a loop parked waiting for a permit still holds its full list. Do not size memory from this knob alone. A loop over cap waits, never drops a pin. Up to two loops run per push that starts its own encrypt/pin task when both IPFS and Pinata are configured; a push coalesced onto an in-flight task adds at most one warm loop. See [docs/STORAGE-AND-PINNING.md](docs/STORAGE-AND-PINNING.md) for the two-tier model, dedup, and parked-list memory. Default 8. | | `GITLAWB_REPO_LEASE_MAX_WAITERS` | Max pushes parked at once waiting for the same repo's write lease. Each waiter pins its buffered pack body, so this bounds that memory for a hot repo; past the cap the newest push sheds a 503 + Retry-After instead of queueing. Pushes to other repos are unaffected, and the lease holder is not counted. Default 8. | | `GITLAWB_MAX_CONCURRENT_IPFS_WALKS` | Max concurrent `GET /ipfs/{cid}` visibility walks across all callers (own pool, disjoint from the served-git pools); over-cap sheds 503. Default 32. | | `GITLAWB_IPFS_WALK_PER_SOURCE` | Max concurrent `/ipfs` walks a single source IP may hold. Default 4. | @@ -478,6 +478,7 @@ Read: - [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md) - [`docs/ECONOMICS.md`](docs/ECONOMICS.md) +- [`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md) Use a dedicated low-balance operator wallet. Do not use a treasury wallet as the heartbeat key. diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md new file mode 100644 index 00000000..04e96e6c --- /dev/null +++ b/docs/STORAGE-AND-PINNING.md @@ -0,0 +1,124 @@ +# Storage and pinning + +How a gitlawb node's **optional post-push pinning tiers** work. This documents +the behavior implemented in `crates/gitlawb-node/src/ipfs_pin.rs` and +`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. Primary +git storage for clone/fetch lives in bare repos under `GITLAWB_REPOS_DIR` +(with optional `GITLAWB_TIGRIS_BUCKET` backing), covered in the README +Configuration table. + +## Two-tier model + +After a push lands, eligible new git objects can be pinned to up to two +independent sinks. Both are **opt-in and independent**: a node runs fine with +neither, either, or both configured. + +Pinning does not run on every push. The post-push tail only reaches the pin +steps when the push passed publish-durability confirmation and the repository +replicates at root (`announce_at_root`): private repositories and other repos +that are not anonymously readable at the root skip both public pin paths, and +path-scoped rules exclude withheld objects. The warm path additionally checks +whether this push ran its own withheld walk: an admitted, non-coalesced push +whose tail walk failed spawns no warm worker. A push that coalesced onto an +in-flight encrypt/pin task has no walk of its own, so it still runs warm +pinning; the detached worker re-derives the withheld set inside its pin permit +(`pinata_object_list_for_refs`) and fails closed on that result instead. A +separate best-effort path can pin encrypted withheld blobs to Kubo for +authorized readers; it is not a two-tier backup and is outside this page's +scope. + +| Tier | Sink | Module | Enabled by | Purpose | +|------|------|--------|-----------|---------| +| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon | +| Warm | Pinata (IPFS) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | + +If a sink's config value is empty, its **upload** step is a no-op, so leaving +`GITLAWB_PINATA_JWT` unset disables warm-tier *uploads*. The tail's cost is +conditional on the same admission gates above: only when the push was admitted +to replication does the warm worker still take a global pin permit and run the +full object-list re-derivation walk before the upload step returns empty, +because its gate is the announce/walk state, not the JWT. On a push that was +not admitted, no warm worker runs at all. Note the no-op applies to pinning +only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error rather +than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the +encrypted-blob read endpoint needs Kubo configured. An empty `GITLAWB_IPFS_API` +can still briefly touch the pin pool: the hot worker may take a permit when the +built object list is non-empty even though the upload call returns immediately. + +## Configuration + +| Env var | Default | Meaning | +|---------|---------|---------| +| `GITLAWB_IPFS_API` | `""` (disabled) | Base URL of the local Kubo HTTP API, e.g. `http://127.0.0.1:5001` | +| `GITLAWB_PINATA_JWT` | `""` (disabled) | Pinata bearer JWT enabling the warm tier | +| `GITLAWB_PINATA_UPLOAD_URL` | `https://uploads.pinata.cloud/v3/files` | Pinata v3 upload endpoint | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | `8` | Cap on concurrent post-push pin loops across all repos | + +## How pinning runs + +Pinning happens **after** a push is accepted, not on the push's critical path: + +- The **hot** tier pins inline in the post-push encrypt/pin task. +- The **warm** (Pinata) tier runs in a spawned replication tail, so a slow or + unreachable Pinata never blocks the pusher. + +Both tiers share a single global **pin admission semaphore** +(`max_concurrent_pin_tasks`). One admitted push that starts its own hot +encrypt/pin task with both backends configured can hold up to **two** loops on +this pool at once, so the default of eight caps loops, not pushes: roughly four +concurrent all-dual-tier pushes fit before new loops wait. A push that coalesced +onto an in-flight task starts no hot task of its own, so it adds at most one new +loop (the warm worker) to that count. The pool **defers rather than sheds**: when it is +saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is +bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot +indefinitely. The Pinata tail re-derives its object list only *after* acquiring a +slot, which bounds *its* outstanding memory to O(refs) rather than +O(pushes × objects), but that bound is the warm tail's alone. A **hot-tier** pin +loop takes an owned object list materialized by its caller *before* it waits for +a slot, so every hot loop parked on a saturated pool is holding its full list: +the hot side of the parked-list memory is O(pushes × objects), capped per repo by +`EncryptInflight` but not across repos. Do not size a node's memory from the +semaphore knob (README says the same). +Note the budget covers the pin batch, not the preceding object-list re-derivation +walk: that walk holds the slot too, and bounds only each child git process +individually (no aggregate deadline). + +De-duplication is per sink and best-effort, backed by the single `pinned_cids` +table: the hot tier skips an object when `has_ipfs_cid` reports true, which is +`local_ipfs_provenance = TRUE` (a row that records a real local Kubo CID), and +the warm tier on the nullable `pinata_cid` column, so later pushes normally skip +objects whose successful pin is already recorded for that sink. The isolation is +one-directional: a hot-only pin does not suppress the warm tier (`pinata_cid` +stays null), and a warm-tier pin does not suppress the hot tier either: a row +with only `pinata_cid` set has `local_ipfs_provenance = FALSE`, so an object +warm-pinned while `GITLAWB_IPFS_API` was unset can still enter the hot upload +path once Kubo is configured, provided a later push selects it (the hot walk +covers objects new to that push) or the enabled reconciliation sweep picks it +up among eligible public objects, subject to visibility checks, budgets, and +failure handling. The check-upload-record sequence is not atomic, +so concurrent post-push tasks for the same object, or a failure to record after a +successful upload, can still cause a repeat upload attempt. + +## Durability notes + +- With only the **hot** tier, availability depends on the node (and any IPFS + peers that have fetched the CIDs). If the node is down and no peer holds the + objects, they are unreachable until it returns. +- The **warm** tier adds off-node durability via Pinata's IPFS storage and makes objects reachable through the public IPFS gateway. +- Running **both** can give a node-local hot copy plus an off-node warm copy: + each copy exists only when its own sink's pin succeeded, so a sink failure or + a truncated batch can leave an object single-copied (the enabled + reconciliation sweep can fill gaps for public objects later). An object + warm-pinned before Kubo was configured can still enter the hot path when a + later push or the sweep selects it (the dedup paragraph above). + +> The two sinks are currently invoked as separate call paths. Unifying them +> behind a single pluggable backend interface (to add providers such as direct +> Filecoin deals or self-hosted clusters without touching push logic) is tracked +> separately. + +## See also + +- [RUN-A-NODE.md](RUN-A-NODE.md): provisioning and running a node +- `crates/gitlawb-node/src/ipfs_pin.rs`: hot-tier implementation +- `crates/gitlawb-node/src/pinata.rs`: warm-tier implementation