diff --git a/docs/adr/ADR-0006-receiver-side-revenue-floor.md b/docs/adr/ADR-0006-receiver-side-revenue-floor.md index 4dc93035..1ca5ba49 100644 --- a/docs/adr/ADR-0006-receiver-side-revenue-floor.md +++ b/docs/adr/ADR-0006-receiver-side-revenue-floor.md @@ -1,7 +1,8 @@ # ADR-0006: Close-group pricing for the receiver-side revenue floor - **Status:** Proposed -- **Date:** 2026-07-14 (amended 2026-08-05 after canary measurement) +- **Date:** 2026-07-14 (amended 2026-08-05 after canary measurement; Amendment 2 + 2026-10-20: enforced on every node) - **Decision owners:** Anselme (@grumbach) - **Reviewers:** - **Supersedes:** none (amends the economic scope of ADR-0004; see Decision) @@ -129,6 +130,13 @@ prospective rather than observed. ## Decision +> **Amended.** [Amendment 2](#amendment-2-2026-10-20-enforced-on-every-node) +> changes four things below: the tolerance (65% → 30%), the reference (now +> capped at the receiver's own price), the scope (paid-list admission is +> floored too) and the rollout (enforced on every node, with nothing to +> configure). Where the two disagree, the amendment wins. The text here is kept +> as decided, so the reasons for each step stay readable. + On single-node store admissions (direct client PUT and immediate fresh replication), the receiver additionally requires, alongside the existing `settled >= 3 x median` check: @@ -260,6 +268,489 @@ payment when it settles below its close group's median-priced floor. revisiting this ADR. Changing the default tolerance or extending the floor to any currently exempt path also requires revisiting it. +## Amendment 2 (2026-10-20): enforced on every node + +This amendment turns enforcement on for every node, with nothing to +configure. To make that safe it also adds a cap to the reference, lowers the +tolerance, floors paid-list admission, and keeps a refusal under the floor, +within a stated margin and once nodes run this release, from costing the +refuser trust. It answers the review +trigger in Validation above. The manipulation result now has a fix for the +dangerous direction and an accepted bound for the other. One-quote liveness +is accepted as a residual that could not be measured. + +### What the shadow period and a review of the code found + +Shadow telemetry from ant-prod-01, 2026-10-14T00:33Z to 2026-10-20T00:07Z, on +node 0.22.0 and 0.23.0: 2,117 floor lines from 699 node instances on 72 hosts. +54 of them (2.6%) skipped, all `thin_sample` with 14 known neighbours; across +all lines the median receiver knew 18. Replaying the logged formula +reproduced all 53 of the nodes' own `below_floor=true` lines (47 distinct +admissions) exactly. Taking each chunk and receiver once leaves 1,898 +admissions over 536 chunks (944 first seen as a client PUT, 954 as fresh +replication), arriving in 48 upload bursts. Of those, 1,801 admissions over +521 chunks were matched to the receiver's own committed count, from the +`Storage commitment rotated` line written when `current()` changes. + +- **65% was not safe.** Against the group median it would have refused 47 of + the 1,898 admissions (2.5%), all among the 1,801 matched, and 2.1% of + chunks would have been refused by every receiver that evaluated them. The + window 2026-09-10 to 09-16 showed the same shape: 3.4% of admissions at + 65%. +- **Honest payments sit further below the group median than the simulation + assumed.** The lowest observed ratio of the settled amount to `3 x` group + median was 0.473, and 1% of admissions settled below 0.57x. Receivers of the + same chunk also disagree about the median: by 1.08x at the median, 1.36x at + p95 (1.37x among receivers matched to their own commitment) and 1.91x at + worst. A node's own committed count swings hour to hour as neighbours come + and go (3,132 to 5,921 keys within nine hours on one node), while the fleet + median held between about 3,700 and 4,100 keys all week. +- **Griefing takes the whole chunk, not one replica.** Every receiver of a + chunk prices against nearly the same median, so neighbours that push it up + make every receiver refuse, after the money has settled. That was the first + enforcement blocker above. +- **One accepting node was enough to replicate a below-floor write.** The node + that stores a write announces it with `PaidNotify` to its paid close group + before it sends fresh offers. `PaidNotify` was verified as + `PaidListAdmission`, which the floor exempted, so peers recorded the key as + paid and paid-list repair later copied the bytes from that one node. The + floor refused the fresh offers and then let repair deliver the same data. +- **A refusal became a trust penalty.** Shadow mode never refused anything, + so none of this has fired, but four existing checks judge a peer on data it + does not hold: + - the storing node probes every fresh-offer target 5 to 15 minutes later + and penalises a peer that does not hold the data: at weight 5 for a chunk + (about a 48% cut in trust) and weight 1 for a pointer (about 12%); + - neighbour sync records a repair proof for every key a node hints to a + peer as a replica, and from an hour later the responsible audit + challenges that peer on the key, at weight 5; + - a node that has held an out-of-range record for three days challenges + the current close group before deleting it, and penalises an absent peer + at weight 5; + - a peer whose quorum check fails for a key only one node hinted penalises + that node, at weight 1, up to three times per verification round. + + A peer that refused on price would be penalised by the first three for + following policy, and the node that accepted the write by the last. + +### Decision + +1. **The reference is capped at the receiver's own price:** + `reference = min(group median price, calculate_price(own committed count))`. + A receiver never demands more than the tolerance of what it would itself + quote right now. So `settled >= 3 x tolerance x own price` always clears + it, however many neighbours lie. While gossip is honest, a receiver above + the group median prices exactly as before. The withdrawn own-price rule's + failure, honest rejections on the fullest nodes, therefore does not return. +2. **Tolerance 30%, a deliberate starting point.** 40% also refuses no + measured admission, but the week measured is light and 40% leaves less + margin (see below). Starting lower gives up some detection for that + margin, and the trade is accepted on purpose. A follow-up reads enforced + traffic, and only a later amendment raises the tolerance toward 40% (see + Validation of this amendment). +3. **Paid-list admission is floored too.** `PaidNotify` is only sent when a + node stores a client's write, and it carries the proof the client + presented for it, within the same reuse window as the PUT itself. Repair + and fetch never call the verifier, and an already verified key is answered + from the cache. So this floors the one remaining fresh economic event and + still reprices no historical receipt. (A client PUT that hits the payment + cache forwards whatever proof the client attached, unverified, exactly as + before this change; its receivers verify it.) +4. **No peer is judged on a contested write.** When a node's verifier first + admits the payment for a write (as the node a client PUT it to, on a fresh + offer, or on a `PaidNotify`), it prices the proof against its current + view; a later admission of the same write from the payment cache reuses + that decision. The write is *contested* in two cases: + - the node cannot price it (no own commitment, or too thin a view); + - its paid price is below 60% of the **uncapped** group median, twice the + 30% floor. + + An admitted contested write, one the node cannot price or one paid between + the floor and that threshold, is still stored and its offers still sent, + but the node does not judge its close-group peers on it: + - a fan-out (chunk or pointer) schedules no possession check. This is + decided from the proof as it stands, even one a cached PUT forwards + unverified, since its peers verify that proof and may refuse it; + - for a chunk, the key's paid-list entry carries a one-byte mark, which + survives restarts. Releases before this one only test presence, so they + read it as an ordinary paid key; + - a marked key goes out in paid hints only, never as a replica hint. No + repair proof is recorded for it, so no responsible audit challenges a + peer on it, and no peer can penalise this node as its sole hinter. A + paid hint is never punished, and repair still works wherever the group + accepted the key. The node also keeps its marked keys in memory, so + leaving them out of the hints it builds for a peer reads nothing on + disk. Each paid-list write updates that copy in commit order, a new + mark just before its commit, so the copy never lacks a committed mark, + and a write that commits the key unmarked, a plain insert that adds it + or a removal, drops any mark the copy holds for it. If the marks cannot be read when the node starts, it still starts, as + before, and hints no key as a replica until a restart can read them; + - the responsible audit never samples a marked key, which also covers one + hinted in the moment before its mark was written; + - a prune audit still needs its proofs before deleting a marked key, but + scores no absent peer for it, and paid-list pruning keeps a marked entry + while the record is held; + - a mark that cannot be read counts as set: when in doubt, judge no one. + While reads fail the node judges no peer on any key they hit, so it + warns at the first failure of a run and, when a read succeeds again, + logs how many failed. + + The mark follows the decision the node took when it verified the payment: + the verifier remembers which admitted chunk payments the floor contested, + in a memory of its own sized for 100,000 of them, so a small payment cache + cannot make it forget one before the key is recorded. A write admitted + later from the payment cache is recorded by that decision. The proof + attached to it was never checked, and anyone can attach one to a + `PaidNotify` or a fresh offer, so it can neither set the mark nor veto it. + The first paid-list record of a key decides its mark, so a later payment + can neither add nor remove it. A fan-out skips its possession check + whenever the write is marked, by this node's verified decision or by the + record of the key that won, read after the write records its own, as + well as when the proof it forwards is contested. Pointers need no mark: + the possession check is the only check that penalises a peer for a + missing pointer. + + The paid quote's price stands in for a third of the settled amount; + settlement is at least that, so an overpaid write is only ever judged more + leniently. Deciding from the proof itself means a write stored after + prices moved is judged like any other. + + The 2x margin covers the largest spread measured between receivers' views + of the same chunk (1.91x; 1.37x at p95 among matched receivers). **It is a + margin, not a guarantee**, because only the refusing peer knows its own + view. Replayed over all 5,536 ordered pairs of measured receivers of the + same chunk, at payment levels from 0.2x to 1x the median in 0.02x steps, + no write one would judge was refused by the other. A 1.5x margin left 4 + such pairs, at payments near 0.3x the median. The cost is coverage: on the + measured week these checks are given up for 1.4% of the 1,801 admissions + matched to their receiver's own commitment, plus the 2.6% of evaluations + the floor could not price. A mark stays while the node holds the key, so a + node keeps unchecked every key it recorded while it could not price + (its first hour before a commitment, or while its gossip view refills + after a restart); the other holders of those keys still check them, + unless the mark was front-run (see the consequences below). + Penalising a peer for following the floor is the worse failure, so that + coverage is given up. A real guarantee needs the + refusing peer to tell the sender. A peer that can say "refused on price" + can also dodge ADR-0003's full-node detection that way, so that is a + separate decision. + + The threshold is its own constant, so a raise of the tolerance needs care. + A node still at 30% judges every write paid above 0.6x of its median, and + a peer already at 40% refuses below 0.4x of its own view. On the measured + pairs that leaves 3 of 5,536 where a refusal would be judged. A raise + therefore takes two releases: the first moves this threshold to 80%, and + the tolerance moves only once nearly every node runs it. Nodes that never + upgrade past the first stay a residual. + + The contest reads the **uncapped** median, which the cap does not + protect. Neighbours that overstate their commitments raise it, more of + their group's writes then count as contested, and the peers holding those + writes go unchecked on them. A minority can lift the median only as far as + the dearest honest members. Overstating also needs fake leaves, which fail + the commitment byte spot-checks (ADR-0004). Neighbours that understate at + the storer narrow the margin instead: on the test fixture, seven of + fifteen pull its median down to the cheapest honest member, which takes + the threshold to about 0.33x of the honest median, so a peer whose own + view and price both sit at least 1.11x above the honest median may refuse + a write the storer judges. That needs the same minority that disarms the + floor (see Griefing below). +5. **Enforced on every node, with nothing to configure.** There is no + per-node switch and no tolerance setting: `ANT_PRICE_FLOOR_ENFORCE` and + `ANT_PRICE_FLOOR_TOLERANCE_PERCENT` are no longer read. A node whose floor + sat above its peers' would refuse writes they store and then fail their + possession checks for it, and one below would just admit underpayment, so + the tolerance is the same everywhere. A node that still sets either + variable gets no warning; the variable is ignored. Backing the floor out + takes a release, so the release train's canary is where it is stopped. A + node whose replication engine fails to start runs without one, as before, + and its floor then has nothing to price from and skips every evaluation + (`skip_reason=no_commitment_cache`). So does a local `ant-devnet`, which + runs no replication engine. + +The telemetry line keeps every earlier field. `enforce` is now always +`true`, which tells these nodes apart from shadow-mode releases while the +fleet upgrades (as does the presence of `target`). `reference_price` is +still the value `required` is computed from. It adds `target` (chunk or +pointer), `group_median_price`, `own_price`, `paid_quote_age_secs` and +`contested`. + +### Why the cap, and why 30% + +The measured admissions replayed through both rules, as admissions refused +out of 1,801: + +| tolerance | median only | capped | +|---|---|---| +| 30% (chosen) | 0 | 0 | +| 40% | 0 | 0 | +| 45% | 0 | 0 | +| 50% | 1 | 0 | +| 55% | 11 | 1 | +| 60% | 26 | 5 | +| 65% (old default) | 47 | 11 | + +- **Margin, not the zero count, picks the tolerance.** Zero refusals out of + 1,801 only bounds the true rate below 0.17% per admission, 0.58% per chunk, + or 6.4% per upload burst at 95% confidence, because admissions in one burst + are not independent. The tolerance therefore rests on how far the worst + observed payment sits above it. With the cap, the lowest ratio is 0.534, so + 30% leaves 1.78x of headroom. 40% would leave 1.34x, 45% 1.19x and 50% only + 1.07x. The week measured is light traffic, 48 upload bursts, so the floor + starts where the margin is widest and moves up as enforced traffic adds to + it. +- **Proof reuse.** A client may present the same proof for up to about a + day. A receiver's capped reference can rise with its own count (one node's + price can double within a day) up to its group median, so a reused proof is + safe only against the uncapped median. There, the worst measured admission + sat at 0.473x, 1.58x above 30% (1.18x above 40%). Fleet-wide the median + reference moved by up to about 1.2x in price from one day to the next, + which 1.58x covers. Neither per-group drift over a day nor refusals of aged + proofs were measured. The new `paid_quote_age_secs` field makes them + measurable. +- **What it catches.** The realistic underpayment pays a near-empty, fresh + or sybil quoter. A baseline quote is about a sixth of the group median. + Share of measured receivers that refuse a 3x payment to a quoter, with the + cap: + + | quoter's count, as a share of the group median | 30% (chosen) | 40% | + |---|---|---| + | 0 (a baseline quote) | 91.4% | 98.0% | + | a quarter | 75.4% | 94.6% | + | 40% | 27.9% | 75.2% | + | half | 0% | 39.1% | + + Every receiver observed for a chunk refuses a baseline quote on 84.5% of + chunks at 30%, and 95.2% at 40%. The median-only rule at 40% refuses the + first three rows at 100%, 99.6% and 95.8%. That detection is what the cap + and the lower tolerance give up. +- **What it does not catch.** A quoter 40% as full as its group is admitted + by 72% of receivers, and one half as full by all of them. Shopping for the + cheapest member of the close group is not caught either: members measured + in August spanned about 1.4x in price, so the cheapest sits no lower than + about 0.7x the median. No tolerance that is safe for honest traffic catches + that, with or without the cap. +- **Griefing.** In the worst case, every neighbour gossips the protocol + maximum. The median-only rule then refuses every honest payment. With the + cap, all 1,801 measured admissions still clear at 30% (1,799 at 40%). On + the test fixture, seven of fifteen overstating neighbours flip no verdict + and eight do. Understating is where 30% costs something. Six neighbours + claiming almost nothing leave the floor standing against a baseline quote, + and seven disarm it, because the median then falls to the cheapest honest + member (3,000 keys), and 30% of that member's price is below a baseline + quote. At 40% it took eight. Disarming admits an underpayment and costs no + one an honest payment, so this minority bound is an accepted residual of + starting at 30%. For overstating, the close group's honest majority stays + the trust boundary the rest of the protocol already rests on. + +### Consequences of this amendment + +- **Liveness is a quorum question.** A client needs 4 stores out of up to 20 + targets (a pointer needs 5 of 7). It moves to the next target, with the same + proof, when one refuses. Under the cap, every receiver whose own price is at + most `settled / (3 x 30%)` accepts unconditionally. That is nearly every + receiver for a payment at the group median. The measurement could not + observe the quorum itself: the median chunk shows only two receivers' client + PUT lines. +- **One-quote liveness is an accepted, unmeasured residual.** An honest client + whose only reachable quoter is a fresh node pays 3x baseline, and nearly + every mature receiver refuses it. This cannot be told apart from the attack. + None of the measured payments was at baseline, and the lowest paid issuer + committed 2,147 keys, but that does not bound how often it happens. A client + retries a refused proof + for up to about a day, then re-quotes and pays again. A client-side change, + such as not settling a lone baseline quote when a committed quote is + available, is the remaining fix and belongs to the client. +- **A write below the floor stays where it was accepted.** A near-empty quoter + accepts its own payment, because its cap is its own price. A node with no + commitment or a thin view skips the floor. Mature peers now refuse both the + fresh offer and the `PaidNotify`. Unless the accepting nodes make up a + majority of the group, neither paid-list repair nor presence repair has the + votes to authorise copying the data. The accepting node finds such a write + contested and marks it (item 4), so it judges none of the peers that + refused it while it holds the key. When the accepting nodes are a majority + of the paid group, the refusers repair the key from those votes like any + other, unmarked, since repair carries no proof to price; a refuser whose own + quorum check fails while its neighbours' passes can then be judged on it. +- **Narrow gaps remain on nodes running this release.** + - On the PUT path the record is stored a moment before its paid-list + entry is written. A sync that scans the store in that moment can hint + the key once; the audit filter keeps that from becoming an audit, but a + refusing peer can charge the storer weight 1 once. A crash in that + moment leaves the stored key unrecorded, so unmarked unless a later + verified `PaidNotify` records it. + - A PUT served from the payment cache with no proof attached stores the + record without writing a paid-list entry at all, as before this change, + so it is not marked either. + - The verifier's memory of contested admissions is bounded: a decision is + lost before its key is recorded only if 100,000 newer contested + admissions reach the node in the moments between its verification and + its record. + - A paid-list insert that fails before its commit leaves the key out of + the paid list, so unmarked. When a marked insert's commit is what + failed, the in-memory copy keeps the mark, since the commit may still + have reached the disk, so this node does not hint the key as a replica + while the mark stays. A later plain insert that adds the key, or a + removal of it, drops the mark once that write commits, and a restart + rebuilds the copy from the database. + - Paid-list pruning checks for the record before and after its network + round, but a record admitted between that check and the delete would + lose its mark. Both store paths require this node inside the same width + the prune requires it outside, so that needs its routing view to change + at that moment. +- **The first paid-list record decides a key's mark, which a quoter can + front-run.** A quoter in the 20 closest to an honest key can pay itself + for the key, for gas only, and announce it before the client's write + lands. At a price between the floor and the contest threshold, or at any + price on nodes that cannot price, the nodes it reaches first mark the key + and leave it unchecked, though they still store and repair it; those among + the client's storers skip their possession check too, while storers it did + not reach first still run theirs. Paying above the + threshold first does the opposite: those nodes record the key unmarked and + can judge peers that later refuse the client's cheaper write, until those + peers repair it from the votes the front-run created. +- **Rollout skew is the main rollout risk.** Until the fleet upgrades, a node + on an older release admits every payment and still probes every + fresh-offer target 5 to 15 minutes later. Each upgraded peer that refused a + below-floor chunk it fanned out is penalised at weight 5. In that node's + view the peer falls from a neutral 0.5 to about 0.26, below the 0.35 swap + threshold, for about 9.5 hours (a pointer costs weight 1: 0.5 to 0.44). The + score is local to that node (for peers in its persisted close group it + survives a restart), and falling below 0.35 only lets a better candidate + replace the peer in that node's routing + table when its bucket is full; nothing is blocked. Honest writes are not + refused, so this needs below-floor writes, which an attacker can buy for + little more than gas by paying its own near-empty node within the 20 + closest to the key: one write stored by `s` older nodes yields up to `s` + penalties on each upgraded member of the close group. Such a key also + stays on the older node unmarked, before and after it upgrades: its replica + hints and responsible and prune audits keep judging the refusers on it, + repeatedly, while it holds the key, and the refusers penalise it (weight 1) + as the sole hinter of a key their quorum rejects. On the v0.17.0 rollout the + managed fleet went from 24 of 73 hosts in the first hour to 65 in six and + 71 within about 17; community nodes follow on their own schedule. Enabling + enforcement one release after item 4 ships would remove this, at the cost + of a release that does not enforce. That was not chosen. +- **Retry load is new.** A proof that was accepted before can now be refused. + Refusals are not cached, so every peer a client tries re-verifies it, the + on-chain read included: up to 20 targets over 4 rounds. The load falls on + below-floor proofs, including the honest residuals above (a lone baseline + quote, an aged proof). +- Floor refusals stay economic. The refusal itself feeds no trust or + misbehaviour scoring, and on nodes running this release item 4 keeps the + checks that judge peers on held data from turning it into a penalty, within + the margin stated there and apart from the gaps above. Merkle batches still + bypass the floor entirely. + +### Validation of this amendment + +- Unit tests pin each piece: + - the cap binds below the median, and no number of overstating neighbours + lifts a receiver above its own price; + - seven of fifteen overstating neighbours leave the verdict unchanged and + eight flip it, and six understating neighbours leave the floor standing + while seven disarm it (pinned as known limits); + - the cheapest member of a realistic group is admitted, and a near-empty + quoter is refused; + - paid-list admission and pointer payments are floored; + - a write below the contest threshold, or one the node cannot price, is + contested, and one at the median is not. The threshold is 60% of the + uncapped median to the atto, it stays on the median when the cap binds, + and a fan-out judges the proof it forwards against the view when it fans + out, while a chunk's paid-list mark keeps the decision its verification + made; + - the tolerance is 30%: exactly `3 x 30%` of the group median clears it + and one atto less does not; + - a node with no floor wired contests nothing; + - the verifier remembers a contested payment it admitted, in a memory a + small payment cache cannot shrink, a later cache-hit admission reads that + decision, and a refused payment is never remembered; the paid-list mark + is written only with a new entry, survives a restart and a later plain + insert, and goes with the entry; + - a marked key is left out of replica hints and of responsible-audit + sampling, and its paid entry outlives paid pruning while its record is + held; + - the in-memory copy of the marks is filled on opening and follows every + write and removal, and a marked insert and a batch removal change it + only in the order writes take; a mark held in memory alone, as a failed + commit can leave it, goes with a later plain insert of the key and with + its removal, single or batched; building + replica hints works while the paid list cannot be read at all, and a + paid list whose marks cannot be read on opening still opens and hints no + replica; + - a mark that cannot be read counts as set in the responsible audit, paid + pruning and a write's fan-out, and a run of such reads logs one + warning. The tests make the read fail by holding the thread's one LMDB + reader slot; + - a record reports the winning record's mark, or its own when that is + set, so a plain decision that loses to a marked record still leaves the + write unjudged. +- Four e2e tests run the real schedulers on a local testnet, each with a + control. They settle nothing on chain, so where a test needs a verified + contested admission it stands one in; unit tests pin that a real + verification records it: + - a chunk PUT through the handler schedules no possession check when the + storer's verified decision was contested (the key is marked and hinted + to a close peer as paid only), when only the proof it forwards is + contested, or when a marked record of the key won before its own plain + one, while the control write is penalised; + - a fresh pointer state the storer cannot price is not used to judge its + peers, while the control is; + - a prune audit of a marked record keeps it but scores none of the absent + peers, while the control scores them; + - a PUT, a fresh offer and a `PaidNotify`, each admitted from the payment + cache with a forged proof the floor would contest, mark the key exactly + when the recording node's own verified decision was contested, and never + from that proof. +- 54 mutations of the floor, the contest decision, the verifier's memory, + the mark, its in-memory copy and every check that reads the mark were run + against these unit and e2e tests on this amendment's code, and 53 fail at + least one of them. The one that survives makes prune-audit scoring count + a mark that cannot be read as unset. Only the e2e prune test reaches that + check, and it cannot make a node's paid-list read fail; the same read + counts as set, under test, in the audit, paid pruning and the fan-out. +- `tests/e2e/price_floor_ab.rs` drives the same honest and near-empty-quoter + uploads through a production-wired local testnet. Nodes are seeded to + production-sized commitments, payments are real (on Anvil), and possession + checks are shortened. It reports who stores each write, whose paid list + holds it, and whether any peer is penalised, so two builds can be compared. + It asserts what the floor promises: + - every honest upload reaches a quorum with no floor refusal; + - no peer that refused a near-empty write, or that the quoter fanned it + out to, loses penalty-sized trust in the quoter's view; + - every near-empty write stays off a quorum, with every refusal the + floor's. A mature storer whose own price is low enough may accept it, as + the cap allows. + + It refuses to report when the floor could not evaluate at all. Its tail is + minutes, too short for the responsible audit (an hour) or a prune audit + (three days). The prune e2e test covers the prune audit; the responsible + audit is covered by keeping marked keys out of replica hints, so no repair + proof exists, and by the sampling filter's unit test. It returns at once + unless `PRICE_FLOOR_AB=1`, so CI only compiles it. +- `tests/poc_price_floor_live.rs` runs the floor against a real Anvil vault + and real signed neighbour commitments. It shows that a 3x-baseline + settlement is refused, and that an at-floor overpayment clears while one + atto less does not. +- After release, watch `ant_node::payment::price_floor` for `below_floor=true` + with `enforce=true`, split by `context` and `target`, along with the + `contested=true` and `skipped=true` rates. Any refusal of a payment whose + paid issuer holds a mature commitment is a reason to revisit this amendment. + So is a refusal rate above 0.1% of evaluations. +- **Raising to 40%.** Each evaluated line carries `settled` and + `reference_price`, so it answers whether 40% would have refused it: + `settled < 3 x 40% x reference_price`. The case for 40% is enforced traffic + where that never happens, counted per upload burst: chunks less than 120 s + apart, and one event if any admission in the burst would be refused. Zero + events in 300 bursts bound the rate below 1% at 95% confidence, against + 6.4% from the 47 bursts measured so far that hold a matched admission. + Bursts from one client are not independent, so that bound is the least the + evidence must show, not proof. Production logs are kept for seven days, so + each week's counts must be saved before they expire. Before any raise, + aged proofs (`paid_quote_age_secs`) and + payments to near-empty quoters are read the same way. The raise then + follows the two steps in item 4. + ## Notes for AI-assisted work AI tools may help draft this ADR, but **must not mark it Accepted without diff --git a/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md index 509f7cc5..aa7bc08e 100644 --- a/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md +++ b/docs/adr/ADR-0008-storage-economics-and-payment-protocol.md @@ -359,7 +359,7 @@ and a node that was actually paid is nominated for a first audit. | Node re-verification of **non-paid** quotes in the bundle | not done (bad signature on an unpaid quote is accepted) | | Node rejection of off-curve quotes (`QUOTE_ARITHMETIC_RECHECK_ENABLED`) | **enforced**, on every quote in the bundle | | Quote/commitment mismatch reported to trust | off (telemetry only) | -| Receiver-side price floor (ADR-0006) | shadow (`ANT_PRICE_FLOOR_ENFORCE`, 65% tolerance), priced against the **close group's median** committed count, not the receiver's own; the own-price basis was withdrawn after canary measurement | +| Receiver-side price floor (ADR-0006) | **enforced** on every node at 30% of a reference price: the **close group's median** committed-count price, capped at the receiver's own; covers client PUT, fresh replication and paid-list admission; no per-node switch (ADR-0006 Amendment 2) | | Payee eligibility gate (ADR-0005) | **not merged**; observe-only on its branch (`ADR5_ENFORCE`) | | Client pays the 3× multiplier on the merkle path | **enforced** by the published client, immediately and unconditionally (no flag, no date) | | Storer requires it (`MERKLE_PARITY_ENFORCED_FROM_UNIX`) | **enforced** by upgraded nodes, for receipts stamped from 2026-08-04 15:00 UTC onward | @@ -452,7 +452,9 @@ indistinguishable from a single receipt. mechanism to reprice. - **Median-of-K is only as honest as the quote set the client presents.** A modified client can shop the neighbourhood and settle the cheapest valid - quote; the receiver-side floor is the counterweight and is not yet enforcing. + quote. The receiver-side floor, enforced on every node (ADR-0006 Amendment + 2), is the counterweight: it refuses near-empty quotes, but not the + cheapest member of the close group. ### Neutral / Operational diff --git a/src/devnet.rs b/src/devnet.rs index 18e459bb..ddf5170c 100644 --- a/src/devnet.rs +++ b/src/devnet.rs @@ -7,7 +7,7 @@ use crate::ant_protocol::CHUNK_PROTOCOL_ID; use crate::config::{default_root_dir, NODES_SUBDIR, NODE_IDENTITY_FILENAME}; use crate::logging::{debug, info, warn}; use crate::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, QuoteGenerator, + EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator, QuotingMetricsTracker, }; use crate::replication::config::ReplicationConfig; @@ -960,7 +960,6 @@ impl Devnet { cache_capacity: DEVNET_PAYMENT_CACHE_CAPACITY, close_group_size: replication_config.close_group_size, local_rewards_address: rewards_address, - price_floor: PriceFloorConfig::default(), }; let payment_verifier = PaymentVerifier::new(payment_config); let metrics_tracker = QuotingMetricsTracker::new(DEVNET_INITIAL_RECORDS); diff --git a/src/node.rs b/src/node.rs index ac1f3588..63580116 100644 --- a/src/node.rs +++ b/src/node.rs @@ -9,9 +9,7 @@ use crate::event::{create_event_channel, NodeEvent, NodeEventsChannel, NodeEvent use crate::logging::{debug, error, info, warn}; use crate::payment::metrics::QuotingMetricsTracker; use crate::payment::wallet::parse_rewards_address; -use crate::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, QuoteGenerator, -}; +use crate::payment::{EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator}; use crate::replication::config::ReplicationConfig; use crate::replication::fresh::FreshWriteEvent; use crate::replication::ReplicationEngine; @@ -504,16 +502,7 @@ impl NodeBuilder { cache_capacity: config.payment.cache_capacity, close_group_size, local_rewards_address: rewards_address, - // Shadow mode by default; enforcement is a per-node canary opt-in - // via ANT_PRICE_FLOOR_ENFORCE (see PriceFloorConfig). - price_floor: PriceFloorConfig::from_env(), }; - if payment_config.price_floor.enforce { - info!( - "Price floor ENFORCEMENT enabled (tolerance {}%)", - payment_config.price_floor.tolerance_percent - ); - } let payment_verifier = PaymentVerifier::new(payment_config); let metrics_tracker = QuotingMetricsTracker::new(0); let mut quote_generator = QuoteGenerator::new(rewards_address, metrics_tracker); diff --git a/src/payment/mod.rs b/src/payment/mod.rs index c9b2a32d..107f16f6 100644 --- a/src/payment/mod.rs +++ b/src/payment/mod.rs @@ -63,8 +63,7 @@ pub use ant_protocol::payment::verify::{ }; pub use single_node::SingleNodePayment; pub use verifier::{ - EvmVerifierConfig, PaymentStatus, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, - VerificationContext, MAX_PAYMENT_PROOF_SIZE_BYTES, MIN_PAYMENT_PROOF_SIZE_BYTES, - PRICE_FLOOR_ENFORCE_ENV, PRICE_FLOOR_TOLERANCE_ENV, + EvmVerifierConfig, PaymentStatus, PaymentVerifier, PaymentVerifierConfig, VerificationContext, + MAX_PAYMENT_PROOF_SIZE_BYTES, MIN_PAYMENT_PROOF_SIZE_BYTES, PRICE_FLOOR_TOLERANCE_PERCENT, }; pub use wallet::{is_valid_address, parse_rewards_address, WalletConfig}; diff --git a/src/payment/verifier.rs b/src/payment/verifier.rs index 0d9a126d..12ca85c2 100644 --- a/src/payment/verifier.rs +++ b/src/payment/verifier.rs @@ -12,6 +12,7 @@ use crate::payment::proof::{ deserialize_merkle_proof, deserialize_single_node_proof, detect_proof_type, ProofType, }; use crate::replication::commitment::MAX_COMMITMENT_KEY_COUNT; +use crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; use crate::replication::config::K_BUCKET_SIZE; use crate::storage::ChunkStore; use ant_protocol::payment::verify::{verify_quote_content, verify_quote_signature}; @@ -30,7 +31,7 @@ use saorsa_core::P2PNode; use std::collections::HashMap; use std::num::NonZeroUsize; use std::sync::Arc; -use std::time::Instant; +use std::time::{Instant, SystemTime}; /// Minimum allowed size for a payment proof in bytes. /// @@ -61,29 +62,33 @@ const PAYMENT_VERIFY_SLOW_LOG_MS: u128 = 500; /// This is the Kademlia K width, intentionally wider than `CLOSE_GROUP_SIZE`. const PAID_QUOTE_ISSUER_CLOSENESS_WIDTH: usize = K_BUCKET_SIZE; -/// Default tolerance for the receiver-side price floor, as a percentage of the -/// close group's MEDIAN commitment-bound price (see -/// [`PaymentVerifier::group_reference_price`]). +/// The receiver-side price floor, as a percentage of the reference price. /// -/// Derived from production, not chosen: on ant-prod-01 over 2026-07-29..08-04, -/// pricing each receiver against its OWN price rejected 0.377% of honest stores -/// while still admitting the cheapest-of-K underpayment at 62.9% of receivers, -/// because node prices span ~4x across the fleet and an honest client pays the -/// group median. Against the group median instead, honest payments land near the -/// reference and the underpayment lands well below it, so a tolerance separates -/// them. +/// The reference is the close group's median commitment-bound price, capped +/// at the receiver's own; a single-node payment must settle at least `3x` +/// this share of it. One network value, the same on every node and never +/// configured: a node whose floor sat above its peers' would refuse writes +/// they store and fail their checks for it. Chosen from production telemetry +/// as a deliberate starting point below what the measurements allow; the +/// measurements, and the rule for raising it, live in ADR-0006 Amendment 2. +pub const PRICE_FLOOR_TOLERANCE_PERCENT: u64 = 30; + +/// A write whose paid price is below this percentage of the UNCAPPED group +/// median is contested: a close-group peer whose view of the group sits higher +/// than this node's may have refused it under its floor, so this node does not +/// judge its peers on it (see [`PaymentVerifier::price_contested`]). /// -/// At the minimum permitted view, 65% rejects 0.140% of honest payments while -/// catching ~90% of underpayment, against the own-price floor's measured 0.377% -/// and 37.1%. Both axes improve, by roughly 2.7x on rejections rather than the -/// order of magnitude an earlier revision claimed. 60% is the conservative -/// alternative: 5x safer on honest traffic, at 66% detection. -/// -/// The full sweep, the sample-size table and the simulation's assumptions live -/// in ADR-0006 — deliberately in one place, because the same tables -/// duplicated here and in the ADR had already drifted apart once. Tighten or -/// loosen only from observed shadow telemetry, never speculatively. -const PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT: u64 = 65; +/// Twice the floor of this node's view, so a peer needs a view twice as high +/// before it can refuse a write this node judges. A margin, not a guarantee: +/// ADR-0006 Amendment 2 carries the measured spread between receivers' views +/// and the replay this was checked against. It is its own constant because a +/// raise of the tolerance moves it first, one release ahead. +const PRICE_FLOOR_CONTESTED_PERCENT: u64 = 60; + +/// Contested chunk admissions the verifier remembers, the most recent first, +/// so each at least until its paid-list record is written (see +/// [`PaymentVerifier::verified_payment_contested`]). +const PRICE_CONTESTED_MEMORY_CAPACITY: usize = 100_000; /// Minimum number of fresh NEIGHBOUR commitments (this node's own is counted /// separately) required before the price floor will evaluate at all. @@ -103,13 +108,12 @@ const PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT: u64 = 65; /// [`PAID_QUOTE_ISSUER_CLOSENESS_WIDTH`] closest, with the full table in /// ADR-0006. /// -/// Availability cost is expected to be small — the gossip TTL is hours against a -/// much shorter sweep, so a settled node should normally know most of its -/// neighbours — but that is an assumption, not a measurement, and the sample is -/// drawn from peers close to the CHUNK while the cache is filled from peers -/// close to THIS NODE, so edge-of-range chunks will skip more often. The -/// `skipped=true` / `skip_reason` telemetry exists to measure that in shadow -/// mode before anyone enforces. +/// Availability cost is small, and measured: on ant-prod-01 over +/// 2026-10-14..20, 54 of 2,117 floor lines (2.6%) skipped, all `thin_sample` +/// with 14 known neighbours; across all lines the median receiver knew 18. +/// The sample is drawn from peers close to the CHUNK while the cache is filled +/// from peers close to THIS NODE, so edge-of-range chunks skip more often. The +/// `skipped=true` / `skip_reason` telemetry keeps measuring it. /// /// A node that could suppress its neighbours' gossip can therefore disable this /// receiver's floor, but cannot make it reject honest traffic. That asymmetry is @@ -118,9 +122,9 @@ const PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS: usize = 15; /// Why a price-floor evaluation did not produce a reference price. /// -/// Shadow mode exists to calibrate -/// [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] from real deployments, and these -/// three causes call for completely different responses: a mid-rotation node is +/// The telemetry calibrates [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] from +/// real deployments, and these three causes call for completely different +/// responses: a mid-rotation node is /// expected and self-correcting, an unwired cache is a startup-order bug, and a /// persistently thin sample means the gate is set above what the network can /// supply. Collapsing them into one counter would make the telemetry unable to @@ -151,6 +155,17 @@ impl SkipReason { } } +/// What the price floor priced one admission against. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct FloorReference { + /// The amount the tolerance applies to: the lower of the two below. + price: Amount, + /// The close group's lower-median commitment-bound price. + group_median: Amount, + /// The price this node would itself quote right now. + own: Amount, +} + /// What the price floor could see of the close group on one admission. /// /// `neighbours` is the quantity the gate actually tests; `total` includes this @@ -188,107 +203,6 @@ impl GroupSample { } } -/// Environment variable enabling price-floor ENFORCEMENT (`1`/`true`). -/// -/// Absent or any other value means shadow mode: the floor is computed and -/// logged, never enforced. Per-node so enforcement can roll out canary-first; -/// unsetting it (and restarting) is the kill switch. -pub const PRICE_FLOOR_ENFORCE_ENV: &str = "ANT_PRICE_FLOOR_ENFORCE"; - -/// Environment variable overriding the price-floor tolerance percentage -/// (`0..=100`). Invalid or absent values use the default. -pub const PRICE_FLOOR_TOLERANCE_ENV: &str = "ANT_PRICE_FLOOR_TOLERANCE_PERCENT"; - -/// Receiver-side price floor for single-node store admissions. -/// -/// ADR-0004 gives every quote a price CEILING (`price == -/// calculate_price(committed_key_count)`), but a ceiling is not a revenue -/// floor: a modified client can fetch quotes from the whole neighbourhood and -/// settle only the cheapest valid one in a 1-quote proof. This policy is the -/// floor half: the settled amount must also clear the close group's MEDIAN -/// commitment-bound price, scaled by a tolerance. -/// -/// The reference is the group's median rather than this receiver's own price. -/// A client pays `3x` the median of the quotes it collected, so comparing that -/// to a single node's price compares a median to one draw from a distribution -/// spanning ~4x — which in production rejected honest stores on the fullest -/// nodes while admitting underpayment at the emptiest. The reference is built -/// only from this node's own commitment and its neighbours' TTL-fresh gossiped -/// ones, never from the payment bundle. -/// -/// Rollout is shadow-first: with `enforce == false` (the default) the floor -/// is evaluated and logged on every single-node store admission but never -/// rejects. Enforcement is enabled per node via [`PRICE_FLOOR_ENFORCE_ENV`] -/// for canary rollout. A floor rejection is an economic admission decision -/// only — it must never feed trust/misbehaviour scoring. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub struct PriceFloorConfig { - /// When true, a below-floor settled payment is rejected. When false - /// (default), the verifier only logs would-reject telemetry. - pub enforce: bool, - /// Floor as a percentage of the close group's median commitment-bound - /// price (`0..=100`). - pub tolerance_percent: u64, -} - -impl Default for PriceFloorConfig { - fn default() -> Self { - Self { - enforce: false, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - } - } -} - -impl PriceFloorConfig { - /// Build from the environment (see [`PRICE_FLOOR_ENFORCE_ENV`] and - /// [`PRICE_FLOOR_TOLERANCE_ENV`]). Never panics. - /// - /// An unset tolerance uses the default. A tolerance that is *present but - /// invalid* (unparseable or `> 100`) is an operator error: rather than - /// silently enforce at the default, this **fails closed to shadow mode** - /// (`enforce = false`) and logs a prominent error, so a fat-fingered - /// tolerance can never enforce an unintended floor against real payments. - #[must_use] - pub fn from_env() -> Self { - let enforce_requested = std::env::var(PRICE_FLOOR_ENFORCE_ENV) - .is_ok_and(|v| matches!(v.trim(), "1" | "true" | "TRUE" | "True")); - - let tolerance_raw = std::env::var(PRICE_FLOOR_TOLERANCE_ENV).ok(); - let valid_tolerance = tolerance_raw - .as_deref() - .map(str::trim) - .and_then(|s| s.parse::().ok()) - .filter(|percent| *percent <= 100); - // A tolerance that is set but does not parse to `0..=100` is an operator - // error. Fail CLOSED: never enforce a tolerance the operator did not - // actually specify. - let tolerance_present_but_invalid = tolerance_raw.is_some() && valid_tolerance.is_none(); - - if tolerance_present_but_invalid { - if enforce_requested { - crate::logging::error!( - "{PRICE_FLOOR_TOLERANCE_ENV}={tolerance_raw:?} is not an integer in 0..=100; \ - refusing to enforce the price floor with an unspecified tolerance. \ - Price-floor enforcement DISABLED (shadow mode). Fix the value and restart \ - to enable enforcement." - ); - } else { - crate::logging::warn!( - "{PRICE_FLOOR_TOLERANCE_ENV}={tolerance_raw:?} is not an integer in 0..=100; \ - using the default {PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT}% for shadow-mode \ - telemetry." - ); - } - } - - Self { - enforce: enforce_requested && !tolerance_present_but_invalid, - tolerance_percent: valid_tolerance.unwrap_or(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT), - } - } -} - #[derive(Clone, Copy)] struct LegacyMedianCandidate<'a> { encoded_peer_id: &'a evmlib::EncodedPeerId, @@ -444,8 +358,6 @@ pub struct PaymentVerifierConfig { /// Kept in the verifier config for payment policies that bind receipts to /// this node's payout address. pub local_rewards_address: RewardsAddress, - /// Receiver-side price floor policy (shadow mode by default). - pub price_floor: PriceFloorConfig, } /// What a payment authorizes, and where it routes. @@ -523,8 +435,12 @@ impl PaymentTarget { /// touching direct-PUT behaviour. /// - **`PaidListAdmission`** — the node is admitting fresh paid-list metadata. /// It runs the same live payment checks, but writes a weaker cache entry -/// that does not authorize future chunk stores. The price floor never -/// applies here: paid-list records reprice no fresh economic decision. +/// that does not authorize future chunk stores. The price floor applies +/// here too: a `PaidNotify` announces a write a client has just made, with +/// the proof the client presented (on a payment-cache hit, whatever proof it +/// attached, as before), and a paid-list vote is what later lets repair copy +/// the data, so exempting it would let one accepting node replicate a +/// below-floor write to its whole group. /// /// The caller must check local receiver/admission membership before invoking /// the verifier for replication admission: fresh chunk replication requires @@ -740,11 +656,6 @@ pub struct PaymentVerifier { /// entirely rather than pricing against anything, so a node in that state /// never rejects. local_commitment_source: RwLock>>, - /// Live price-floor policy, initialized from - /// [`PaymentVerifierConfig::price_floor`]. Behind a lock so tests (and any - /// future ops surface) can flip enforcement without rebuilding the - /// verifier. - price_floor: RwLock, /// ADR-0008: the instant from which a merkle receipt must settle at the /// single-node 3x multiplier, defaulting to /// [`crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX`]. @@ -769,6 +680,17 @@ pub struct PaymentVerifier { /// Incremented only on first emission for a pool, so it measures /// settlements sampled rather than chunks admitted. merkle_parity_emissions: std::sync::atomic::AtomicUsize, + /// ADR-0006: chunk addresses whose payment this verifier admitted while the + /// price floor contested it (see [`Self::verified_payment_contested`]). + /// Written during a full verification, before the payment cache learns + /// the chunk, so every later record of it, a cache hit included, is judged + /// by the proof that was verified and never by whatever proof came with + /// it. Only contested admissions enter, and a chunk is recorded within + /// moments of its verification, so the fixed + /// [`PRICE_CONTESTED_MEMORY_CAPACITY`], whatever the payment cache is sized + /// to, loses a decision before it is read only if that many newer + /// contested admissions arrive in between. + verified_price_contested: Mutex>, /// Configuration. config: PaymentVerifierConfig, } @@ -858,6 +780,9 @@ impl PaymentVerifier { "pool cache capacity must be > 0" ); let cache = VerifiedCache::with_capacity(config.cache_capacity); + let verified_price_contested = Mutex::new(LruCache::new( + NonZeroUsize::new(PRICE_CONTESTED_MEMORY_CAPACITY).unwrap_or(NonZeroUsize::MIN), + )); let pool_cache_size = NonZeroUsize::new(DEFAULT_POOL_CACHE_CAPACITY).unwrap_or(NonZeroUsize::MIN); let pool_cache = Mutex::new(LruCache::new(pool_cache_size)); @@ -897,7 +822,6 @@ impl PaymentVerifier { ))), monetized_pin_tx: RwLock::new(None), local_commitment_source: RwLock::new(None), - price_floor: RwLock::new(config.price_floor), merkle_parity_from: RwLock::new( crate::replication::config::MERKLE_PARITY_ENFORCED_FROM_UNIX, ), @@ -906,6 +830,7 @@ impl PaymentVerifier { .unwrap_or(NonZeroUsize::MIN), )), merkle_parity_emissions: std::sync::atomic::AtomicUsize::new(0), + verified_price_contested, config, } } @@ -986,17 +911,82 @@ impl PaymentVerifier { debug!("PaymentVerifier: local commitment source attached for price-floor checks"); } - /// The live price-floor policy this verifier applies. - #[must_use] - pub fn price_floor_config(&self) -> PriceFloorConfig { - *self.price_floor.read() + /// Whether this node must not judge its close-group peers on a write routed + /// at `routing` and paid by `proof`: a peer may have refused it under its + /// own floor, and refusing a below-floor payment is network policy, never + /// misbehaviour (ADR-0006). + /// + /// True when the floor cannot price the write here (no own commitment, or + /// too thin a view), or when its paid price is below + /// [`PRICE_FLOOR_CONTESTED_PERCENT`] of the UNCAPPED group median: the + /// question is what a peer whose own price sits above this node's may have + /// refused. The paid quote's price stands in for a third of the settled + /// amount: settlement is at least that, so an overpaid write can only be + /// judged more leniently, never less. Evaluated from the proof as it + /// stands, verified or not: this is what a fan-out uses, since the peers + /// it offers the proof to verify it themselves and may refuse it. + /// + /// False for a merkle proof, which the floor never refuses, for a proof + /// that does not parse (its peers refuse it on structure, as before), and + /// where no floor is wired at all (no cache or no routing view). + pub(crate) async fn price_contested(&self, routing: &XorName, proof: &[u8]) -> bool { + let Some(paid_price) = Self::single_node_paid_price(proof) else { + return false; + }; + let (reference, sample) = self.group_reference_price(routing).await; + Self::contested(paid_price, reference, sample) } - /// Test-only setter for the price-floor policy, so enforcement and - /// tolerance can be exercised without environment variables. + /// Whether this node admitted a payment for the chunk at `address` that + /// the price floor contested when it verified it. + /// + /// Unlike [`Self::price_contested`], this only ever reflects a proof whose + /// signature and settlement this node checked: a write admitted later from + /// the payment cache reads the decision taken then, so a proof attached to + /// a cache hit, which nobody checked, can neither set nor veto it. + pub(crate) fn verified_payment_contested(&self, address: &XorName) -> bool { + self.verified_price_contested.lock().contains(address) + } + + /// Test-only stand-in for a verified admission of the chunk at `address` + /// that the price floor contested, for tests that cannot settle payments + /// on chain. #[cfg(any(test, feature = "test-utils"))] - pub fn set_price_floor_for_tests(&self, config: PriceFloorConfig) { - *self.price_floor.write() = config; + pub fn note_contested_admission_for_tests(&self, address: &XorName) { + self.verified_price_contested.lock().put(*address, ()); + } + + /// The contest decision for a write paid at `paid_price`, from what the + /// floor could price it against. + fn contested( + paid_price: Amount, + reference: Option, + sample: GroupSample, + ) -> bool { + reference.map_or( + matches!( + sample.skip_reason, + Some(SkipReason::NoOwnCommitment | SkipReason::ThinSample) + ), + |reference| paid_price < Self::contested_price(reference.group_median), + ) + } + + /// The paid (median) quote's price in a single-node proof, or `None` for a + /// merkle proof or one that does not parse. + fn single_node_paid_price(proof: &[u8]) -> Option { + if !matches!(detect_proof_type(proof), Some(ProofType::SingleNode)) { + return None; + } + let parsed = deserialize_single_node_proof(proof).ok()?; + let candidates = Self::legacy_median_candidates(&parsed.proof_of_payment).ok()?; + candidates.first().map(|candidate| candidate.quote.price) + } + + /// The paid (per-quote) price below which a write is contested. + fn contested_price(group_median: Amount) -> Amount { + group_median.saturating_mul(Amount::from(PRICE_FLOOR_CONTESTED_PERCENT)) + / Amount::from(100u64) } /// Test-only setter for the ADR-0008 merkle parity boundary, so both the @@ -1475,9 +1465,9 @@ impl PaymentVerifier { Self::validate_quote_arithmetic(payment)?; let candidates = Self::legacy_median_candidates(payment)?; let mut failures = Vec::with_capacity(candidates.len()); - // The paid (median) price and settled on-chain amount of the winning - // candidate, kept for the receiver-side price-floor policy below. - let mut verified_paid_quote: Option<(Amount, Amount)> = None; + // The winning candidate's quote and settled on-chain amount, kept for + // the receiver-side price floor below. + let mut verified_paid_quote: Option<(&PaymentQuote, Amount)> = None; // ADR-0004 Amendment 2: remember WHICH candidate's on-chain settlement // verified — only that peer's commitment actually earned money, so only // its pin is nominated for a deterministic first audit below. The @@ -1486,11 +1476,10 @@ impl PaymentVerifier { let mut paid_peer: Option<[u8; 32]> = None; for candidate in candidates { - let paid_price = candidate.quote.price; let candidate_peer = *candidate.encoded_peer_id.as_bytes(); match self.verify_legacy_median_candidate(target, candidate).await { Ok(settled_amount) => { - verified_paid_quote = Some((paid_price, settled_amount)); + verified_paid_quote = Some((candidate.quote, settled_amount)); // First settlement-verified median candidate wins the paid // slot and the sole first-audit nomination. If a client // settled several TIED-median candidates, only this one is @@ -1504,7 +1493,7 @@ impl PaymentVerifier { } } - let Some((paid_price, settled_amount)) = verified_paid_quote else { + let Some((paid_quote, settled_amount)) = verified_paid_quote else { let xorname_hex = hex::encode(xorname); let details = if failures.is_empty() { "no median-priced candidates were available".to_string() @@ -1516,17 +1505,24 @@ impl PaymentVerifier { ))); }; - // Receiver-side price floor (single-node store admissions only). Runs - // AFTER the paid quote's signature and settlement verified above, so - // unauthenticated bundles can never poison floor telemetry. Shadow - // mode logs; enforcement rejects — an economic admission decision - // only, never trust/misbehaviour evidence. - // The floor compares against the median commitment price of the close - // group that is RESPONSIBLE for the data, so it takes the routing - // address. For a chunk that is the same value; for a pointer the paid - // content is its state, which names no close group at all. - self.enforce_price_floor(target.routing(), paid_price, settled_amount, context) + // Receiver-side price floor, in every context. Runs AFTER the paid + // quote's signature and settlement verified above, so unauthenticated + // bundles can never poison floor telemetry. A rejection is an economic + // admission decision only, never trust/misbehaviour evidence. + // The floor prices against the close group that is RESPONSIBLE for the + // data, so it reads the target's routing address. For a chunk that is + // the content address; for a pointer the paid content is its state, + // which names no close group at all. + let contested = self + .enforce_price_floor(target, paid_quote, settled_amount, context) .await?; + // Remembered before the payment cache learns the chunk, so any record + // of it from a cache hit reads this decision (see + // `verified_payment_contested`). Only ever set: a later payment does + // not unsay an earlier one. + if let (true, PaymentTarget::Chunk(address)) = (contested, target) { + self.verified_price_contested.lock().put(*address, ()); + } // ADR-0004 observe-only telemetry: log off-curve quotes only AFTER the // paid (median) quote's ML-DSA-65 signature has verified above, so @@ -1679,70 +1675,52 @@ impl PaymentVerifier { .collect() } - /// The close group's MEDIAN commitment-bound price for `xorname`, and the - /// number of peers that median was taken over. - /// - /// This is the price floor's reference, and choosing the median of the group - /// rather than this node's own price is the whole point of the policy. - /// - /// A client pays `3x` the MEDIAN of the quotes it collected. Pricing the - /// floor against a receiver's OWN price compares that median to a single - /// draw from a distribution that spans ~4x across the fleet, so the fullest - /// receivers reject honest payments while the emptiest admit deep - /// underpayment — measured on ant-prod-01, 0.377% of honest stores rejected - /// with the cheapest-of-K underpayment still admitted by 62.9% of receivers. - /// Both sides of that are the same defect: the reference is a sample, and - /// the payment is a median. Comparing median to median removes it. + /// The price floor's reference for `routing`: the close group's MEDIAN + /// commitment-bound price, capped at this node's own, plus the sample the + /// median was taken over. /// - /// The reference is built ONLY from state this node already holds: + /// **Why the median.** A client pays `3x` the MEDIAN of the quotes it + /// collected. Pricing the floor against a receiver's OWN price compares + /// that median to a single draw from a distribution that spans ~4x across + /// the fleet, so the fullest receivers reject honest payments while the + /// emptiest admit deep underpayment — measured on ant-prod-01, 0.377% of + /// honest stores rejected with the cheapest-of-K underpayment still + /// admitted by 62.9% of receivers. Comparing median to median removes that. /// - /// * the K peers closest to `xorname` in its own routing table, and - /// * those peers' own signed commitments from the gossip cache, TTL-gated - /// by [`GOSSIP_ANSWERABILITY_TTL`], plus this node's own live commitment. + /// **Why the cap.** The median is taken over gossiped commitments, and a + /// signature proves who said a number, never that the number is true. + /// Neighbours that OVERSTATE their counts push the median up and would + /// make this node refuse payments that are already settled and cannot be + /// refunded. Capping the reference at this node's own price bounds it by + /// the one number no neighbour controls: a payment of `3x tolerance x` + /// this node's own price always clears here, however many neighbours lie. + /// While the gossip is honest an above-median receiver prices against the + /// median exactly as before. The cost is detection: a receiver below the + /// median admits payments down to the tolerance of its own, lower, price. + /// Neighbours that UNDERSTATE still pull the median down, which costs a + /// missed underpayment, the accepted direction. ADR-0006 Amendment 2 + /// carries both bounds. /// - /// Nothing in the payment bundle feeds it. A client cannot pad, prune or - /// reorder quotes to move this number, which is what makes the floor - /// resistant to the quote-selection games it exists to stop. + /// The reference is built ONLY from state this node already holds: the K + /// peers closest to `routing` in its own routing table, those peers' signed + /// commitments from the gossip cache, TTL-gated by + /// [`GOSSIP_ANSWERABILITY_TTL`], and this node's own live commitment. + /// Nothing in the payment bundle feeds it, so a client cannot pad, prune or + /// reorder quotes to move it. Counts above `MAX_COMMITMENT_KEY_COUNT` are + /// dropped, since gossip ingest authenticates the sender without bounding + /// the count. /// - /// The remaining lever is gossiping a false commitment, and it cuts BOTH - /// ways, which is why the sample gate is set where it is: - /// - /// * **Understating** drags the median down and weakens the floor. The cost - /// is a missed underpayment. - /// * **Overstating** drags it up and makes this node reject settled, honest - /// payments. That is the dangerous direction, because the money is already - /// spent and cannot be refunded, so a griefer would be spending nothing to - /// destroy someone else's payment. - /// - /// The bound is NOT "a majority of the group", and it is worth stating - /// precisely because it is what an enforcement decision rests on. The floor - /// does not need the median to reach the liars' value — only to move past - /// the tolerance headroom. Each liar shifts the order statistic by one rank, - /// and on the measured fleet spread one rank is worth roughly 15% of price - /// while a 65% tolerance buys 1.54x. So **about a quarter of the sample** - /// suffices in either direction: measured on this crate's own fixture, 4 of - /// 15 overstating neighbours reject an honest payment sitting at the true - /// median, and 4 of 15 understating ones (placed above the median, where an - /// attacker would put them) admit a cheapest-of-K underpayment. - /// - /// Counts above `MAX_COMMITMENT_KEY_COUNT` are dropped outright, since - /// gossip ingest authenticates the sender without bounding the count. That - /// caps how far a single liar can reach but does not change the cardinality - /// bound above. A signature proves who said a number, never that the number - /// is true, and address grinding into a close group is a known adjacent - /// problem. This is why the policy ships shadow-only: the ~25% bound is a - /// reason not to enforce yet, not a property to rely on. - /// - /// Returns the reference price only when at least + /// Returns a reference only when at least /// [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] neighbour commitments are - /// fresh AND this node has its own live commitment; the `usize` is the - /// sample size either way, so the caller can log how close it came. A `None` - /// price means "do not evaluate". - /// - /// [`GOSSIP_ANSWERABILITY_TTL`]: crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL - async fn group_reference_price(&self, xorname: &XorName) -> (Option, GroupSample) { + /// fresh AND this node has its own live commitment; the sample comes back + /// either way, so the caller can log how close it came. `None` means "do + /// not evaluate". + async fn group_reference_price( + &self, + routing: &XorName, + ) -> (Option, GroupSample) { // This node's own live commitment. It is in the group whenever it is - // being asked to store the chunk, so it belongs in the sample. Bound to + // being asked to store the data, so it belongs in the sample. Bound to // a local first so the source lock is released before the awaits below. // // It is counted separately from the neighbour tally: the gate below asks @@ -1752,13 +1730,13 @@ impl PaymentVerifier { // Wiring faults are reported before state faults, so a startup-order bug // is never mistaken for "this node is mid-rotation" or for "the gate is // above what the network supplies". Those three have three different - // fixes and shadow mode exists to tell them apart. + // fixes and the telemetry exists to tell them apart. let Some(cache) = self.commitment_cache.read().as_ref().map(Arc::clone) else { return (None, GroupSample::none(SkipReason::NoCommitmentCache)); }; // Cache is checked first so a node without one does not pay for a // routing-table scan it will discard. - let closest = self.local_k_closest_peer_ids(xorname).await; + let closest = self.local_k_closest_peer_ids(routing).await; if closest.is_empty() { return (None, GroupSample::none(SkipReason::NoRoutingView)); } @@ -1772,8 +1750,11 @@ impl PaymentVerifier { let Some(own) = own_source.and_then(|source| source.current_binding_snapshot()) else { return (None, GroupSample::none(SkipReason::NoOwnCommitment)); }; - let now = std::time::Instant::now(); - let ttl = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL; + // The routing scan includes this node, which the gossip cache never + // holds (ingest only accepts senders in the routing table, and the + // routing table never holds self), so its own commitment joins the + // sample exactly once, below. + let now = Instant::now(); let mut key_counts: Vec = { let guard = cache.read().await; closest @@ -1783,7 +1764,8 @@ impl PaymentVerifier { // Stale gossip is treated as unknown rather than as a low // price, so a peer going quiet can never drag the median // down, nor count toward the minimum sample. - if now.saturating_duration_since(record.received_at) >= ttl { + if now.saturating_duration_since(record.received_at) >= GOSSIP_ANSWERABILITY_TTL + { return None; } record @@ -1827,72 +1809,100 @@ impl PaymentVerifier { .get((sample.total.saturating_sub(1)) / 2) .copied() .unwrap_or_default(); - let reference = calculate_price(usize::try_from(median_key_count).unwrap_or_default()); + let group_median = calculate_price(usize::try_from(median_key_count).unwrap_or_default()); + // The price this node quotes right now, from the same commitment its + // `QuoteGenerator` binds quotes to. + let own_price = calculate_price(usize::try_from(own.key_count).unwrap_or_default()); + let reference = FloorReference { + price: group_median.min(own_price), + group_median, + own: own_price, + }; (Some(reference), sample) } - /// Receiver-side price floor for single-node store admissions. + /// Receiver-side price floor for single-node payments, in every context. /// /// Requires the settled on-chain amount to also clear - /// `PAID_QUOTE_PAYMENT_MULTIPLIER × tolerance% ×` the close group's median - /// commitment-bound price (see [`Self::group_reference_price`]). Together - /// with the per-quote check above, the effective rule is - /// `settled >= 3 × max(paid_price, tolerated_floor)`. + /// `PAID_QUOTE_PAYMENT_MULTIPLIER × tolerance% ×` the reference price (see + /// [`Self::group_reference_price`]). Together with the per-quote check + /// above, the effective rule is `settled >= 3 × max(paid_price, + /// tolerated_floor)`. + /// + /// It applies to direct PUTs, fresh replication and paid-list admission + /// alike. Any one of them left open would let a single accepting node + /// spread a below-floor write to its whole group: fresh replication copies + /// it directly, and a paid-list vote lets repair copy it later. /// - /// When the group reference cannot be built — too few fresh commitments, no - /// P2P handle, no gossip cache — the floor SKIPS the admission rather than - /// falling back to this node's own price. Falling back is what the previous - /// revision did, and it is precisely the comparison that rejected honest - /// stores on the fullest nodes. A skipped evaluation is logged with its - /// real `neighbours` / `group_sample` counts and a `skip_reason`, so shadow + /// When the reference cannot be built — too few fresh commitments, no own + /// commitment, no P2P handle, no gossip cache — the floor SKIPS the + /// admission rather than guessing. A skipped evaluation is logged with its + /// real `neighbours` / `group_sample` counts and a `skip_reason`, so the /// telemetry can separate an unwired node from one whose gate simply sits /// above what the network supplies. /// + /// Returns, for an admitted payment, whether it is `contested`: the + /// decision [`Self::price_contested`] makes from the same reference. A + /// node that stores a contested write does not judge its peers on it. The + /// line logs it too. + /// /// Always emits one telemetry line per evaluated admission (target - /// `ant_node::payment::price_floor`) so shadow mode measures the exact - /// rejections enforcement would cause. Rejects only when - /// [`PriceFloorConfig::enforce`] is set. + /// `ant_node::payment::price_floor`), and rejects every admission below + /// the floor. The line keeps a constant `enforce=true`, which separates + /// these nodes from releases that only logged (`enforce=false`) while the + /// fleet upgrades. async fn enforce_price_floor( &self, - xorname: &XorName, - paid_price: Amount, + target: &PaymentTarget, + paid_quote: &PaymentQuote, settled_amount: Amount, context: VerificationContext, - ) -> Result<()> { - if !context.is_store_admission() { - return Ok(()); - } - - let floor = *self.price_floor.read(); - let tolerance_percent = floor.tolerance_percent.min(100); - - let (reference, group_sample) = self.group_reference_price(xorname).await; - let Some(reference_price) = reference else { + ) -> Result { + let routing = target.routing(); + let paid_price = paid_quote.price; + // Only read by the telemetry, which compiles to nothing without the + // `logging` feature. + #[cfg_attr(not(feature = "logging"), allow(unused_variables))] + let target_kind = match target { + PaymentTarget::Chunk(_) => "chunk", + PaymentTarget::Pointer { .. } => "pointer", + }; + #[cfg_attr(not(feature = "logging"), allow(unused_variables))] + let paid_quote_age_secs = SystemTime::now() + .duration_since(paid_quote.timestamp) + .map_or(0, |age| age.as_secs()); + + let (reference, group_sample) = self.group_reference_price(routing).await; + let contested = Self::contested(paid_price, reference, group_sample); + let Some(reference) = reference else { // Not enough of the close group is known to price anything. Skip // rather than guess: see `group_reference_price`. The reason and - // both counts are logged because shadow telemetry has to separate a + // both counts are logged because the telemetry has to separate a // mid-rotation node from an unwired cache from a gate set above what // the network can supply — three causes with three different fixes. info!( target: "ant_node::payment::price_floor", "Price floor: context={context:?}, xorname={}, paid_price={paid_price}, \ settled={settled_amount}, neighbours={}, group_sample={}, reference_price=0, \ - tolerance_percent={tolerance_percent}, required=0, below_floor=false, \ - skipped=true, skip_reason={}, enforce={}", - hex::encode(xorname), + tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}, required=0, below_floor=false, \ + skipped=true, skip_reason={}, enforce=true, target={target_kind}, \ + group_median_price=0, own_price=0, paid_quote_age_secs={paid_quote_age_secs}, \ + contested={contested}", + hex::encode(routing), group_sample.neighbours, group_sample.total, group_sample.skip_reason_str(), - floor.enforce, ); - return Ok(()); + return Ok(contested); }; - // Amount is a U256: reference_price is bounded by the saturating pricing - // curve and the multipliers are <= 300, so this arithmetic cannot + // Amount is a U256: prices are bounded by the saturating pricing curve + // and every multiplier here is at most 100, so this arithmetic cannot // overflow; saturating keeps that a proof instead of an assumption. - let tolerated_price = - reference_price.saturating_mul(Amount::from(tolerance_percent)) / Amount::from(100u64); + let tolerated_price = reference + .price + .saturating_mul(Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT)) + / Amount::from(100u64); let required_amount = tolerated_price.saturating_mul(Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER)); let below_floor = settled_amount < required_amount; @@ -1901,25 +1911,29 @@ impl PaymentVerifier { target: "ant_node::payment::price_floor", "Price floor: context={context:?}, xorname={}, paid_price={paid_price}, \ settled={settled_amount}, neighbours={}, group_sample={}, \ - reference_price={reference_price}, tolerance_percent={tolerance_percent}, \ + reference_price={}, tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}, \ required={required_amount}, below_floor={below_floor}, skipped=false, \ - skip_reason=none, enforce={}", - hex::encode(xorname), + skip_reason=none, enforce=true, target={target_kind}, group_median_price={}, \ + own_price={}, paid_quote_age_secs={paid_quote_age_secs}, contested={contested}", + hex::encode(routing), group_sample.neighbours, group_sample.total, - floor.enforce, + reference.price, + reference.group_median, + reference.own, ); - if below_floor && floor.enforce { + if below_floor { return Err(Error::Payment(format!( "Settled payment {settled_amount} is below the close-group price floor \ - {required_amount} ({PAID_QUOTE_PAYMENT_MULTIPLIER}x {tolerance_percent}% of \ - the group median price {reference_price}, taken over {} peers)", - group_sample.total + {required_amount} ({PAID_QUOTE_PAYMENT_MULTIPLIER}x {PRICE_FLOOR_TOLERANCE_PERCENT}% of \ + the reference price {}: the lower of the group median {} over {} peers and \ + this node's own price {})", + reference.price, reference.group_median, group_sample.total, reference.own ))); } - Ok(()) + Ok(contested) } fn validate_paid_quote_content( @@ -3770,7 +3784,7 @@ mod tests { use saorsa_core::MlDsa65; use saorsa_pqc::pqc::types::MlDsaSecretKey; use saorsa_pqc::pqc::MlDsaOperations; - use std::time::SystemTime; + use std::time::{Duration, SystemTime}; /// Create a verifier for unit tests. EVM is always on, but tests can /// pre-populate the cache to bypass on-chain verification. @@ -3835,7 +3849,6 @@ mod tests { cache_capacity: 100, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([1u8; 20]), - price_floor: PriceFloorConfig::default(), }; PaymentVerifier::new(config) } @@ -4507,36 +4520,32 @@ mod tests { } } - /// Receiver commitment count for floor tests, chosen so the local price is - /// comfortably more than double the baseline (the 50% floor then rejects a - /// baseline-priced settlement). Asserted in each test so a pricing-curve - /// change fails loudly instead of silently weakening the tests. - const FLOOR_TEST_LOCAL_KEY_COUNT: u32 = 1_000_000; + /// Receiver commitment count for floor tests: the protocol maximum, far + /// above every group member, so the own-price cap never binds in the tests + /// that are about the group median. The cap has tests of its own, built on + /// [`floor_test_verifier_with_own_count`]. + const FLOOR_TEST_LOCAL_KEY_COUNT: u32 = MAX_COMMITMENT_KEY_COUNT; - fn floor_test_verifier(enforce: bool) -> PaymentVerifier { + fn floor_test_verifier() -> PaymentVerifier { + floor_test_verifier_with_own_count(FLOOR_TEST_LOCAL_KEY_COUNT) + } + + /// A verifier whose own live commitment holds `own_key_count` keys, at the + /// default tolerance. + fn floor_test_verifier_with_own_count(own_key_count: u32) -> PaymentVerifier { let verifier = create_test_verifier(); verifier.attach_local_commitment_source(Arc::new(FixedFloorSource { - key_count: FLOOR_TEST_LOCAL_KEY_COUNT, + key_count: own_key_count, })); - verifier.set_price_floor_for_tests(PriceFloorConfig { - enforce, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - }); - let local_price = price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize); - assert!( - local_price > price_at_records(0) * Amount::from(2u64), - "floor tests need a local price above twice baseline, got {local_price}" - ); verifier } - /// The settled amount the floor requires: 3x the tolerated fraction of the - /// close group's median commitment-bound price. - fn group_floor_required_amount() -> Amount { - let tolerated = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) - / Amount::from(100u64); - tolerated * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER) + /// The tolerated fraction of the close group's median commitment-bound + /// price: a quote priced below this fails the floor. + fn tolerated_group_floor_price() -> Amount { + price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) + / Amount::from(100u64) } /// One cheap (baseline-priced) quote paid `settled`: the cheapest-of-K @@ -4557,23 +4566,34 @@ mod tests { /// /// Exactly [`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`] entries, so these tests /// sit on the gate rather than comfortably past it: drop one and the floor - /// must skip. Spread widely enough that the cheapest sits below the tolerated - /// fraction of the median (asserted in the rejection test), and kept in the - /// low thousands because each entry builds a real signed merkle commitment. + /// must skip. Shaped like a real close group: members measured in August + /// spanned about 1.4x in price around a fleet median near 4,000 keys, and + /// this one spans 1.9x in keys (2.9x in price), so the cheapest member is + /// shown to clear the floor with room to spare. Kept in the low thousands because each + /// entry builds a real signed merkle commitment. const GROUP_TEST_KEY_COUNTS: [u32; 15] = [ - 500, 800, 1_100, 1_400, 1_700, 2_000, 2_300, 2_600, 2_900, 3_200, 3_500, 3_800, 4_100, - 4_400, 4_700, + 3_000, 3_200, 3_400, 3_600, 3_800, 4_000, 4_200, 4_400, 4_600, 4_800, 5_000, 5_200, 5_400, + 5_600, 5_800, ]; /// The LOWER median of [`GROUP_TEST_KEY_COUNTS`] plus this node's own /// [`FLOOR_TEST_LOCAL_KEY_COUNT`]. /// - /// Sixteen samples, so the lower median is the eighth smallest: `2_600`. The + /// Sixteen samples, so the lower median is the eighth smallest: `4_400`. The /// receiver's own count is an extreme outlier at the top and does not move /// it, which is exactly the robustness a median buys. An even sample is /// deliberate — it is the case where upper and lower median differ, and the /// lower one is what production safety depends on. - const GROUP_TEST_MEDIAN_KEY_COUNT: u32 = 2_600; + const GROUP_TEST_MEDIAN_KEY_COUNT: u32 = 4_400; + + /// A quoter holding under a quarter of the group median: the near-empty + /// node a modified client pays instead of the median. Its price sits well + /// below the floor, while the cheapest real group member sits well above. + const GROUP_TEST_NEAR_EMPTY_KEY_COUNT: usize = 1_000; + + /// A quote that clears the 30% floor but sits below the 0.6x contest + /// threshold: about half the group median price. + const GROUP_TEST_NEAR_FLOOR_KEY_COUNT: usize = 2_800; /// Build a gossip commitment cache holding one fresh commitment per entry of /// `key_counts`, attach it, and return the peer ids so the caller can put @@ -4688,10 +4708,10 @@ mod tests { /// does not move it. #[tokio::test] async fn group_reference_price_is_the_close_group_median() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD0u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); verifier.set_paid_quote_k_closest_for_tests(neighbours); let (reference, sample) = verifier.group_reference_price(&xorname).await; @@ -4712,13 +4732,19 @@ mod tests { "a fully populated group must not report a skip reason" ); assert_eq!( - reference, + reference.group_median, price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize), - "the reference must be the group median price, not this node's own" + "the median must be taken over the whole group" ); - assert!( - reference < price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize), - "this node's own price is far above the median and must not be the reference" + assert_eq!( + reference.own, + price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize), + "this node's own price is the price of its own commitment" + ); + assert_eq!( + reference.price, reference.group_median, + "this node's own price is far above the median, so the cap does not bind \ + and the reference is the group median" ); } @@ -4731,10 +4757,10 @@ mod tests { /// of them on the fullest nodes. #[tokio::test] async fn price_floor_prices_against_the_group_median_not_the_receiver() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD1u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let median_price = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize); let own_price = price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize); @@ -4763,23 +4789,23 @@ mod tests { ); } - /// The attack the floor exists to stop: collect the neighbourhood's quotes, - /// settle only the cheapest one, submit it as a one-quote proof. + /// What the floor does NOT catch, stated as a test so nobody reads more + /// into it: settling the cheapest real member of the close group. Members + /// measured in August spanned about 1.4x in price, so the cheapest sits no + /// lower than about 0.7x the median, far above any tolerance that is safe + /// for honest traffic. #[tokio::test] - async fn price_floor_rejects_cheapest_of_group_payment() { - let verifier = floor_test_verifier(true); + async fn price_floor_admits_the_cheapest_member_of_a_realistic_group() { + let verifier = floor_test_verifier(); let xorname = [0xD2u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let cheapest = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0) as usize; let cheapest_price = price_at_records(cheapest); - let tolerated = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) - / Amount::from(100u64); assert!( - cheapest_price < tolerated, - "the cheapest group price must sit below the tolerated floor for this test to bite" + cheapest_price >= tolerated_group_floor_price(), + "the cheapest real member must clear the floor for this test to mean anything" ); let proof_bytes = group_floor_proof( @@ -4790,10 +4816,43 @@ mod tests { &neighbours, ); + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("shopping inside the close group is not what the floor stops"), + PaymentStatus::PaymentVerified + ); + } + + /// The underpayment the floor exists to stop: collect the neighbourhood's + /// quotes, settle only a near-empty quoter, submit it as a one-quote proof. + #[tokio::test] + async fn price_floor_rejects_a_near_empty_quoter() { + let verifier = floor_test_verifier(); + let xorname = [0xD7u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let near_empty_price = price_at_records(GROUP_TEST_NEAR_EMPTY_KEY_COUNT); + assert!( + near_empty_price < tolerated_group_floor_price(), + "the near-empty quote must sit below the tolerated floor for this test to bite" + ); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_NEAR_EMPTY_KEY_COUNT, + three_times(near_empty_price), + &neighbours, + ); + let err = verifier .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) .await - .expect_err("settling only the cheapest group quote must not clear the floor"); + .expect_err("settling a near-empty quoter must not clear the floor"); assert!( format!("{err}").contains("below the close-group price floor"), @@ -4801,20 +4860,359 @@ mod tests { ); } + /// The cap: a receiver below the group median prices against its own, + /// lower, price, so it never demands more than the tolerance of what it + /// would itself quote. The settlement here fails the median-only rule and + /// clears the capped one, so the cap is what decides it. + #[tokio::test] + async fn price_floor_caps_the_reference_at_this_nodes_own_price() { + let own_count = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0); + let verifier = floor_test_verifier_with_own_count(own_count); + let xorname = [0xD8u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours.clone()); + + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + let own_price = price_at_records(own_count as usize); + assert_eq!(reference.own, own_price); + assert!( + reference.group_median > own_price, + "this receiver must sit below its group's median for the cap to bind" + ); + assert_eq!( + reference.price, own_price, + "the reference must be capped at this node's own price" + ); + + // A quote priced between the tolerance of the capped reference and the + // tolerance of the median. + let quote_records = 1_200; + let quote_price = price_at_records(quote_records); + let tolerance = Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT); + assert!( + quote_price >= own_price * tolerance / Amount::from(100u64) + && quote_price < reference.group_median * tolerance / Amount::from(100u64), + "the quote must fail the median-only floor and clear the capped one" + ); + let proof_bytes = group_floor_proof( + &verifier, + xorname, + quote_records, + three_times(quote_price), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("a receiver must not refuse what clears the tolerance of its own price"), + PaymentStatus::PaymentVerified + ); + } + + /// A write paid close to the floor is contested: it clears the floor here, + /// but not the contest threshold, so a peer with a higher view of the + /// group may have refused it. + #[tokio::test] + async fn a_write_near_the_floor_is_contested() { + let verifier = floor_test_verifier(); + let xorname = [0xD9u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let quote_price = price_at_records(GROUP_TEST_NEAR_FLOOR_KEY_COUNT); + let contest_price = PaymentVerifier::contested_price(price_at_records( + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + )); + assert!( + quote_price >= tolerated_group_floor_price() && quote_price < contest_price, + "the quote must clear the floor but not the contest margin" + ); + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_NEAR_FLOOR_KEY_COUNT, + three_times(quote_price), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("a payment above the floor must be admitted"), + PaymentStatus::PaymentVerified + ); + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write inside the contest margin must not be used to judge peers" + ); + assert!( + verifier.verified_payment_contested(&xorname), + "the verified admission must be remembered as contested" + ); + + // A later admission from the payment cache reads that decision, so a + // proof attached to it, here one the floor would not contest, can + // neither set nor veto it. + let median_proof = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + let cached = verifier + .verify_payment( + &xorname, + Some(&median_proof), + VerificationContext::PaidListAdmission, + ) + .await + .expect("a cached payment is admitted"); + assert_eq!(cached, PaymentStatus::CachedAsVerified); + assert!(verifier.verified_payment_contested(&xorname)); + } + + /// A write paid at the group median is not contested: this node keeps + /// judging its peers on it exactly as before. + #[tokio::test] + async fn a_write_at_the_group_median_is_not_contested() { + let verifier = floor_test_verifier(); + let xorname = [0xE3u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect("an honest median payment must clear the floor"), + PaymentStatus::PaymentVerified + ); + assert!( + !verifier.price_contested(&xorname, &proof_bytes).await, + "a payment at the group median must leave possession checks in place" + ); + assert!(!verifier.verified_payment_contested(&xorname)); + // A merkle or unparseable proof is never refused on price. + assert!(!verifier.price_contested(&xorname, &[0u8; 8]).await); + } + + /// The memory of contested admissions has its own capacity, so a payment + /// cache configured small cannot make it forget a decision before the key + /// is recorded. + #[test] + fn contested_admissions_outlive_a_small_payment_cache() { + let verifier = PaymentVerifier::new(PaymentVerifierConfig { + evm: EvmVerifierConfig::default(), + cache_capacity: 1, + close_group_size: CLOSE_GROUP_SIZE, + local_rewards_address: RewardsAddress::new([1u8; 20]), + }); + let keys = [[0xF1u8; 32], [0xF2u8; 32], [0xF3u8; 32]]; + for key in &keys { + verifier.note_contested_admission_for_tests(key); + } + assert!(keys + .iter() + .all(|key| verifier.verified_payment_contested(key))); + } + + /// A node with no floor wired (no gossip cache, or no routing view) refuses + /// nothing on price, so it judges its peers on every write as before. + #[tokio::test] + async fn a_node_with_no_floor_wired_contests_nothing() { + let xorname = [0xEDu8; 32]; + let (peer_id, quote) = make_signed_quote(xorname, price_at_records(0), 1); + let proof_bytes = serialize_proof(vec![(peer_id, quote)]); + + let verifier = create_test_verifier(); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + + verifier.attach_commitment_cache(CommitmentCache::default()); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + } + + /// A fan-out, chunk or pointer, judges the proof it forwards against the + /// view when it fans out, so a write fanned out after the group's prices + /// rose is judged against the view peers will price it against now. (A + /// chunk's paid-list mark keeps the decision its verification made.) + #[tokio::test] + async fn the_contest_follows_the_current_view() { + let verifier = floor_test_verifier(); + let xorname = [0xE7u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), + &neighbours, + ); + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); + + // Every neighbour's commitment grows to twice the old median, as after + // heavy churn: the same proof is now close to the floor. + forge_key_counts( + &verifier, + Some(&neighbours), + GROUP_TEST_MEDIAN_KEY_COUNT * 2, + ) + .await; + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write that peers would now price near the floor must not be used to judge them" + ); + } + + /// The contest threshold is 60% of the UNCAPPED group median, to the atto: + /// a write paid at it is judged and one paid an atto below is not. Written + /// out rather than read from the constant, since moving it is an ADR + /// decision. + #[tokio::test] + async fn contest_threshold_is_sixty_percent_of_the_group_median() { + let verifier = floor_test_verifier(); + let xorname = [0xEBu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours); + let threshold = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize) + * Amount::from(60u64) + / Amount::from(100u64); + let proof_at = |price: Amount| { + let (peer_id, quote) = make_signed_quote(xorname, price, 1); + serialize_proof(vec![(peer_id, quote)]) + }; + + assert!( + !verifier + .price_contested(&xorname, &proof_at(threshold)) + .await + ); + assert!( + verifier + .price_contested(&xorname, &proof_at(threshold - Amount::from(1u64))) + .await + ); + } + + /// A receiver whose own price caps its floor still contests against the + /// group median: a peer whose own price is higher may refuse what this + /// node's lower floor admitted. + #[tokio::test] + async fn contest_reads_the_uncapped_median_when_the_cap_binds() { + let verifier = floor_test_verifier_with_own_count(3_000); + let xorname = [0xECu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + verifier.set_paid_quote_k_closest_for_tests(neighbours); + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + assert!( + reference.price < reference.group_median, + "the cap must bind for this test to mean anything" + ); + + // Above the contest threshold of the capped reference, below that of + // the median. + let price = (PaymentVerifier::contested_price(reference.price) + + PaymentVerifier::contested_price(reference.group_median)) + / Amount::from(2u64); + let (peer_id, quote) = make_signed_quote(xorname, price, 1); + let proof_bytes = serialize_proof(vec![(peer_id, quote)]); + + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "the contest must be judged against the median, not the capped reference" + ); + } + + /// A pointer payment is floored against the close group of the pointer's + /// address, and its fan-out is contested the same way as a chunk's. + #[tokio::test] + async fn price_floor_floors_and_contests_a_pointer_state() { + let verifier = floor_test_verifier(); + let address = [0xE4u8; 32]; + let state_id = [0xE5u8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + + let quote_price = price_at_records(GROUP_TEST_NEAR_FLOOR_KEY_COUNT); + let proof_bytes = group_floor_proof( + &verifier, + state_id, + GROUP_TEST_NEAR_FLOOR_KEY_COUNT, + three_times(quote_price), + &neighbours, + ); + + verifier + .verify_pointer_payment_in( + &address, + &state_id, + &proof_bytes, + VerificationContext::ClientPut, + ) + .await + .expect("a pointer payment above the floor must be admitted"); + + assert!( + verifier.price_contested(&address, &proof_bytes).await, + "a pointer write inside the contest margin must not be used to judge peers" + ); + + // And a near-empty quoter is refused for a pointer exactly as for a chunk. + let cheap_state = [0xE6u8; 32]; + let cheap_proof = group_floor_proof( + &verifier, + cheap_state, + GROUP_TEST_NEAR_EMPTY_KEY_COUNT, + three_times(price_at_records(GROUP_TEST_NEAR_EMPTY_KEY_COUNT)), + &neighbours, + ); + let err = verifier + .verify_pointer_payment_in( + &address, + &cheap_state, + &cheap_proof, + VerificationContext::ClientPut, + ) + .await + .expect_err("a near-empty quoter must not pay for a pointer state either"); + assert!(format!("{err}").contains("below the close-group price floor")); + } + /// The anti-manipulation property: the reference price is built from this /// node's own routing view and the neighbours' signed gossip, never from the /// bundle. Padding the bundle with cheap quotes (the median-padding shape) /// must not move the floor. #[tokio::test] async fn price_floor_reference_ignores_client_supplied_quotes() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD3u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); - // A bundle stuffed with baseline-priced quotes: the client's own median - // is now the floor price, which is exactly what it would want. - let cheapest = GROUP_TEST_KEY_COUNTS.first().copied().unwrap_or(0) as usize; + // A bundle stuffed with near-empty quotes: the client's own median is + // now the cheap price, which is exactly what it would want. + let cheapest = GROUP_TEST_NEAR_EMPTY_KEY_COUNT; let mut peer_quotes = Vec::new(); let mut k_closest: Vec<[u8; 32]> = neighbours.clone(); for seed in 1..=CLOSE_GROUP_SIZE { @@ -4852,7 +5250,7 @@ mod tests { /// counts (the obvious choice) would pass either way and prove nothing. #[tokio::test] async fn price_floor_skips_when_group_view_is_too_thin() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD4u8; 32]; let thin: Vec = GROUP_TEST_KEY_COUNTS .iter() @@ -4861,7 +5259,7 @@ mod tests { .take(PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS - 1) .collect(); assert_eq!(thin.len(), PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS - 1); - let neighbours = attach_group_commitments(&verifier, &thin, std::time::Duration::ZERO); + let neighbours = attach_group_commitments(&verifier, &thin, Duration::ZERO); // Deep underpayment that this sample, if it were used, would reject. let settled = three_times(price_at_records(0)); @@ -4869,7 +5267,7 @@ mod tests { assert!( settled < price_at_records(would_be_median as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) / Amount::from(100u64) * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER), "the thin sample must price ABOVE the settlement, so deleting the gate fails this test" @@ -4884,13 +5282,21 @@ mod tests { result.expect("a thin group view must skip the floor, not reject"), PaymentStatus::PaymentVerified ); + assert!( + verifier.price_contested(&xorname, &proof_bytes).await, + "a write this node cannot price must not be used to judge its peers" + ); + assert!( + verifier.verified_payment_contested(&xorname), + "a write admitted unpriced is remembered as contested" + ); } /// Exactly ON the gate must evaluate, so the boundary is pinned from both /// sides alongside the one-short test above. #[tokio::test] async fn price_floor_evaluates_at_exactly_the_minimum_sample() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDAu8; 32]; assert_eq!( GROUP_TEST_KEY_COUNTS.len(), @@ -4898,7 +5304,7 @@ mod tests { "the fixture must sit exactly on the gate for this boundary test" ); let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let proof_bytes = group_floor_proof( &verifier, @@ -4915,7 +5321,8 @@ mod tests { assert!(format!("{err}").contains("below the close-group price floor")); } - /// A receiver with no live commitment of its own must skip. + /// A receiver with no live commitment of its own must skip, and leave its + /// peers unjudged on the write. /// /// Such a node prices its OWN quotes at baseline, so holding an incoming /// payment to its neighbours' higher median would reject settlements it @@ -4924,13 +5331,9 @@ mod tests { #[tokio::test] async fn price_floor_skips_when_this_node_has_no_commitment() { let verifier = create_test_verifier(); - verifier.set_price_floor_for_tests(PriceFloorConfig { - enforce: true, - tolerance_percent: PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT, - }); let xorname = [0xDBu8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let proof_bytes = group_floor_proof( &verifier, @@ -4948,6 +5351,9 @@ mod tests { result.expect("a receiver with no commitment of its own must skip the floor"), PaymentStatus::PaymentVerified ); + // It cannot price the write, so it cannot tell whether its peers + // refused it either, and must not judge them on it. + assert!(verifier.price_contested(&xorname, &proof_bytes).await); } /// A neighbour gossiping a count no honest node could hold must be dropped, @@ -4959,18 +5365,19 @@ mod tests { /// floor becomes a way to reject other people's settled payments. #[tokio::test] async fn price_floor_ignores_commitments_above_the_protocol_cap() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDCu8; 32]; // The honest builder refuses to construct an over-cap commitment, so // forge one the way a modified node would: build a valid commitment and // overwrite the signed count. Dropping it leaves the sample one short of // the gate, so the floor skips — that cardinality change is what this // test detects if the filter is removed. One over-cap value alone would - // NOT move a 16-sample median; inflating the reference that way needs - // about a quarter of the sample, which is the cardinality bound on - // `group_reference_price` rather than something this filter fixes. + // NOT move a 16-sample median; inflating the reference that way needs a + // majority of the sample at the default tolerance, which is the + // cardinality bound on `group_reference_price` rather than something + // this filter fixes. let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let over_cap = crate::replication::commitment::MAX_COMMITMENT_KEY_COUNT + 1; { let cache = verifier @@ -5012,22 +5419,29 @@ mod tests { ); } - /// Pins the other KNOWN LIMIT: four of fifteen OVERSTATING neighbours push - /// the reference up far enough to reject an honest payment sitting at the - /// true group median. + /// Pins the KNOWN LIMIT for a receiver whose own price sits far above its + /// group, where the cap does not bind: overstating neighbours that make up + /// a MAJORITY of the sample reject an honest payment at the true median. /// /// This is the griefing direction, and the more serious of the two: the /// attacker spends nothing and the victim's money is already settled and - /// unrefundable. Like its understating sibling this test asserts a weakness - /// deliberately. If a future change improves the bound it will fail and must - /// be re-derived rather than deleted. + /// unrefundable. On this realistic fixture 65% falls to seven liars. At 30% + /// the headroom covers every honest member of the group, so liars lift the + /// median only as far as the dearest honest member until they outnumber + /// the rest: seven still clear (pinned by + /// `group_median_resists_seven_overstated_commitments`), eight take the + /// median. A receiver at or below the payment's price is immune at any + /// count (pinned by + /// `capped_reference_cannot_be_griefed_by_any_number_of_overstated_commitments`). + /// This test asserts the weakness deliberately. If a future change improves + /// the bound it will fail and must be re-derived rather than deleted. #[tokio::test] - async fn group_median_is_griefed_by_four_overstated_commitments() { - let verifier = floor_test_verifier(true); + async fn group_median_alone_is_griefed_by_a_majority_of_overstated_commitments() { + let verifier = floor_test_verifier(); let xorname = [0xE0u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - forge_key_counts(&verifier, neighbours.get(..4), MAX_COMMITMENT_KEY_COUNT).await; + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, neighbours.get(..8), MAX_COMMITMENT_KEY_COUNT).await; // A completely honest payment, priced at the TRUE group median. let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); @@ -5043,34 +5457,38 @@ mod tests { .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) .await .expect_err( - "KNOWN LIMIT: four overstated neighbours reject an honest at-median payment. \ - If this now succeeds, the bound improved — re-derive it and update the docs.", + "KNOWN LIMIT: a majority of overstated neighbours rejects an honest at-median \ + payment at a receiver far above its group. If this now succeeds, the bound \ + improved — re-derive it and update the docs.", ); assert!(format!("{err}").contains("below the close-group price floor")); } - /// Pins the KNOWN LIMIT, so it is a fact in the test suite rather than a - /// claim in a comment: four of fifteen understating neighbours DISARM the - /// floor, admitting the cheapest-of-K underpayment it exists to reject. + /// The griefing direction below a majority: seven of fifteen neighbours + /// gossiping the protocol maximum must not make even a receiver far above + /// its group refuse an honest, already settled payment. /// - /// This test asserts a weakness on purpose. It is the concrete reason - /// enforcement stays off, and if a future change improves the bound this - /// test will fail and must be re-derived rather than quietly deleted. + /// They take the seven LOWEST slots, which is where a liar moves the + /// median most, and lift the lower median from `4_400` to `5_800`, the + /// dearest honest member. The honest payment still clears 30% of that by + /// 2.07x (1.55x at 40%). Forged by overwriting the signed count rather than + /// building real 1,000,000-leaf commitments, which cost about a second and + /// 200 MB each. #[tokio::test] - async fn group_median_is_disarmed_by_four_understated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xDFu8; 32]; - let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); - for slot in counts.iter_mut().rev().take(4) { - *slot = 1; - } - let neighbours = attach_group_commitments(&verifier, &counts, std::time::Duration::ZERO); + async fn group_median_resists_seven_overstated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xDDu8; 32]; + let neighbours = + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, neighbours.get(..7), MAX_COMMITMENT_KEY_COUNT).await; + // An honest payment at the TRUE group median must still be admitted. + let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); let proof_bytes = group_floor_proof( &verifier, xorname, - 0, - three_times(price_at_records(0)), + GROUP_TEST_MEDIAN_KEY_COUNT as usize, + honest, &neighbours, ); @@ -5079,49 +5497,40 @@ mod tests { .await; assert_eq!( - result.expect( - "KNOWN LIMIT: four understated neighbours disarm the floor. If this now \ - rejects, the bound improved — re-derive it and update the docs." - ), + result.expect("a minority of overstated commitments must not reject honest payments"), PaymentStatus::PaymentVerified ); } - /// The griefing direction: a small minority gossiping OVERSTATED - /// commitments must not push the reference up far enough to make this node - /// reject an honest, already settled payment. - /// - /// This is the dangerous direction of a false commitment. Understating only - /// costs a missed underpayment; overstating destroys someone else's money, - /// and the attacker spends nothing to do it. + /// The cap's guarantee: no number of overstating neighbours can make a + /// receiver refuse a payment of `3x tolerance x` its own price. /// - /// The bound is real but partial, and worth stating precisely rather than - /// claiming immunity. Each liar shifts the order statistic up by one rank, - /// so what protects the payment is the tolerance headroom, not the median - /// itself. In this fixture three liars move the reference `2_600` -> - /// `3_500` and an at-median payment still clears, by only 1.9%; four move it - /// to `3_800` - /// and the same payment is REJECTED (pinned by - /// `group_median_is_griefed_by_four_overstated_commitments`). The griefing - /// bound is therefore about a quarter of the sample, the same as the - /// disarming one — NOT "roughly half", which was the incorrect claim this - /// policy originally rested on. - /// - /// Estimator coverage: this case fails under a 25th percentile, and its - /// disarming sibling fails under a mean and under the upper median, so the - /// pair pins the estimator jointly. Neither does so alone. + /// Every neighbour here claims the protocol maximum, so the group median is + /// the liars' value. A receiver that itself holds the group's true median + /// still admits the honest at-median payment, because its reference is its + /// own price. #[tokio::test] - async fn group_median_resists_three_overstated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xDDu8; 32]; - // Three of fifteen neighbours claim the protocol maximum. Forged by - // overwriting the signed count rather than building real 1,000,000-leaf - // commitments, which cost about a second and 200 MB each. + async fn capped_reference_cannot_be_griefed_by_any_number_of_overstated_commitments() { + let verifier = floor_test_verifier_with_own_count(GROUP_TEST_MEDIAN_KEY_COUNT); + let xorname = [0xE1u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - forge_key_counts(&verifier, neighbours.get(..3), MAX_COMMITMENT_KEY_COUNT).await; + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + forge_key_counts(&verifier, Some(&neighbours), MAX_COMMITMENT_KEY_COUNT).await; + + verifier.set_paid_quote_k_closest_for_tests(neighbours.clone()); + let (reference, _) = verifier.group_reference_price(&xorname).await; + let reference = reference.expect("a fully populated group must produce a reference"); + assert_eq!( + reference.group_median, + price_at_records(MAX_COMMITMENT_KEY_COUNT as usize), + "every neighbour lied, so the median is the liars' value" + ); + assert_eq!( + reference.price, + price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize), + "the reference must be capped at this node's own price" + ); - // An honest payment at the TRUE group median must still be admitted. let honest = three_times(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)); let proof_bytes = group_floor_proof( &verifier, @@ -5136,7 +5545,85 @@ mod tests { .await; assert_eq!( - result.expect("a minority of overstated commitments must not reject honest payments"), + result.expect("lying neighbours must not lift a receiver above its own price"), + PaymentStatus::PaymentVerified + ); + } + + /// Understated commitments pull the reference DOWN and weaken the floor. + /// That costs a missed underpayment rather than anyone's money. + /// + /// Six of fifteen neighbours claiming almost nothing leave the floor + /// standing against a baseline quote. The placement matters and is easy to + /// get wrong: understating a peer that was already below the median + /// changes nothing, so a test that rewrites the CHEAPEST entries passes + /// while testing nothing. These rewrite the DEAREST ones, which is where an + /// attacker would sit. `1`, not `0`: a commitment over an empty key set + /// cannot be built at all, so one key is the least a peer can commit to. + #[tokio::test] + async fn group_median_resists_six_understated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xD6u8; 32]; + let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); + for slot in counts.iter_mut().rev().take(6) { + *slot = 1; + } + let neighbours = attach_group_commitments(&verifier, &counts, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + 0, + three_times(price_at_records(0)), + &neighbours, + ); + + let err = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await + .expect_err("a minority of understated commitments must not disarm the floor"); + + assert!( + format!("{err}").contains("below the close-group price floor"), + "Error should name the group price floor: {err}" + ); + } + + /// Pins the KNOWN LIMIT, so it is a fact in the test suite rather than a + /// claim in a comment: seven understating neighbours of fifteen DISARM the + /// floor, admitting the baseline payment it exists to reject. The lower + /// median then falls to the cheapest honest member (3,000 keys), and 30% of + /// that price is below a baseline quote. At 40% it took eight. + /// + /// This test asserts a weakness on purpose. If a future change improves the + /// bound it will fail and must be re-derived rather than quietly deleted. + #[tokio::test] + async fn group_median_is_disarmed_by_seven_understated_commitments() { + let verifier = floor_test_verifier(); + let xorname = [0xDFu8; 32]; + let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); + for slot in counts.iter_mut().rev().take(7) { + *slot = 1; + } + let neighbours = attach_group_commitments(&verifier, &counts, Duration::ZERO); + + let proof_bytes = group_floor_proof( + &verifier, + xorname, + 0, + three_times(price_at_records(0)), + &neighbours, + ); + + let result = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await; + + assert_eq!( + result.expect( + "KNOWN LIMIT: seven understated neighbours of fifteen disarm the floor. If \ + this now rejects, the bound improved: re-derive it and update the docs." + ), PaymentStatus::PaymentVerified ); } @@ -5149,7 +5636,7 @@ mod tests { /// floor skips instead of pricing against this node's own (huge) count. #[tokio::test] async fn price_floor_ignores_stale_group_commitments() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xD5u8; 32]; let stale_age = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL + std::time::Duration::from_mins(1); @@ -5190,14 +5677,13 @@ mod tests { async fn price_floor_excludes_only_the_stale_entries_from_a_mixed_cache() { use crate::replication::commitment_state::PeerCommitmentRecord; - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xDEu8; 32]; let stale_age = crate::replication::commitment_state::GOSSIP_ANSWERABILITY_TTL + std::time::Duration::from_mins(1); // Fresh: the full fixture, exactly on the gate. - let fresh = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + let fresh = attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); // Stale: four cheap counts, keeping the total within the 19 neighbours a // real routing view can supply. @@ -5236,83 +5722,19 @@ mod tests { "stale entries must not count toward the gate" ); assert_eq!( - reference, + reference.map(|reference| reference.group_median), Some(price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize)), "the reference must be the fresh-only median; counting the stale cheap \ entries would drag it below" ); } - /// Understated commitments pull the reference DOWN and weaken the floor. - /// - /// Three adversarially-placed liars leave it standing; four disarm it. The - /// placement matters and is easy to get wrong: understating a peer that was - /// already below the median changes nothing, so a test that rewrites the - /// CHEAPEST entries passes while testing nothing. These rewrite the DEAREST - /// ones, which is where an attacker would sit, and the count is set to the - /// largest that still holds rather than to an aspirational majority. - #[tokio::test] - async fn group_median_resists_three_understated_commitments() { - let verifier = floor_test_verifier(true); - let xorname = [0xD6u8; 32]; - // Three of fifteen neighbours claim to store almost nothing. `1`, not - // `0`: a commitment over an empty key set cannot be built at all, so one - // key is the least a peer can honestly (or dishonestly) commit to. - let mut counts: Vec = GROUP_TEST_KEY_COUNTS.to_vec(); - for slot in counts.iter_mut().rev().take(3) { - *slot = 1; - } - let neighbours = attach_group_commitments(&verifier, &counts, std::time::Duration::ZERO); - - let cheapest_price = price_at_records(0); - let proof_bytes = group_floor_proof( - &verifier, - xorname, - 0, - three_times(cheapest_price), - &neighbours, - ); - - let err = verifier - .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await - .expect_err("a minority of understated commitments must not disarm the floor"); - - assert!( - format!("{err}").contains("below the close-group price floor"), - "Error should name the group price floor: {err}" - ); - } - - #[tokio::test] - async fn test_price_floor_shadow_mode_accepts_below_floor_settlement() { - let verifier = floor_test_verifier(false); - let xorname = [0xC1u8; 32]; - let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - let settled = three_times(price_at_records(0)); - assert!( - settled < group_floor_required_amount(), - "shadow mode is only meaningful on a settlement that WOULD be rejected" - ); - let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); - - let result = verifier - .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await; - - assert_eq!( - result.expect("shadow mode must never reject on the floor"), - PaymentStatus::PaymentVerified - ); - } - #[tokio::test] async fn test_price_floor_enforced_rejects_cheapest_of_k_settlement() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC2u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); @@ -5327,30 +5749,44 @@ mod tests { ); } + /// An honest client may OVERPAY a cheap quote to clear stricter receivers: + /// the floor compares the settled amount, not the quote price, and + /// exactly-at-floor must pass. The floor is 30% of the reference (ADR-0006 + /// Amendment 2), written out here rather than read from the constant, so + /// moving the tolerance either way, an ADR decision, fails here first. #[tokio::test] async fn test_price_floor_enforced_accepts_exactly_at_floor_overpayment() { - // An honest client may OVERPAY a cheap quote to clear stricter - // receivers: the floor compares the settled amount, not the quote - // price, and exactly-at-floor must pass. - let verifier = floor_test_verifier(true); - let xorname = [0xC3u8; 32]; + assert_eq!(PRICE_FLOOR_TOLERANCE_PERCENT, 30); + let verifier = floor_test_verifier(); let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); - let proof_bytes = group_floor_proof( - &verifier, - xorname, - 0, - group_floor_required_amount(), - &neighbours, - ); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); + let median_price = price_at_records(GROUP_TEST_MEDIAN_KEY_COUNT as usize); + let at_thirty_percent = + three_times(median_price * Amount::from(30u64) / Amount::from(100u64)); - let result = verifier + let xorname = [0xC3u8; 32]; + let proof_bytes = group_floor_proof(&verifier, xorname, 0, at_thirty_percent, &neighbours); + let status = verifier .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) - .await; + .await + .expect("exactly 3x 30% of the group median must clear the floor"); + assert_eq!(status, PaymentStatus::PaymentVerified); - assert_eq!( - result.expect("an exactly-at-floor settlement must pass"), - PaymentStatus::PaymentVerified + let xorname = [0xCAu8; 32]; + let one_short = at_thirty_percent - Amount::from(1u64); + let proof_bytes = group_floor_proof(&verifier, xorname, 0, one_short, &neighbours); + let err = verifier + .verify_payment(&xorname, Some(&proof_bytes), VerificationContext::ClientPut) + .await + .expect_err("one atto under 3x 30% of the group median must not clear the floor"); + assert!( + err.to_string() + .contains("below the close-group price floor"), + "{err}" + ); + assert!( + !verifier.verified_payment_contested(&xorname), + "a refused payment is never remembered as an admitted contested one" ); } @@ -5359,10 +5795,10 @@ mod tests { // Fresh replication is the same fresh economic event as a direct PUT; // exempting it would let one cheap accepting node fan the proof out // around every other storer's floor. - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC4u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); @@ -5381,26 +5817,31 @@ mod tests { ); } + /// Paid-list admission is floored like a store. A `PaidNotify` only ever + /// announces a write that was just paid for, and a paid-list vote is what + /// lets repair copy the data later, so exempting it would let one + /// accepting node replicate a below-floor write to its whole group. #[tokio::test] - async fn test_price_floor_never_reprices_paid_list_admission() { - let verifier = floor_test_verifier(true); + async fn test_price_floor_enforced_applies_to_paid_list_admission() { + let verifier = floor_test_verifier(); let xorname = [0xC5u8; 32]; let neighbours = - attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, std::time::Duration::ZERO); + attach_group_commitments(&verifier, &GROUP_TEST_KEY_COUNTS, Duration::ZERO); let settled = three_times(price_at_records(0)); let proof_bytes = group_floor_proof(&verifier, xorname, 0, settled, &neighbours); - let result = verifier + let err = verifier .verify_payment( &xorname, Some(&proof_bytes), VerificationContext::PaidListAdmission, ) - .await; + .await + .expect_err("a below-floor proof must not earn a paid-list vote"); - assert_eq!( - result.expect("paid-list admission reprices no fresh economic decision"), - PaymentStatus::PaymentVerified + assert!( + format!("{err}").contains("below the close-group price floor"), + "Error should name the price floor: {err}" ); } @@ -5414,13 +5855,13 @@ mod tests { /// nothing. #[tokio::test] async fn test_price_floor_without_any_group_view_skips() { - let verifier = floor_test_verifier(true); + let verifier = floor_test_verifier(); let xorname = [0xC6u8; 32]; let settled = three_times(price_at_records(0)); assert!( settled < price_at_records(FLOOR_TEST_LOCAL_KEY_COUNT as usize) - * Amount::from(PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT) + * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) / Amount::from(100u64) * Amount::from(PAID_QUOTE_PAYMENT_MULTIPLIER), "the old own-price rule would have rejected this settlement, so the skip is load-bearing" @@ -5435,42 +5876,9 @@ mod tests { result.expect("no group view must skip the floor, never reject"), PaymentStatus::PaymentVerified ); - } - - /// `from_env` must fail CLOSED: a present-but-invalid tolerance can never - /// leave enforcement on with a tolerance the operator did not specify. - /// (This test owns the `PRICE_FLOOR_*` env vars — no other test reads them.) - #[test] - fn price_floor_from_env_fails_closed_on_invalid_tolerance() { - // Defaults with nothing set. - std::env::remove_var(PRICE_FLOOR_ENFORCE_ENV); - std::env::remove_var(PRICE_FLOOR_TOLERANCE_ENV); - let cfg = PriceFloorConfig::from_env(); - assert!(!cfg.enforce); - assert_eq!(cfg.tolerance_percent, PRICE_FLOOR_DEFAULT_TOLERANCE_PERCENT); - - // Enforce on, valid tolerance: honoured. - std::env::set_var(PRICE_FLOOR_ENFORCE_ENV, "1"); - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "80"); - let cfg = PriceFloorConfig::from_env(); - assert!(cfg.enforce); - assert_eq!(cfg.tolerance_percent, 80); - - // Enforce on, out-of-range tolerance: enforcement DISABLED (fail closed). - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "150"); - let cfg = PriceFloorConfig::from_env(); - assert!( - !cfg.enforce, - "an out-of-range tolerance must not silently enforce a default" - ); - - // Enforce on, unparseable tolerance: also disabled. - std::env::set_var(PRICE_FLOOR_TOLERANCE_ENV, "loose"); - let cfg = PriceFloorConfig::from_env(); - assert!(!cfg.enforce); - - std::env::remove_var(PRICE_FLOOR_ENFORCE_ENV); - std::env::remove_var(PRICE_FLOOR_TOLERANCE_ENV); + // No cache means no floor is wired here at all, and so nothing to + // contest: possession checks stay as they were. + assert!(!verifier.price_contested(&xorname, &proof_bytes).await); } #[tokio::test] @@ -6988,7 +7396,6 @@ mod tests { cache_capacity: 100, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([1u8; 20]), - price_floor: PriceFloorConfig::default(), }; let verifier = PaymentVerifier::new(config); diff --git a/src/pointer/service.rs b/src/pointer/service.rs index 7d4c866f..8af79e91 100644 --- a/src/pointer/service.rs +++ b/src/pointer/service.rs @@ -561,7 +561,7 @@ fn cross_kind_refusal(address: XorName, chunk_present: Result) -> Option

, paid_list: &PaidList) -> Vec { + sampled + .into_iter() + .filter(|key| !paid_list.counts_as_price_contested(key)) + .collect() +} + // --------------------------------------------------------------------------- // Main audit tick // --------------------------------------------------------------------------- @@ -107,7 +120,7 @@ pub(crate) fn responsible_audit_response_timeout( /// /// This is the production path used by the replication engine. Direct /// callers that have not adopted repair proofs remain conservative and do not -/// audit peers for unproven keys. +/// audit peers for unproven keys, nor for price-contested ones (ADR-0006). #[allow( clippy::implicit_hasher, clippy::too_many_arguments, @@ -116,6 +129,7 @@ pub(crate) fn responsible_audit_response_timeout( pub async fn audit_tick_with_repair_proofs( p2p_node: &Arc, storage: &Arc, + paid_list: &PaidList, config: &ReplicationConfig, sync_history: &HashMap, repair_proofs: &Arc>, @@ -172,6 +186,7 @@ pub async fn audit_tick_with_repair_proofs( .copied() .collect() }; + let sampled_keys = keys_open_to_audit(sampled_keys, paid_list); // Step 4: Filter to keys where the chosen peer is in the close group and // this node has proof that it already sent the peer a repair hint for the @@ -901,6 +916,44 @@ mod tests { /// incoming audit limit (`2 * sqrt(N)`) never rejects small test challenges. const TEST_STORED_CHUNKS: usize = 1_000_000; + /// ADR-0006: the responsible audit never samples a key marked + /// price-contested, and samples every other key as before. + #[tokio::test] + async fn price_contested_keys_are_not_audited() { + let dir = TempDir::new().expect("temp dir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain: XorName = [0x71; 32]; + let contested: XorName = [0x72; 32]; + let unpaid: XorName = [0x73; 32]; + paid_list.insert(&plain).await.expect("insert"); + paid_list + .insert_price_contested(&contested) + .await + .expect("mark"); + + assert_eq!( + keys_open_to_audit(vec![plain, contested, unpaid], &paid_list), + vec![plain, unpaid] + ); + } + + /// ADR-0006: while the paid list cannot be read, the responsible audit + /// samples no key, since any of them may be marked. + #[tokio::test] + async fn an_unreadable_mark_keeps_a_key_out_of_the_audit() { + let dir = TempDir::new().expect("temp dir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain: XorName = [0x74; 32]; + let unpaid: XorName = [0x75; 32]; + paid_list.insert(&plain).await.expect("insert"); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert_eq!( + keys_open_to_audit(vec![plain, unpaid], &paid_list), + Vec::::new() + ); + } + #[test] fn first_challenged_key_label_truncates_to_16_hex_chars() { let mut key = [0u8; 32]; diff --git a/src/replication/fresh.rs b/src/replication/fresh.rs index af656cb2..6de7174c 100644 --- a/src/replication/fresh.rs +++ b/src/replication/fresh.rs @@ -11,7 +11,7 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; -use crate::logging::{debug, warn}; +use crate::logging::{debug, info, warn}; use bytes::Bytes; use rand::Rng; use saorsa_core::identity::PeerId; @@ -19,6 +19,7 @@ use saorsa_core::P2PNode; use tokio::sync::{mpsc, OwnedSemaphorePermit, Semaphore}; use crate::ant_protocol::XorName; +use crate::payment::PaymentVerifier; use crate::replication::config::{ ReplicationConfig, FRESH_REPLICATION_DELIVERY_MAX_RETRIES, REPLICATION_PROTOCOL_ID, }; @@ -48,6 +49,9 @@ pub struct FreshWriteEvent { pub(crate) struct FreshOfferEvent { pub(crate) key: XorName, pub(crate) payment_proof: Vec, + /// The fan-out must not judge peers on this write (see + /// [`announce_paid_write`]). + pub(crate) contested: bool, /// Storage read-backs attempted so far; see `MAX_FRESH_READ_ATTEMPTS`. pub(crate) read_attempts: u32, } @@ -84,8 +88,48 @@ impl AsRef<[u8]> for EncodedOffer { } } -/// Rules 6-8: record the paid key locally and announce it to -/// `PaidCloseGroup(K)`. +/// Rule 6: a node that validates `PoP` adds K to `PaidForList(self)`, marked +/// when `mark`. +/// +/// `mark` is the decision this node took when it verified the payment +/// (`PaymentVerifier::verified_payment_contested`), so a write +/// admitted from the payment cache is recorded by that decision, never by the +/// proof attached to it, which nobody checked. A contested write is marked +/// (ADR-0006) because a close-group peer may have refused it under its own +/// floor, and refusing a below-floor payment is network policy, never +/// misbehaviour. While this node holds the key, the mark keeps it from judging +/// its peers on it: the key goes out in paid hints only, so no repair proof is +/// recorded and no responsible audit samples it, and a prune audit of it +/// scores no one. Every other check that reads the mark points here. +/// +/// The first record of a key decides its mark: an existing entry is never +/// changed, so a later payment can neither add nor remove it. Returns whether +/// `mark` or the key's record is marked once this call is done, whichever +/// record won: ours, or one a concurrent `PaidNotify` or fresh offer wrote +/// first. A mark that cannot be read counts as set (see +/// `PaidList::counts_as_price_contested`). +pub(crate) async fn record_paid_key(key: &XorName, mark: bool, paid_list: &PaidList) -> bool { + let recorded = if mark { + paid_list.insert_price_contested(key).await + } else { + paid_list.insert(key).await + }; + if let Err(e) = recorded { + warn!("Failed to add key {} to PaidForList: {e}", hex::encode(key)); + } + mark || paid_list.counts_as_price_contested(key) +} + +/// Rules 6-8: record the paid key locally (see [`record_paid_key`]) and +/// announce it to `PaidCloseGroup(K)`. Returns whether the write's fan-out +/// must leave its peers unjudged (ADR-0006). +/// +/// The record is marked by the decision this node took when it verified the +/// payment. The fan-out is left unjudged when that decision or the key's +/// record is marked once recorded, whichever record won, so a marked write is +/// never judged, and also when the proof being forwarded is contested: a +/// cached PUT forwards it unverified, and its peers verify it and may refuse +/// it. /// /// This is the evidence peers need to repair the key later, so it runs the /// moment a write is accepted and is never gated by the pending-offer permit @@ -93,20 +137,28 @@ impl AsRef<[u8]> for EncodedOffer { pub(crate) async fn announce_paid_write( key: &XorName, proof_of_payment: &[u8], + payment_verifier: &PaymentVerifier, paid_list: &PaidList, p2p_node: &Arc, config: &ReplicationConfig, -) { - // Rule 6: Node that validates PoP adds K to PaidForList(self). - if let Err(e) = paid_list.insert(key).await { - warn!("Failed to add key {} to PaidForList: {e}", hex::encode(key)); - } +) -> bool { + let marked = record_paid_key( + key, + payment_verifier.verified_payment_contested(key), + paid_list, + ) + .await; // Rules 7-8: PaidNotify to every member of PaidCloseGroup(K). send_paid_notify(key, proof_of_payment, p2p_node, config).await; + marked + || payment_verifier + .price_contested(key, proof_of_payment) + .await } /// Rules 2-3: send fresh offers to the close group and schedule the delayed -/// possession check (ADR-0003) for the responsible peers. +/// possession check (ADR-0003) for the responsible peers, unless `contested` +/// (see [`announce_paid_write`]). /// /// `pending_offer` is the caller's permit from the pending-offer semaphore; /// it is held with the encoded offer until the last handle to it is dropped. @@ -117,6 +169,7 @@ pub(crate) async fn send_fresh_offers( key: &XorName, data: Vec, proof_of_payment: Vec, + contested: bool, pending_offer: OwnedSemaphorePermit, ) { let self_id = *ctx.p2p_node.peer_id(); @@ -210,12 +263,21 @@ pub(crate) async fn send_fresh_offers( // Schedule the delayed possession check (ADR-0003) for the responsible // close-group peers. A closed receiver (engine shutting down) is ignored. + // Not for a contested write: a peer may have refused it (ADR-0006). if !target_peers.is_empty() { ctx.dispatched.fetch_add(1, Ordering::Relaxed); - let _ = ctx.possession_check_tx.send(PossessionCheckEvent { - key: *key, - peers: target_peers, - }); + if contested { + info!( + "Fresh replication of {}: no possession check, the payment is contested \ + under the price floor", + hex::encode(key) + ); + } else { + let _ = ctx.possession_check_tx.send(PossessionCheckEvent { + key: *key, + peers: target_peers, + }); + } } } @@ -275,6 +337,52 @@ async fn send_paid_notify( mod tests { use super::*; + /// ADR-0006: a contested payment marks a newly recorded key, an + /// uncontested one records it plainly, and the first record of a key + /// decides: a later record, either way, changes nothing. Each call + /// reports the winning record's mark, or its own when that is set, so a + /// write whose plain decision lost to a marked record is still left + /// unjudged. + #[tokio::test] + async fn the_first_record_of_a_key_decides_its_mark() { + let dir = tempfile::tempdir().expect("tempdir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let contested = [0x91u8; 32]; + let plain = [0x92u8; 32]; + + let reported = vec![ + record_paid_key(&contested, true, &paid_list).await, + record_paid_key(&plain, false, &paid_list).await, + record_paid_key(&contested, false, &paid_list).await, + record_paid_key(&plain, true, &paid_list).await, + ]; + + let marks: Vec<(bool, bool)> = [contested, plain] + .iter() + .map(|key| { + ( + paid_list.contains(key).expect("contains"), + paid_list.is_price_contested(key).expect("read mark"), + ) + }) + .collect(); + assert_eq!(marks, vec![(true, true), (true, false)]); + assert_eq!(reported, vec![true, false, true, true]); + } + + /// ADR-0006: a plain write is left unjudged while the paid list cannot be + /// read, since the record that won may be marked. + #[tokio::test] + async fn a_write_whose_mark_cannot_be_read_is_left_unjudged() { + let dir = tempfile::tempdir().expect("tempdir"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let plain = [0x93u8; 32]; + assert!(!record_paid_key(&plain, false, &paid_list).await); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(record_paid_key(&plain, false, &paid_list).await); + } + /// The permit is released with the last handle to the encoded offer, not /// the first: a slice the transport still holds keeps the offer counted. #[test] diff --git a/src/replication/mod.rs b/src/replication/mod.rs index 62937413..2845c847 100644 --- a/src/replication/mod.rs +++ b/src/replication/mod.rs @@ -2487,11 +2487,14 @@ impl ReplicationEngine { /// /// Unlike the PUT path, this waits for a pending-offer permit before /// returning, and it offers the caller's bytes rather than reading them - /// back from storage. + /// back from storage. Test-only: production writes enter through the PUT + /// handler's fresh-write channel. + #[cfg(any(test, feature = "test-utils"))] pub async fn replicate_fresh(&self, key: &XorName, data: &[u8], proof_of_payment: &[u8]) { - fresh::announce_paid_write( + let contested = fresh::announce_paid_write( key, proof_of_payment, + &self.payment_verifier, &self.paid_list, &self.p2p_node, &self.config, @@ -2509,6 +2512,7 @@ impl ReplicationEngine { key, data.to_vec(), proof_of_payment.to_vec(), + contested, pending_offer, ) .await; @@ -2550,6 +2554,7 @@ impl ReplicationEngine { ) { let p2p = Arc::clone(&self.p2p_node); let paid_list = Arc::clone(&self.paid_list); + let payment_verifier = Arc::clone(&self.payment_verifier); let config = Arc::clone(&self.config); let shutdown = self.shutdown.clone(); @@ -2565,9 +2570,10 @@ impl ReplicationEngine { // Stage one never waits for chunk back-pressure: the paid-list // entry and PaidNotify are what let peers repair the key later, // so every queued write gets them at arrival rate. - fresh::announce_paid_write( + let contested = fresh::announce_paid_write( &event.key, &event.payment_proof, + &payment_verifier, &paid_list, &p2p, &config, @@ -2577,6 +2583,7 @@ impl ReplicationEngine { .send(fresh::FreshOfferEvent { key: event.key, payment_proof: event.payment_proof, + contested, read_attempts: 0, }) .is_err() @@ -2689,6 +2696,7 @@ impl ReplicationEngine { &event.key, data, event.payment_proof, + event.contested, pending_offer, ) .await; @@ -3539,6 +3547,7 @@ impl ReplicationEngine { fn start_audit_loop(&mut self) { let p2p = Arc::clone(&self.p2p_node); let storage = Arc::clone(&self.storage); + let paid_list = Arc::clone(&self.paid_list); let config = Arc::clone(&self.config); let shutdown = self.shutdown.clone(); let sync_history = Arc::clone(&self.sync_history); @@ -3573,6 +3582,7 @@ impl ReplicationEngine { audit::audit_tick_with_repair_proofs( &p2p, &storage, + &paid_list, &config, &history, &repair_proofs, @@ -3598,6 +3608,7 @@ impl ReplicationEngine { audit::audit_tick_with_repair_proofs( &p2p, &storage, + &paid_list, &config, &history, &repair_proofs, @@ -6602,10 +6613,15 @@ async fn handle_fresh_offer( } } - // Rule 6: add to PaidForList. - if let Err(e) = ctx.paid_list.insert(key).await { - warn!("Failed to add key to PaidForList: {e}"); - } + // Rule 6: add to PaidForList, marked if the price floor contested the + // payment when this node verified it, before the record is stored and + // can be hinted. + fresh::record_paid_key( + key, + ctx.payment_verifier.verified_payment_contested(key), + &ctx.paid_list, + ) + .await; // Store the record. match ctx.storage.put(key, data).await { @@ -6811,9 +6827,15 @@ async fn handle_paid_notify( } } - if let Err(e) = paid_list.insert(¬ify.key).await { - warn!("Failed to add paid notify key to PaidForList: {e}"); - } + // Marked if the price floor contested the payment when this node verified + // it, so a record later repaired onto this node is not used to judge its + // peers either. + fresh::record_paid_key( + ¬ify.key, + payment_verifier.verified_payment_contested(¬ify.key), + paid_list, + ) + .await; Ok(()) } diff --git a/src/replication/neighbor_sync.rs b/src/replication/neighbor_sync.rs index b1f0d859..d035f32f 100644 --- a/src/replication/neighbor_sync.rs +++ b/src/replication/neighbor_sync.rs @@ -13,6 +13,7 @@ use saorsa_core::identity::PeerId; use saorsa_core::P2PNode; use crate::ant_protocol::XorName; +use crate::error::Result; use crate::replication::config::{ReplicationConfig, REPLICATION_PROTOCOL_ID}; use crate::replication::paid_list::PaidList; use crate::replication::protocol::{ @@ -61,27 +62,50 @@ pub(crate) struct PeerSyncHints { /// This intentionally scans all locally stored records, not only records for /// which this node is still responsible. Out-of-range records retained by /// prune hysteresis therefore continue to repair their new close group before -/// this node is allowed to delete them. +/// this node is allowed to delete them. Price-contested records are left out +/// (ADR-0006). pub async fn build_replica_hints_for_peer( peer: &PeerId, storage: &Arc, + paid_list: &PaidList, p2p_node: &Arc, close_group_size: usize, ) -> Vec { - build_replica_hints_for_peer_with_close_groups(peer, storage, p2p_node, close_group_size) - .await - .into_iter() - .map(|hint| hint.key) - .collect() + build_replica_hints_for_peer_with_close_groups( + peer, + storage, + paid_list, + p2p_node, + close_group_size, + ) + .await + .into_iter() + .map(|hint| hint.key) + .collect() +} + +/// Stored keys this node may hint as replicas: every stored key except those +/// marked price-contested in the paid list (see `fresh::record_paid_key`), +/// which go out as paid hints only. A replica hint records a repair proof the +/// responsible audit later judges the peer on, and makes this node the sole +/// hinter a peer penalises when its quorum check fails; a paid hint does +/// neither. The marks are read from memory, so this scans nothing on disk; if +/// they could not be read when the paid list opened, no key is hinted as a +/// replica. +async fn replica_hint_keys(storage: &ChunkStore, paid_list: &PaidList) -> Result> { + let mut keys = storage.all_keys().await?; + paid_list.retain_unmarked(&mut keys); + Ok(keys) } pub(crate) async fn build_replica_hints_for_peer_with_close_groups( peer: &PeerId, storage: &Arc, + paid_list: &PaidList, p2p_node: &Arc, close_group_size: usize, ) -> Vec { - let all_keys = match storage.all_keys().await { + let all_keys = match replica_hint_keys(storage, paid_list).await { Ok(keys) => keys, Err(e) => { warn!("Failed to read stored keys for hint construction: {e}"); @@ -125,7 +149,7 @@ pub(crate) async fn build_sync_hints_for_peers( return hints_by_peer; } - let all_keys = match storage.all_keys().await { + let all_keys = match replica_hint_keys(storage, paid_list).await { Ok(keys) => keys, Err(e) => { warn!("Failed to read stored keys for batch hint construction: {e}"); @@ -521,6 +545,7 @@ pub(crate) async fn handle_sync_request_with_proofs( let sent_replica_hints = build_replica_hints_for_peer_with_close_groups( sender, storage, + paid_list, p2p_node, config.close_group_size, ) @@ -558,6 +583,7 @@ pub(crate) async fn handle_sync_request_with_proofs( mod tests { use super::*; use crate::replication::types::PeerSyncRecord; + use crate::storage::ChunkStoreConfig; use std::collections::HashMap; /// Build a `PeerId` from a single byte (zero-padded to 32 bytes). @@ -567,6 +593,44 @@ mod tests { PeerId::from_bytes(bytes) } + // -- replica_hint_keys --------------------------------------------------- + + /// ADR-0006: a stored key marked price-contested is never hinted as a + /// replica; every other stored key still is. The filter reads no paid-list + /// database, so it works while this thread cannot read one. + #[tokio::test] + async fn price_contested_keys_are_not_hinted_as_replicas() { + let dir = tempfile::tempdir().expect("tempdir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("chunks"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let contents: [&[u8]; 3] = [b"plain record", b"contested record", b"unpaid record"]; + let [plain, contested, unpaid] = contents.map(ChunkStore::compute_address); + for (key, content) in [plain, contested, unpaid].iter().zip(contents) { + storage.put(key, content).await.expect("put"); + } + paid_list.insert(&plain).await.expect("insert"); + paid_list + .insert_price_contested(&contested) + .await + .expect("mark"); + + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(paid_list.is_price_contested(&contested).is_err()); + let mut hinted = replica_hint_keys(&storage, &paid_list) + .await + .expect("read keys"); + hinted.sort_unstable(); + let mut expected = vec![plain, unpaid]; + expected.sort_unstable(); + + assert_eq!(hinted, expected); + } + // -- select_sync_batch --------------------------------------------------- #[test] diff --git a/src/replication/paid_list.rs b/src/replication/paid_list.rs index 26d01210..a6d9c27b 100644 --- a/src/replication/paid_list.rs +++ b/src/replication/paid_list.rs @@ -11,7 +11,11 @@ //! ``` //! //! One unnamed database stores set membership: key = 32-byte `XorName`, -//! value = empty byte slice. +//! value = empty byte slice, or one marker byte for a key whose +//! payment the price floor contested here (ADR-0006). Releases before that +//! marker only ever test presence, so they read a marked key as paid. The +//! marked keys are also held in memory, so hint construction never scans the +//! database for them. //! //! ## Out-of-range timestamps //! @@ -22,24 +26,70 @@ use crate::ant_protocol::XorName; use crate::error::{Error, Result}; -use crate::logging::{debug, trace, warn}; +use crate::logging::{debug, info, trace, warn}; use heed::types::Bytes; -use heed::{Database, Env, EnvOpenOptions}; -use parking_lot::RwLock; -use std::collections::HashMap; +#[cfg(test)] +use heed::WithTls; +use heed::{Database, Env, EnvOpenOptions, RoTxn}; +use parking_lot::{Mutex, RwLock}; +use std::collections::{HashMap, HashSet}; use std::path::Path; +use std::slice; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::Arc; use std::time::Instant; use tokio::task::spawn_blocking; use tokio_util::task::TaskTracker; use crate::ant_protocol::XORNAME_LEN; +/// Value stored for a key whose payment the price floor contested here (see +/// `fresh::record_paid_key`). +const PRICE_CONTESTED_VALUE: &[u8] = &[1]; + /// Default LMDB map size for the paid list: 256 MiB. /// -/// The paid list stores only 32-byte keys with empty values, so this is -/// generous even for very large close-group memberships. +/// The paid list stores only 32-byte keys with empty or one-byte values, so +/// this is generous even for very large close-group memberships. const DEFAULT_MAP_SIZE: usize = 256 * 1_024 * 1_024; +/// What the open task returns: the environment, the database and the keys +/// marked price-contested in it, if they could be read. +type OpenedPaidList = (Env, Database, Result>); + +/// Every key whose entry in `db` carries the price-contested mark. +fn read_price_contested_keys(db: &Database, txn: &RoTxn) -> Result> { + let mut keys = HashSet::new(); + for result in db + .iter(txn) + .map_err(|e| Error::Storage(format!("Failed to iterate paid-list: {e}")))? + { + let (key_bytes, value) = + result.map_err(|e| Error::Storage(format!("Failed to read paid-list entry: {e}")))?; + if value.is_empty() { + continue; + } + if let Ok(key) = XorName::try_from(key_bytes) { + keys.insert(key); + } + } + Ok(keys) +} + +/// Drop `keys` from the in-memory marks, after a commit that left none of +/// them marked in the database. The set is copied only if one of them is in +/// it. +fn unmark_in_memory(marks: &RwLock>>>, keys: &[XorName]) { + if let Some(marked) = marks.write().as_mut() { + if keys.iter().any(|key| marked.contains(key)) { + let marked = Arc::make_mut(marked); + for key in keys { + marked.remove(key); + } + } + } +} + /// Persistent paid-for-list backed by LMDB. /// /// Tracks which keys this node believes are paid-authorized. @@ -47,8 +97,27 @@ const DEFAULT_MAP_SIZE: usize = 256 * 1_024 * 1_024; pub struct PaidList { /// LMDB environment. env: Env, - /// The unnamed default database (key = `XorName` bytes, value = empty). + /// The unnamed default database (key = `XorName` bytes, value = empty, or + /// the price-contested marker). db: Database, + /// In-memory copy of the keys whose entry carries the price-contested + /// mark, or `None` if they could not be read on open. Changed only by the + /// blocking task that writes the database, under `write_order`, so a + /// cancelled caller cannot leave it stale. Readers clone the `Arc` and + /// filter outside the lock; a write while such a snapshot is alive + /// copies the set first (`Arc::make_mut`). + price_contested: Arc>>>>, + /// Held by every database write from before its write transaction until + /// it has updated `price_contested`, so the copy changes in commit order. + /// A new mark goes in just before its commit and a removed one just after, + /// so the copy never lacks a committed mark. A mark whose commit fails + /// stays, since that commit may still have reached the disk: when in + /// doubt, judge no one. Any later write that commits the key unmarked, a + /// plain insert that adds it or a removal, drops that mark. + write_order: Arc>, + /// Mark reads that have failed since the last one that succeeded, so a + /// run of failures is logged once (see `counts_as_price_contested`). + failed_mark_reads: AtomicU64, /// In-memory: when each paid key first went out of `PaidCloseGroup` range. /// Cleared on restart (safe: hysteresis clock restarts from zero). paid_out_of_range: RwLock>, @@ -85,7 +154,7 @@ impl PaidList { // Constructor-only blocking task: it runs before `self` (and its // `blocking_tracker`) exists, so it is deliberately untracked. The // constructor awaits it right here, so it cannot outlive this call. - let (env, db) = spawn_blocking(move || -> Result<(Env, Database)> { + let (env, db, price_contested) = spawn_blocking(move || -> Result { // SAFETY: `EnvOpenOptions::open()` is unsafe because LMDB uses // memory-mapped I/O and relies on OS file-locking to prevent // corruption from concurrent access by multiple processes. We @@ -110,15 +179,36 @@ impl PaidList { .map_err(|e| Error::Storage(format!("Failed to create paid-list database: {e}")))?; wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit db creation: {e}")))?; + // A failed read leaves its transaction unable to commit, so the + // scan gets one of its own, dropped uncommitted. A write + // transaction takes no reader slot on this thread. + let price_contested = env + .write_txn() + .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}"))) + .and_then(|scan| read_price_contested_keys(&db, &scan)); - Ok((env, db)) + Ok((env, db, price_contested)) }) .await .map_err(|e| Error::Storage(format!("Paid-list init task failed: {e}")))??; + // A paid list whose marks cannot be read still opens; the node then + // sends no replica hints until it restarts (see `retain_unmarked`). + let price_contested = price_contested + .inspect_err(|e| { + warn!( + "Cannot read the paid list's price-contested marks; sending no replica \ + hints until restart: {e}" + ); + }) + .ok() + .map(Arc::new); let paid_list = Self { env, db, + price_contested: Arc::new(RwLock::new(price_contested)), + write_order: Arc::new(Mutex::new(())), + failed_mark_reads: AtomicU64::new(0), paid_out_of_range: RwLock::new(HashMap::new()), record_out_of_range: RwLock::new(HashMap::new()), paid_prune_cursor: RwLock::new(0), @@ -139,6 +229,25 @@ impl PaidList { /// /// Returns an error if the LMDB write transaction fails. pub async fn insert(&self, key: &XorName) -> Result { + self.insert_with_value(key, &[]).await + } + + /// Insert `key` as paid at a price the floor contested here (see + /// `fresh::record_paid_key`). Like [`Self::insert`], an existing entry is + /// left exactly as it is, so a key already recorded can never gain the + /// mark later. The mark survives restarts and later inserts, and goes + /// with the entry on [`Self::remove`]. + /// + /// Returns `true` if the key was newly added, `false` if it already existed. + /// + /// # Errors + /// + /// Returns an error if the LMDB write transaction fails. + pub async fn insert_price_contested(&self, key: &XorName) -> Result { + self.insert_with_value(key, PRICE_CONTESTED_VALUE).await + } + + async fn insert_with_value(&self, key: &XorName, value: &'static [u8]) -> Result { // Fast-path: avoid write transaction if key already present. if self.contains(key)? { trace!("Paid-list key {} already present", hex::encode(key)); @@ -148,10 +257,13 @@ impl PaidList { let key_owned = *key; let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let was_new = self .blocking_tracker .spawn_blocking(move || -> Result { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -167,11 +279,20 @@ impl PaidList { return Ok(false); } - db.put(&mut wtxn, &key_owned, &[]) + db.put(&mut wtxn, &key_owned, value) .map_err(|e| Error::Storage(format!("Failed to insert into paid-list: {e}")))?; + if !value.is_empty() { + if let Some(keys) = price_contested.write().as_mut() { + Arc::make_mut(keys).insert(key_owned); + } + } wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list insert: {e}")) })?; + if value.is_empty() { + unmark_in_memory(&price_contested, slice::from_ref(&key_owned)); + } + drop(ordered); Ok(true) }) @@ -179,12 +300,87 @@ impl PaidList { .map_err(|e| Error::Storage(format!("Paid-list insert task failed: {e}")))??; if was_new { - debug!("Added key {} to paid-list", hex::encode(key)); + debug!( + "Added key {} to paid-list{}", + hex::encode(key), + if value.is_empty() { + "" + } else { + " as price-contested" + } + ); } Ok(was_new) } + /// Whether `key` is paid at a price the floor contested here (see + /// [`Self::insert_price_contested`]). + /// + /// Reads the database, which is authoritative: the in-memory copy behind + /// [`Self::retain_unmarked`] can hold a mark whose commit is under way or + /// failed, or be missing. A caller that has just seen a racing write's + /// entry sees its mark here as well (see `fresh::record_paid_key`). + /// + /// # Errors + /// + /// Returns an error if the LMDB read transaction fails. + pub fn is_price_contested(&self, key: &XorName) -> Result { + let rtxn = self + .env + .read_txn() + .map_err(|e| Error::Storage(format!("Failed to create read txn: {e}")))?; + let marked = self + .db + .get(&rtxn, key.as_ref()) + .map_err(|e| Error::Storage(format!("Failed to read paid-list entry: {e}")))? + .is_some_and(|value| !value.is_empty()); + Ok(marked) + } + + /// Whether `key` must be treated as price-contested: its entry carries the + /// mark, or the mark cannot be read (when in doubt, judge no one). While + /// reads fail no peer is judged on any key they hit, so the first failure + /// of a run is a warning, and the read that ends the run reports how many + /// failed. + pub fn counts_as_price_contested(&self, key: &XorName) -> bool { + match self.is_price_contested(key) { + Ok(marked) => { + let failed = self.failed_mark_reads.swap(0, Ordering::Relaxed); + if failed > 0 { + info!("Price-contested marks are readable again, after {failed} failed reads"); + } + marked + } + Err(e) => { + if self.failed_mark_reads.fetch_add(1, Ordering::Relaxed) == 0 { + warn!( + "Cannot read the price-contested mark of key {}; counting unreadable \ + marks as set until a read succeeds: {e}", + hex::encode(key) + ); + } else { + debug!( + "Price-contested mark of key {} still unreadable: {e}", + hex::encode(key) + ); + } + true + } + } + } + + /// Drop from `keys` every key marked price-contested, using the in-memory + /// copy rather than the database. If the marks could not be read on open, + /// no key is known to be unmarked, so every key is dropped. + pub fn retain_unmarked(&self, keys: &mut Vec) { + let snapshot = self.price_contested.read().clone(); + match snapshot { + Some(marked) => keys.retain(|key| !marked.contains(key)), + None => keys.clear(), + } + } + /// Remove a key from the paid-for set. /// /// Also clears any in-memory out-of-range timestamps for this key. @@ -198,10 +394,13 @@ impl PaidList { let key_owned = *key; let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let existed = self .blocking_tracker .spawn_blocking(move || -> Result { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -211,6 +410,8 @@ impl PaidList { wtxn.commit().map_err(|e| { Error::Storage(format!("Failed to commit paid-list delete: {e}")) })?; + unmark_in_memory(&price_contested, slice::from_ref(&key_owned)); + drop(ordered); Ok(deleted) }) .await @@ -415,10 +616,13 @@ impl PaidList { let keys_owned: Vec = keys.to_vec(); let env = self.env.clone(); let db = self.db; + let price_contested = Arc::clone(&self.price_contested); + let write_order = Arc::clone(&self.write_order); let removed_keys = self .blocking_tracker .spawn_blocking(move || -> Result> { + let ordered = write_order.lock(); let mut wtxn = env .write_txn() .map_err(|e| Error::Storage(format!("Failed to create write txn: {e}")))?; @@ -435,6 +639,10 @@ impl PaidList { wtxn.commit() .map_err(|e| Error::Storage(format!("Failed to commit batch remove: {e}")))?; + // Every key of the batch, not only those the database held: a + // mark whose commit failed can be in memory alone. + unmark_in_memory(&price_contested, &keys_owned); + drop(ordered); Ok(removed) }) @@ -480,18 +688,48 @@ impl PaidList { self.blocking_tracker.wait().await; self.blocking_tracker.reopen(); } + + /// Hold a read transaction on the calling thread. LMDB gives each thread + /// one reader slot, so until it is dropped every paid-list read on this + /// thread fails, which lets a test drive the paths a mark that cannot be + /// read takes. + #[cfg(test)] + pub(crate) fn hold_reader_for_tests(&self) -> Result> { + self.env + .read_txn() + .map_err(|e| Error::Storage(format!("Failed to create read txn: {e}"))) + } } #[cfg(test)] #[allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod tests { use super::*; + #[cfg(target_pointer_width = "64")] + use std::fs::OpenOptions; + #[cfg(target_pointer_width = "64")] + use std::io::{Read, Seek, SeekFrom, Write}; + use tokio::sync::oneshot; + use tokio::task::JoinHandle; + #[cfg(feature = "logging")] + use tracing::subscriber::with_default; + #[cfg(feature = "logging")] + use tracing_subscriber::fmt::writer::{MakeWriter, MutexGuardWriter}; /// Short probe used to prove `wait_idle` is still blocked on a parked op. const WAIT_IDLE_BLOCKED_PROBE: std::time::Duration = std::time::Duration::from_millis(200); /// Generous ceiling for `wait_idle` to complete once the op is released. const WAIT_IDLE_COMPLETE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(10); + /// The in-memory copy of the marks, which must have been read on open. + fn marked_in_memory(pl: &PaidList) -> HashSet { + pl.price_contested + .read() + .as_deref() + .cloned() + .expect("marks read on open") + } + async fn create_test_paid_list() -> (PaidList, tempfile::TempDir) { let temp_dir = tempfile::TempDir::new().expect("create temp dir"); let paid_list = PaidList::new(temp_dir.path()) @@ -572,6 +810,317 @@ mod tests { } } + /// ADR-0006: the price-contested mark is written only with a new entry, so + /// an already recorded key can never gain it. It survives a restart and a + /// later plain insert, and goes with the entry on removal. + #[tokio::test] + async fn price_contested_mark_persists_until_the_key_is_removed() { + let temp_dir = tempfile::TempDir::new().expect("create temp dir"); + let marked: XorName = [0x51; 32]; + let recorded_first: XorName = [0x52; 32]; + + { + let pl = PaidList::new(temp_dir.path()) + .await + .expect("create paid list"); + assert!(pl.insert(&recorded_first).await.expect("insert")); + assert!(pl.insert_price_contested(&marked).await.expect("mark")); + assert!(!pl + .insert_price_contested(&recorded_first) + .await + .expect("mark an existing key")); + assert!(!pl.insert(&marked).await.expect("re-insert")); + } + + let pl = PaidList::new(temp_dir.path()) + .await + .expect("reopen paid list"); + assert_eq!(pl.count().expect("count"), 2); + assert!(pl.contains(&marked).expect("contains")); + assert!(pl.is_price_contested(&marked).expect("read mark")); + assert!(!pl.is_price_contested(&recorded_first).expect("read mark")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + + pl.remove(&marked).await.expect("remove"); + pl.insert(&marked).await.expect("insert again"); + assert!(!pl.is_price_contested(&marked).expect("read mark")); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// ADR-0006: the in-memory marks follow every insert, removal and batch + /// removal. + #[tokio::test] + async fn the_in_memory_marks_follow_every_write() { + let (pl, _temp) = create_test_paid_list().await; + let first: XorName = [0x61; 32]; + let second: XorName = [0x62; 32]; + let plain: XorName = [0x63; 32]; + pl.insert_price_contested(&first).await.expect("mark"); + pl.insert_price_contested(&second).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + assert_eq!(marked_in_memory(&pl), HashSet::from([first, second])); + let mut keys = vec![first, plain, second]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, vec![plain]); + pl.remove(&first).await.expect("remove"); + assert_eq!(marked_in_memory(&pl), HashSet::from([second])); + pl.remove_batch(&[second, plain]) + .await + .expect("remove batch"); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// ADR-0006: while a database write holds the write order, a marked + /// insert or a batch removal neither commits nor changes the in-memory + /// copy, so writes and the copy move in one order. + #[tokio::test] + async fn the_in_memory_marks_change_in_commit_order() { + let (pl, _temp) = create_test_paid_list().await; + let key: XorName = [0x6B; 32]; + let (release_tx, holder) = hold_write_order(&pl).await; + + let insert = pl.insert_price_contested(&key); + tokio::pin!(insert); + let blocked = tokio::time::timeout(WAIT_IDLE_BLOCKED_PROBE, &mut insert).await; + assert!(blocked.is_err(), "the insert must wait for the write order"); + assert!(!pl.contains(&key).expect("contains")); + assert!(marked_in_memory(&pl).is_empty()); + + drop(release_tx); + holder.await.expect("holder task"); + assert!(insert.await.expect("insert")); + assert_eq!(marked_in_memory(&pl), HashSet::from([key])); + + let (release_tx, holder) = hold_write_order(&pl).await; + + let batch = [key]; + let removal = pl.remove_batch(&batch); + tokio::pin!(removal); + let blocked = tokio::time::timeout(WAIT_IDLE_BLOCKED_PROBE, &mut removal).await; + assert!( + blocked.is_err(), + "the removal must wait for the write order" + ); + assert!(pl.contains(&key).expect("contains")); + assert_eq!(marked_in_memory(&pl), HashSet::from([key])); + + drop(release_tx); + holder.await.expect("holder task"); + assert_eq!(removal.await.expect("remove batch"), 1); + assert!(marked_in_memory(&pl).is_empty()); + } + + /// Mark `key` in memory only, with no entry in the database: what a marked + /// insert whose commit failed before reaching the disk leaves. + fn mark_in_memory_only(pl: &PaidList, key: XorName) { + let mut marks = pl.price_contested.write(); + let marks = marks.as_mut().expect("marks read on open"); + Arc::make_mut(marks).insert(key); + } + + /// ADR-0006: a mark left in memory alone by a failed commit goes with a + /// later plain insert that adds the key, so the copy agrees with the + /// database again and the key is hinted as a replica. A plain insert of a + /// key whose entry carries the mark leaves both as they are. + #[tokio::test] + async fn a_plain_insert_drops_a_mark_held_in_memory_only() { + let (pl, _temp) = create_test_paid_list().await; + let stale: XorName = [0x71; 32]; + let marked: XorName = [0x72; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + mark_in_memory_only(&pl, stale); + let mut keys = vec![stale]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + + assert!(pl.insert(&stale).await.expect("insert")); + assert!(!pl.insert(&marked).await.expect("insert")); + assert!(!pl.is_price_contested(&stale).expect("read mark")); + assert!(pl.is_price_contested(&marked).expect("read mark")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + let mut keys = vec![stale, marked]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, vec![stale]); + } + + /// ADR-0006: a mark left in memory alone by a failed commit goes with a + /// later removal of the key, single or batched, although the database + /// holds no entry to delete. A mark the removal does not name stays. + #[tokio::test] + async fn a_removal_drops_a_mark_held_in_memory_only() { + let (pl, _temp) = create_test_paid_list().await; + let batched: XorName = [0x73; 32]; + let single: XorName = [0x74; 32]; + let marked: XorName = [0x75; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + mark_in_memory_only(&pl, batched); + mark_in_memory_only(&pl, single); + + assert_eq!(pl.remove_batch(&[batched]).await.expect("remove batch"), 0); + assert_eq!(marked_in_memory(&pl), HashSet::from([single, marked])); + assert!(!pl.remove(&single).await.expect("remove")); + assert_eq!(marked_in_memory(&pl), HashSet::from([marked])); + assert!(pl.is_price_contested(&marked).expect("read mark")); + } + + /// Byte offsets in an LMDB meta page on a 64-bit target: the magic number + /// follows the 16-byte page header, the free list's record keeps the page + /// size at 40, and the main database's root page number sits at 128. + #[cfg(target_pointer_width = "64")] + const LMDB_MAGIC_AT: u64 = 16; + #[cfg(target_pointer_width = "64")] + const LMDB_PAGE_SIZE_AT: u64 = 40; + #[cfg(target_pointer_width = "64")] + const LMDB_MAIN_ROOT_AT: u64 = 128; + + /// ADR-0006: a paid list whose marks cannot be read still opens, with no + /// marks in memory, so the node hints no key as a replica. Both meta pages + /// point the database's root past the end of the file, so the scan for + /// the marks fails as it would on a lost page. + #[cfg(target_pointer_width = "64")] + #[tokio::test] + async fn a_paid_list_whose_marks_cannot_be_read_still_opens() { + let temp_dir = tempfile::TempDir::new().expect("create temp dir"); + let marked: XorName = [0x70; 32]; + { + let pl = PaidList::new(temp_dir.path()) + .await + .expect("create paid list"); + pl.insert_price_contested(&marked).await.expect("mark"); + } + + let mut data = OpenOptions::new() + .read(true) + .write(true) + .open(temp_dir.path().join("paid_list.mdb").join("data.mdb")) + .expect("open data file"); + let mut read_u32 = |at: u64| { + let mut word = [0u8; 4]; + data.seek(SeekFrom::Start(at)).expect("seek"); + data.read_exact(&mut word).expect("read"); + u32::from_ne_bytes(word) + }; + let page_size = u64::from(read_u32(LMDB_PAGE_SIZE_AT)); + for meta in [0, page_size] { + assert_eq!( + read_u32(meta + LMDB_MAGIC_AT), + 0xBEEF_C0DE, + "LMDB meta page" + ); + } + for meta in [0, page_size] { + data.seek(SeekFrom::Start(meta + LMDB_MAIN_ROOT_AT)) + .expect("seek"); + data.write_all(&(1u64 << 40).to_ne_bytes()).expect("write"); + } + data.sync_all().expect("sync"); + drop(data); + + let pl = PaidList::new(temp_dir.path()) + .await + .expect("a paid list whose marks cannot be read must open"); + assert!(pl.price_contested.read().is_none()); + assert!(pl.is_price_contested(&marked).is_err()); + let mut keys = vec![marked]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + } + + /// ADR-0006: a paid list whose marks could not be read on open shows no + /// key as unmarked, so the node hints no key as a replica, while writes and + /// point reads of the marks still work. + #[tokio::test] + async fn marks_unread_on_open_hint_no_replicas() { + let (pl, _temp) = create_test_paid_list().await; + let marked: XorName = [0x6C; 32]; + let plain: XorName = [0x6D; 32]; + *pl.price_contested.write() = None; + + let mut keys = vec![plain]; + pl.retain_unmarked(&mut keys); + assert_eq!(keys, Vec::::new()); + pl.insert_price_contested(&marked).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + pl.remove_batch(&[plain]).await.expect("remove batch"); + assert!(pl.price_contested.read().is_none()); + assert!(pl.counts_as_price_contested(&marked)); + assert!(!pl.counts_as_price_contested(&plain)); + } + + /// ADR-0006: a key counts as price-contested when it is marked, or when + /// its mark cannot be read; an unmarked or unpaid key that reads fine + /// does not. + #[tokio::test] + async fn an_unreadable_mark_counts_as_price_contested() { + let (pl, _temp) = create_test_paid_list().await; + let marked: XorName = [0x64; 32]; + let plain: XorName = [0x65; 32]; + let unpaid: XorName = [0x66; 32]; + pl.insert_price_contested(&marked).await.expect("mark"); + pl.insert(&plain).await.expect("insert"); + + assert!(pl.counts_as_price_contested(&marked)); + assert!(!pl.counts_as_price_contested(&plain)); + assert!(!pl.counts_as_price_contested(&unpaid)); + + let _reader = pl.hold_reader_for_tests().expect("hold reader"); + assert!(pl.counts_as_price_contested(&plain)); + assert!(pl.counts_as_price_contested(&unpaid)); + } + + /// In-memory sink a test subscriber writes its lines to. + #[cfg(feature = "logging")] + #[derive(Clone, Default)] + struct LogSink(Arc>>); + + #[cfg(feature = "logging")] + impl<'a> MakeWriter<'a> for LogSink { + type Writer = MutexGuardWriter<'a, Vec>; + + fn make_writer(&'a self) -> Self::Writer { + self.0.make_writer() + } + } + + /// ADR-0006: a run of mark reads that fail logs one warning, not one per + /// key, and the read that ends the run reports how many failed. + #[cfg(feature = "logging")] + #[tokio::test] + async fn a_run_of_unreadable_marks_warns_once() { + let (pl, _temp) = create_test_paid_list().await; + let first: XorName = [0x6E; 32]; + let second: XorName = [0x6F; 32]; + pl.insert(&first).await.expect("insert"); + let sink = LogSink::default(); + let subscriber = tracing_subscriber::fmt() + .with_writer(sink.clone()) + .with_ansi(false) + .finish(); + + with_default(subscriber, || { + { + let _reader = pl.hold_reader_for_tests().expect("hold reader"); + for key in [first, second, first] { + assert!(pl.counts_as_price_contested(&key)); + } + } + assert!(!pl.counts_as_price_contested(&first)); + assert!(!pl.counts_as_price_contested(&second)); + }); + + let logs = String::from_utf8(sink.0.lock().expect("sink").clone()).expect("utf-8"); + assert_eq!( + logs.matches("Cannot read the price-contested mark").count(), + 1, + "{logs}" + ); + assert_eq!( + logs.matches("readable again, after 3 failed reads").count(), + 1, + "{logs}" + ); + } + #[tokio::test] async fn test_all_keys() { let (pl, _temp) = create_test_paid_list().await; @@ -757,6 +1306,24 @@ mod tests { (release_tx, holder) } + /// Hold the paid list's write order on a blocking thread, so the next + /// write waits before its write transaction. + /// + /// Returns once the order is held. Dropping the returned sender releases + /// it. + async fn hold_write_order(pl: &PaidList) -> (oneshot::Sender<()>, JoinHandle<()>) { + let write_order = Arc::clone(&pl.write_order); + let (release_tx, release_rx) = oneshot::channel::<()>(); + let (held_tx, held_rx) = oneshot::channel::<()>(); + let holder = spawn_blocking(move || { + let _ordered = write_order.lock(); + let _ = held_tx.send(()); + let _ = release_rx.blocking_recv(); + }); + assert!(held_rx.await.is_ok(), "write order held"); + (release_tx, holder) + } + /// Dropping an `insert` awaiter does not cancel its `spawn_blocking` LMDB /// transaction; `wait_idle` must wait for that detached write, and the /// paid list must remain usable afterwards. diff --git a/src/replication/pointer.rs b/src/replication/pointer.rs index 7afa5565..88aca32e 100644 --- a/src/replication/pointer.rs +++ b/src/replication/pointer.rs @@ -1450,6 +1450,13 @@ impl PointerReplication { warn!("A fresh pointer write did not parse; not replicating it"); return; }; + // Decided from the proof being forwarded, before it moves into the + // offer: a state the price floor contests here is not used to judge + // this node's peers (ADR-0006). + let contested = self + .payments + .price_contested(&state.address, &write.payment_proof) + .await; let self_id = *self.p2p.peer_id(); let targets: Vec = self .p2p @@ -1504,6 +1511,14 @@ impl PointerReplication { hex::encode(state.address), targets.len() ); + if contested { + info!( + "Fresh pointer {}: no possession check, the payment is contested under \ + the price floor", + hex::encode(state.address) + ); + return; + } self.schedule_possession_check(state, targets); } diff --git a/src/replication/pruning.rs b/src/replication/pruning.rs index da8d7f9f..06052436 100644 --- a/src/replication/pruning.rs +++ b/src/replication/pruning.rs @@ -342,6 +342,7 @@ struct PruneAuditReportState { #[derive(Clone, Copy)] struct PruneAuditContext<'a> { storage: &'a Arc, + paid_list: &'a PaidList, p2p_node: &'a Arc, config: &'a ReplicationConfig, sync_state: &'a Arc>, @@ -360,6 +361,7 @@ pub async fn run_prune_pass_with_context(ctx: PrunePassContext<'_>) -> PruneResu let (paid_count, paid_stats) = prune_paid_entries( ctx.self_id, ctx.paid_list, + ctx.storage, ctx.p2p_node, ctx.config, now, @@ -508,16 +510,8 @@ async fn prune_stored_records(ctx: &PrunePassContext<'_>) -> (usize, RecordPrune ) .await; - let present_by_key = collect_record_prune_proofs( - &audit_selection.candidates, - stored_keys.len(), - ctx.storage, - ctx.p2p_node, - ctx.config, - ctx.sync_state, - ctx.audit_challenge_coordinator, - ) - .await; + let present_by_key = + collect_record_prune_proofs(&audit_selection.candidates, stored_keys.len(), ctx).await; let (keys_to_delete, revalidated_cleared, audit_below_threshold) = revalidated_record_prune_keys(&audit_selection.candidates, &present_by_key, ctx).await; stats.cleared += revalidated_cleared; @@ -801,6 +795,7 @@ fn select_record_prune_audits( async fn prune_paid_entries( self_id: &PeerId, paid_list: &Arc, + storage: &ChunkStore, p2p_node: &Arc, config: &ReplicationConfig, now: Instant, @@ -849,7 +844,9 @@ async fn prune_paid_entries( let elapsed = now .checked_duration_since(first_seen) .unwrap_or(Duration::ZERO); - if elapsed >= config.prune_hysteresis_duration { + if elapsed >= config.prune_hysteresis_duration + && !keeps_price_contested_mark(key, paid_list, storage) + { match select_paid_prune_candidate( key, &closest, @@ -886,7 +883,7 @@ async fn prune_paid_entries( let confirmed_by_key = collect_paid_prune_confirmations(&expired_candidates, p2p_node, config).await; - let (paid_keys_to_delete, revalidated_cleared) = revalidated_paid_prune_keys( + let (mut paid_keys_to_delete, revalidated_cleared) = revalidated_paid_prune_keys( &expired_candidates, &confirmed_by_key, self_id, @@ -895,12 +892,24 @@ async fn prune_paid_entries( config, ) .await; + // Checked again after the network round, in case the record arrived + // meanwhile. + paid_keys_to_delete.retain(|key| !keeps_price_contested_mark(key, paid_list, storage)); stats.cleared += revalidated_cleared; stats.pruned = delete_paid_entries(&paid_keys_to_delete, paid_list).await; (paid_keys.len(), stats) } +/// Whether the paid entry for `key` must outlive the paid-prune rules: it +/// carries the price-contested mark (see `fresh::record_paid_key`) and this +/// node still holds the record the mark protects. State that cannot be read +/// counts as both (see `PaidList::counts_as_price_contested`). Such an entry +/// is not even put to the paid group, so it costs no verification budget. +fn keeps_price_contested_mark(key: &XorName, paid_list: &PaidList, storage: &ChunkStore) -> bool { + paid_list.counts_as_price_contested(key) && storage.exists(key).unwrap_or(true) +} + fn select_paid_prune_candidate( key: &XorName, closest: &[PeerId], @@ -1202,14 +1211,13 @@ async fn delete_stored_records( /// local, and the retained record continues to participate in normal /// neighbor-sync repair because replica hint construction walks all locally /// stored keys, including out-of-range keys retained by hysteresis. +/// +/// A price-contested key still needs its proofs before it is deleted, but no +/// absent peer is scored for it (see `fresh::record_paid_key`). async fn collect_record_prune_proofs( candidates: &[RecordPruneCandidate], local_stored_key_count: usize, - storage: &Arc, - p2p_node: &Arc, - config: &ReplicationConfig, - sync_state: &Arc>, - audit_challenge_coordinator: &Arc, + ctx: &PrunePassContext<'_>, ) -> HashMap> { if candidates.is_empty() { return HashMap::new(); @@ -1220,11 +1228,12 @@ async fn collect_record_prune_proofs( let challenges = build_peer_audit_challenges(candidates, max_keys_per_challenge); let report_state = PruneAuditReportState::default(); let audit_context = PruneAuditContext { - storage, - p2p_node, - config, - sync_state, - audit_challenge_coordinator, + storage: ctx.storage, + paid_list: ctx.paid_list, + p2p_node: ctx.p2p_node, + config: ctx.config, + sync_state: ctx.sync_state, + audit_challenge_coordinator: ctx.audit_challenge_coordinator, report_state: &report_state, }; let mut requests = stream::iter(challenges) @@ -1618,15 +1627,7 @@ async fn peer_proves_records( "failed" }; if !audit_failure_reported - && report_prune_audit_failure_once( - &peer, - &key, - context.p2p_node, - context.config, - context.report_state, - failure_class, - ) - .await + && report_prune_audit_failure_once(&peer, &key, context, failure_class).await { audit_failure_reported = true; } @@ -1671,16 +1672,7 @@ async fn receive_prune_audit_response( }; for key in challenge_keys { - if report_prune_audit_failure_once( - peer, - key, - context.p2p_node, - context.config, - context.report_state, - failure_class, - ) - .await - { + if report_prune_audit_failure_once(peer, key, context, failure_class).await { break; } } @@ -1964,11 +1956,25 @@ fn audit_digest_proves_key( async fn report_prune_audit_failure_once( peer: &PeerId, key: &XorName, - p2p_node: &Arc, - config: &ReplicationConfig, - report_state: &PruneAuditReportState, + context: PruneAuditContext<'_>, audit_failure_class: &'static str, ) -> bool { + let PruneAuditContext { + paid_list, + p2p_node, + config, + report_state, + .. + } = context; + // No peer is scored for a price-contested key (see + // `fresh::record_paid_key`); a mark that cannot be read counts as set. + if paid_list.counts_as_price_contested(key) { + debug!( + "Prune audit: not scoring {peer} for {}, a price-contested key", + hex::encode(key) + ); + return false; + } let should_report = peer_is_currently_responsible(peer, key, p2p_node, config).await && reserve_prune_audit_failure_report(report_state, peer).await; if !should_report { @@ -2118,6 +2124,53 @@ mod tests { RecordPruneCandidate { key, target_peers } } + /// ADR-0006: a price-contested paid entry is kept while its record is + /// held, and prunes like any other once the record is gone. Unmarked + /// entries are unaffected. + #[tokio::test] + async fn a_price_contested_paid_entry_outlives_paid_pruning_while_its_record_is_held() { + let dir = TempDir::new().expect("temp dir"); + let storage = ChunkStore::new(ChunkStoreConfig { + root_dir: dir.path().join("chunks"), + ..ChunkStoreConfig::test_default() + }) + .await + .expect("chunk store"); + let paid_list = PaidList::new(dir.path()).await.expect("paid list"); + let held_bytes = b"held price-contested record".to_vec(); + let held = ChunkStore::compute_address(&held_bytes); + storage.put(&held, &held_bytes).await.expect("put"); + let gone: XorName = [0x81; 32]; + let plain: XorName = [0x82; 32]; + paid_list.insert_price_contested(&held).await.expect("mark"); + paid_list.insert_price_contested(&gone).await.expect("mark"); + paid_list.insert(&plain).await.expect("insert"); + + assert!(keeps_price_contested_mark(&held, &paid_list, &storage)); + assert!(!keeps_price_contested_mark(&gone, &paid_list, &storage)); + assert!(!keeps_price_contested_mark(&plain, &paid_list, &storage)); + + // A held record whose mark cannot be read is kept as if marked. + let plain_held_bytes = b"held plain record".to_vec(); + let plain_held = ChunkStore::compute_address(&plain_held_bytes); + storage + .put(&plain_held, &plain_held_bytes) + .await + .expect("put"); + paid_list.insert(&plain_held).await.expect("insert"); + assert!(!keeps_price_contested_mark( + &plain_held, + &paid_list, + &storage + )); + let _reader = paid_list.hold_reader_for_tests().expect("hold reader"); + assert!(keeps_price_contested_mark( + &plain_held, + &paid_list, + &storage + )); + } + /// A prune audit judges holders against this node's own copy. A copy that /// no longer hashes to its key would fail every honest holder and hold the /// prune back forever, so it is not used as a reference: the read takes it diff --git a/src/storage/handler.rs b/src/storage/handler.rs index 62e2db75..6212adf4 100644 --- a/src/storage/handler.rs +++ b/src/storage/handler.rs @@ -864,9 +864,9 @@ impl AntProtocol { self.quote_generator.record_store(); // 7. Notify replication engine for fresh fan-out. - // Only emit when a real proof is present — cached-as-verified - // PUTs have no proof to forward, and the chunk would have - // already replicated on the original write that carried one. + // Only emit when a real proof is present. A cached-as-verified + // PUT normally carries none; one that does forwards it + // unverified, and the fan-out prices it as it stands (ADR-0006). if let (Some(ref tx), Some(proof)) = (&self.fresh_write_tx, request.payment_proof) { // ADR-0004: strip any commitment sidecars before forwarding // to replication — the ADR specifies persisted/replicated @@ -1249,7 +1249,6 @@ mod tests { cache_capacity: 100_000, close_group_size: crate::ant_protocol::CLOSE_GROUP_SIZE, local_rewards_address: rewards_address, - price_floor: crate::payment::PriceFloorConfig::default(), }; let payment_verifier = Arc::new(PaymentVerifier::new(payment_config)); let metrics_tracker = QuotingMetricsTracker::new(100); diff --git a/tests/e2e/data_types/chunk.rs b/tests/e2e/data_types/chunk.rs index 650b8e08..96480f88 100644 --- a/tests/e2e/data_types/chunk.rs +++ b/tests/e2e/data_types/chunk.rs @@ -64,8 +64,8 @@ mod tests { use crate::{TestHarness, TestNetwork}; use ant_node::payment::{ - EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, - QuoteGenerator, QuotingMetricsTracker, + EvmVerifierConfig, PaymentVerifier, PaymentVerifierConfig, QuoteGenerator, + QuotingMetricsTracker, }; use ant_node::storage::{AntProtocol, ChunkStore, ChunkStoreConfig}; use ant_node::ReplicationConfig; @@ -445,7 +445,6 @@ mod tests { cache_capacity: 100, close_group_size: ReplicationConfig::default().close_group_size, local_rewards_address: rewards_address, - price_floor: PriceFloorConfig::default(), }); let metrics_tracker = QuotingMetricsTracker::new(100); let quote_generator = QuoteGenerator::new(rewards_address, metrics_tracker); diff --git a/tests/e2e/harness.rs b/tests/e2e/harness.rs index 786b18c5..efa0531f 100644 --- a/tests/e2e/harness.rs +++ b/tests/e2e/harness.rs @@ -6,12 +6,45 @@ use super::anvil::TestAnvil; use super::testnet::{TestNetwork, TestNetworkConfig, TestNode, TestnetError}; use ant_node::client::XorName; -use evmlib::common::TxHash; +use ant_node::payment::{serialize_single_node_proof, PaymentProof}; +use evmlib::common::{Amount, TxHash}; +use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment, RewardsAddress}; use saorsa_core::P2PNode; use std::collections::HashMap; use std::sync::{Arc, Mutex}; +use std::time::SystemTime; use tracing::{info, warn}; +/// A possession or audit failure is reported at weight 5, which cuts a peer's +/// trust by `1 - (1 - 0.124)^5`, about 48%; lighter events cut 12% to 33%. A +/// fall to this fraction of a peer's score or below is read as that penalty. +pub const POSSESSION_PENALTY_TRUST_RATIO: f64 = 0.6; + +/// A single-node payment proof that parses, naming `content`, whose one quote +/// is never valid: no signature, no settlement. For tests that need a node to +/// see a proof the price floor could price, without anyone accepting it. +#[allow(clippy::expect_used)] +pub fn parseable_single_node_proof(content: XorName) -> Vec { + let quote = PaymentQuote { + content: xor_name::XorName(content), + timestamp: SystemTime::now(), + price: Amount::from(1u64), + rewards_address: RewardsAddress::new([0x11; 20]), + committed_key_count: 0, + commitment_pin: None, + pub_key: Vec::new(), + signature: Vec::new(), + }; + serialize_single_node_proof(&PaymentProof { + proof_of_payment: ProofOfPayment { + peer_quotes: vec![(EncodedPeerId::new([0x22; 32]), quote)], + }, + tx_hashes: Vec::new(), + commitment_sidecars: Vec::new(), + }) + .expect("serialize a single-node proof") +} + /// How many port ranges a network is brought up on before its bind failure /// is reported. /// diff --git a/tests/e2e/mod.rs b/tests/e2e/mod.rs index ec2a74c7..cdec4e56 100644 --- a/tests/e2e/mod.rs +++ b/tests/e2e/mod.rs @@ -78,6 +78,9 @@ mod fetch_responsibility_recheck; #[cfg(test)] mod first_audit_ab; +#[cfg(test)] +mod price_floor_ab; + #[cfg(test)] mod fresh_offer_capacity; diff --git a/tests/e2e/pointer_replication.rs b/tests/e2e/pointer_replication.rs index 8548a7c0..586ccf0c 100644 --- a/tests/e2e/pointer_replication.rs +++ b/tests/e2e/pointer_replication.rs @@ -8,6 +8,7 @@ #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use super::harness::parseable_single_node_proof; use super::testnet::{TestNetworkConfig, TestNode}; use super::TestHarness; use ant_node::ant_protocol::chunk::{ @@ -32,8 +33,10 @@ use bytes::Bytes; use saorsa_core::identity::PeerId; use saorsa_pqc::api::sig::{ml_dsa_65, MlDsaPublicKey, MlDsaSecretKey}; use serial_test::serial; +use std::collections::HashMap; use std::sync::Arc; use std::time::Duration; +use tokio::sync::RwLock; /// How long to wait for replication to reach a node. const SETTLE: Duration = Duration::from_secs(30); @@ -475,6 +478,98 @@ async fn the_possession_check_penalises_only_a_member_that_dropped_the_record() harness.teardown().await.expect("teardown"); } +/// ADR-0006: a fresh pointer write the offering node cannot price is not +/// used to judge its peers. The checker's price floor is wired (a gossip cache +/// is attached) but it has no commitment of its own, so it cannot price the +/// single-node proof it forwards, and no possession check may follow. The +/// control write that follows forwards a proof the floor never prices, and its +/// check does penalise the members, so the first result is not vacuous. +#[tokio::test] +#[serial] +async fn a_fresh_pointer_write_the_floor_cannot_price_is_not_used_to_judge_peers() { + // Long enough for the scheduled check (at most 500 ms here) and its probes + // to have run, short enough that unrelated trust events are unlikely. + const CONTESTED_WATCH: Duration = Duration::from_secs(5); + let mut config = TestNetworkConfig::minimal(); + config.replication_config = Some(ReplicationConfig { + possession_check_delay_min: Duration::from_millis(200), + possession_check_delay_max: Duration::from_millis(500), + ..ReplicationConfig::default() + }); + let harness = TestHarness::setup_with_config(config).await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let checker = 3; + // The checker only judges peers that understand pointers. + let everyone: Vec = (0..harness.node_count()).collect(); + exchange_hints(&harness, &everyone, &[checker]).await; + + let checker_node = harness.test_node(checker).expect("node"); + checker_node + .ant_protocol + .as_ref() + .expect("protocol") + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + let checker_p2p = checker_node.p2p_node.as_ref().expect("p2p"); + let members: Vec = others(&harness, &[checker]) + .into_iter() + .map(|i| peer(harness.test_node(i).expect("node"))) + .collect(); + + let (pk, sk) = owner(); + let contested = signed(&pk, &sk, 0, 1); + let control = signed(&pk, &sk, 1, 2); + let mut outcomes = Vec::new(); + for (record, proof, watch) in [ + ( + &contested, + parseable_single_node_proof(contested.state_id()), + CONTESTED_WATCH, + ), + (&control, DUMMY_PROOF.to_vec(), SETTLE), + ] { + let before: Vec = members.iter().map(|p| checker_p2p.peer_trust(p)).collect(); + store(checker_node) + .put_bytes(&record.to_bytes()) + .await + .expect("put"); + // Nothing is paid anywhere, so every member refuses the offer and is + // absent when, and only if, the checker probes it. + replication(checker_node) + .replicate_fresh(PointerFreshWrite { + record: record.to_bytes(), + payment_proof: proof, + }) + .await; + + // A pointer possession failure is a weight-1 trust event, so any fall + // counts. + let deadline = tokio::time::Instant::now() + watch; + let mut penalised = false; + while tokio::time::Instant::now() < deadline { + penalised = members + .iter() + .zip(before.iter()) + .any(|(p, &b)| checker_p2p.peer_trust(p) < b - f64::EPSILON); + if penalised { + break; + } + tokio::time::sleep(POLL).await; + } + outcomes.push(penalised); + } + + assert_eq!( + outcomes, + vec![false, true], + "a write the floor cannot price must not be used to penalise members, while the \ + control write must" + ); + + harness.teardown().await.expect("teardown"); +} + /// A transfer written to one node reaches the whole group, and after that no /// node takes a second final state — not even one whose target sorts first, /// which the previous merge rule let displace the first everywhere (ADR-0018). diff --git a/tests/e2e/price_floor_ab.rs b/tests/e2e/price_floor_ab.rs new file mode 100644 index 00000000..c8522344 --- /dev/null +++ b/tests/e2e/price_floor_ab.rs @@ -0,0 +1,743 @@ +//! ADR-0006 price-floor A/B workload test, env-gated and never run in CI. +//! +//! Builds a production-wired local testnet whose nodes hold production-sized +//! commitments, then drives two workloads against it with real on-chain +//! payments (Anvil): +//! +//! - **honest** uploads: quotes from the close group, the median settled 3x, +//! the chunk PUT to every close-group storer; +//! - **near-empty quoter** uploads: a node that joined after seeding (no +//! commitment, so it quotes the baseline price) is paid 3x its quote, and the +//! chunk is PUT to it and to the mature storers. +//! +//! After a tail long enough for fresh replication, `PaidNotify` and the +//! (shortened) possession checks to play out, it reports who stored each key +//! and whose paid list holds it. Compare the `PF-METRIC` lines, plus the log +//! counts the runner greps for (possession failures naming a near-empty key, +//! contested fan-outs, floor refusals), across builds. It returns at once +//! unless `PRICE_FLOOR_AB=1` is set, so CI only compiles it. +//! +//! ```bash +//! PRICE_FLOOR_AB=1 RUST_LOG=info \ +//! cargo test --test e2e --features test-utils price_floor_ab_workload -- --nocapture +//! ``` +//! +//! Neighbour commitments are delivered straight into each node's gossip cache +//! after seeding, exactly the signed commitments neighbour sync would carry. +//! At production-sized key sets the in-process testnet's sync rounds do not +//! deliver them within any useful time, and the floor skips every admission +//! until they arrive. Everything after that (quotes, payments, PUTs, fresh +//! replication, `PaidNotify`, possession checks) runs on the live paths. +//! +//! It fails, rather than reporting, when its own preconditions do not hold: +//! fewer than 15 known neighbour commitments on the median node (the floor +//! would skip everything) or fewer uploads completed than requested. It also +//! asserts what the floor promises: every honest upload reaches a storage +//! quorum with no floor refusal, no peer that refused a near-empty write or +//! that the quoter fanned it out to loses penalty-sized trust in the quoter's +//! view, and every near-empty write stays off a quorum, every refusal being +//! the floor's (a mature storer whose own price is low enough may accept it, +//! as the cap allows). +//! +//! Knobs (env): `PF_NODES`, `PF_KEYS_PER_NODE`, `PF_HONEST`, `PF_NEAR_EMPTY`, +//! `PF_TAIL_SECS`. + +#![allow( + clippy::unwrap_used, + clippy::expect_used, + clippy::panic, + clippy::unnested_or_patterns, + clippy::cast_precision_loss, + clippy::too_many_lines +)] + +use ant_node::ant_protocol::{ + ChunkMessage, ChunkMessageBody, ChunkPutRequest, ChunkPutResponse, ChunkQuoteRequest, + ChunkQuoteResponse, CLOSE_GROUP_SIZE, +}; +use ant_node::client::{compute_address, send_and_await_chunk_response}; +use ant_node::payment::quote::CommitmentSource; +use ant_node::payment::{serialize_single_node_proof, PaymentProof, PRICE_FLOOR_TOLERANCE_PERCENT}; +use ant_node::replication::commitment_state::PeerCommitmentRecord; +use ant_node::ReplicationConfig; +use bytes::Bytes; +use evmlib::common::Amount; +use evmlib::wallet::Wallet; +use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment}; +use rand::Rng; +use saorsa_core::identity::PeerId; +use saorsa_core::P2PNode; +use serial_test::serial; +use std::collections::{BTreeSet, HashMap}; +use std::sync::Arc; +use std::time::{Duration, Instant}; +use tracing_subscriber::EnvFilter; + +use super::anvil::TestAnvil; +use super::harness::POSSESSION_PENALTY_TRUST_RATIO; +use super::testnet::{TestNetwork, TestNetworkConfig, TestNode}; + +/// The verifier accepts a settlement of at least 3x the median quote price. +const PAYMENT_MULTIPLIER: u64 = 3; + +/// Per-request timeout for quote/PUT round trips. +const REQUEST_TIMEOUT: Duration = Duration::from_secs(20); + +/// Substring of the node's price-floor refusal. +const FLOOR_REFUSAL: &str = "below the close-group price floor"; + +/// Size of each synthetic seeding chunk. +const SEED_CHUNK_BYTES: usize = 64; + +/// Neighbour commitments a node must know before its floor evaluates at all +/// (`PRICE_FLOOR_MIN_NEIGHBOUR_COMMITMENTS`, private to the verifier). +const FLOOR_MIN_NEIGHBOURS: usize = 15; + +/// Stores a chunk upload needs (`CLOSE_GROUP_MAJORITY`). +const STORE_QUORUM: usize = 4; + +fn env_usize(name: &str, default: usize) -> usize { + std::env::var(name) + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(default) +} + +fn xor_distance(a: &[u8; 32], b: &[u8; 32]) -> [u8; 32] { + let mut out = [0u8; 32]; + for (o, (x, y)) in out.iter_mut().zip(a.iter().zip(b.iter())) { + *o = x ^ y; + } + out +} + +/// Indices of `peers` sorted by XOR distance to `address`, closest first. +fn closest_indices(address: &[u8; 32], peers: &[[u8; 32]]) -> Vec { + let mut order: Vec = (0..peers.len()).collect(); + order.sort_by_key(|&i| peers.get(i).map(|p| xor_distance(address, p))); + order +} + +/// Wire one node exactly like production `node.rs`: commitment source into the +/// quote generator, gossip cache into the verifier, monetized-pin sender into +/// the verifier. The e2e testnet does not wire these by default. +fn wire_like_production(node: &TestNode) { + if let (Some(protocol), Some(engine)) = + (node.ant_protocol.as_ref(), node.replication_engine.as_ref()) + { + let concrete = Arc::clone(engine.commitment_state()); + let source: Arc = concrete; + protocol.attach_commitment_source(source); + protocol + .payment_verifier_arc() + .attach_commitment_cache(Arc::clone(engine.last_commitment_by_peer())); + protocol + .payment_verifier_arc() + .attach_monetized_pin_sender(engine.monetized_pin_sender()); + } +} + +fn peer_bytes(node: &TestNode) -> [u8; 32] { + *node + .p2p_node + .as_ref() + .expect("node p2p") + .peer_id() + .as_bytes() +} + +async fn request_quote( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data_size: u64, +) -> Result { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::QuoteRequest(ChunkQuoteRequest { + address, + data_size, + data_type: 0, + }), + }; + let message_bytes = message + .encode() + .map_err(|e| format!("encode quote request: {e}"))?; + let quote_bytes = send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Success { quote, .. }) => { + Some(Ok(quote)) + } + ChunkMessageBody::QuoteResponse(ChunkQuoteResponse::Error(e)) => { + Some(Err(format!("quote error: {e:?}"))) + } + _ => None, + }, + |e| format!("send quote request: {e}"), + || "quote request timed out".to_string(), + ) + .await?; + rmp_serde::from_slice("e_bytes).map_err(|e| format!("decode quote: {e}")) +} + +/// One storer's answer to a paid PUT. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum PutOutcome { + Stored, + FloorRefused, + OtherError, +} + +async fn paid_put( + client: &Arc, + peer: &PeerId, + address: [u8; 32], + data: &[u8], + proof_bytes: Vec, +) -> (PutOutcome, String) { + let request_id: u64 = rand::thread_rng().gen(); + let message = ChunkMessage { + request_id, + body: ChunkMessageBody::PutRequest(ChunkPutRequest::with_payment( + address, + Bytes::copy_from_slice(data), + proof_bytes, + )), + }; + let Ok(message_bytes) = message.encode() else { + return (PutOutcome::OtherError, "encode put".to_string()); + }; + let result = send_and_await_chunk_response( + client, + peer, + message_bytes, + request_id, + REQUEST_TIMEOUT, + &[], + |body| match body { + ChunkMessageBody::PutResponse(ChunkPutResponse::Success { .. }) + | ChunkMessageBody::PutResponse(ChunkPutResponse::AlreadyExists { .. }) => Some(Ok(())), + ChunkMessageBody::PutResponse(ChunkPutResponse::PaymentRequired { message }) => { + Some(Err(format!("payment required: {message}"))) + } + ChunkMessageBody::PutResponse(ChunkPutResponse::Error(e)) => { + Some(Err(format!("put error: {e:?}"))) + } + _ => None, + }, + |e| format!("send put: {e}"), + || "put timed out".to_string(), + ) + .await; + match result { + Ok(()) => (PutOutcome::Stored, String::new()), + Err(e) if e.contains(FLOOR_REFUSAL) => (PutOutcome::FloorRefused, e), + Err(e) => (PutOutcome::OtherError, e), + } +} + +/// Settle `amount` for `quote` on chain and build the single-quote proof. +async fn pay(wallet: &Wallet, issuer: &PeerId, quote: &PaymentQuote, amount: Amount) -> Vec { + let (tx_by_quote, _gas) = wallet + .pay_for_quotes(vec![(quote.hash(), quote.rewards_address, amount)]) + .await + .expect("pay_for_quotes on Anvil"); + let proof = PaymentProof { + proof_of_payment: ProofOfPayment { + peer_quotes: vec![(EncodedPeerId::new(*issuer.as_bytes()), quote.clone())], + }, + tx_hashes: tx_by_quote.values().copied().collect(), + commitment_sidecars: vec![], + }; + serialize_single_node_proof(&proof).expect("serialize proof") +} + +/// How many of `nodes` store `key`, and how many hold it in their paid list. +fn holders_and_votes(nodes: &[TestNode], key: &[u8; 32]) -> (usize, usize) { + let mut holders = 0; + let mut votes = 0; + for node in nodes { + if let Some(protocol) = node.ant_protocol.as_ref() { + if protocol.storage().exists(key).unwrap_or(false) { + holders += 1; + } + } + if let Some(engine) = node.replication_engine.as_ref() { + if engine.paid_list().contains(key).unwrap_or(false) { + votes += 1; + } + } + } + (holders, votes) +} + +fn median(mut xs: Vec) -> usize { + xs.sort_unstable(); + xs.get(xs.len() / 2).copied().unwrap_or(0) +} + +async fn sync_rounds(network: &TestNetwork, rounds: usize) { + for _ in 0..rounds { + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + engine.trigger_neighbor_sync(); + } + } + tokio::time::sleep(Duration::from_secs(4)).await; + } +} + +async fn cache_sizes(network: &TestNetwork) -> Vec { + let mut sizes = Vec::new(); + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + sizes.push(engine.last_commitment_by_peer().read().await.len()); + } + } + sizes +} + +/// Put every node's current signed commitment into every other node's gossip +/// cache, as neighbour sync would. +async fn deliver_commitments(network: &TestNetwork) { + let commitments: Vec<(PeerId, _)> = network + .nodes() + .iter() + .filter_map(|node| { + let engine = node.replication_engine.as_ref()?; + let current = engine.commitment_state().current()?; + Some(( + PeerId::from_bytes(peer_bytes(node)), + current.commitment().clone(), + )) + }) + .collect(); + let now = Instant::now(); + for node in network.nodes() { + let own = PeerId::from_bytes(peer_bytes(node)); + if let Some(engine) = node.replication_engine.as_ref() { + let mut cache = engine.last_commitment_by_peer().write().await; + for (peer, commitment) in &commitments { + if *peer != own { + cache.insert( + *peer, + PeerCommitmentRecord::from_verified(commitment.clone(), now), + ); + } + } + } + } +} + +#[tokio::test(flavor = "multi_thread")] +#[serial] +async fn price_floor_ab_workload() { + if std::env::var("PRICE_FLOOR_AB").ok().as_deref() != Some("1") { + eprintln!("price_floor_ab_workload skipped: set PRICE_FLOOR_AB=1 to run"); + return; + } + let _ = tracing_subscriber::fmt() + .with_env_filter( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")), + ) + .with_writer(std::io::stderr) + .try_init(); + + let nodes = env_usize("PF_NODES", 24); + let keys_per_node = env_usize("PF_KEYS_PER_NODE", 4_500); + let honest_uploads = env_usize("PF_HONEST", 12); + let near_empty_uploads = env_usize("PF_NEAR_EMPTY", 8); + + let tail_secs = env_usize("PF_TAIL_SECS", 90); + eprintln!( + "PF-CONFIG nodes={nodes} keys_per_node={keys_per_node} honest={honest_uploads} \ + near_empty={near_empty_uploads} tail_secs={tail_secs}" + ); + + let anvil = TestAnvil::new().await.expect("start anvil"); + let wallet = anvil.create_funded_wallet().expect("funded wallet"); + let replication_config = ReplicationConfig { + neighbor_sync_interval_min: Duration::from_secs(5), + neighbor_sync_interval_max: Duration::from_secs(10), + possession_check_delay_min: Duration::from_secs(20), + possession_check_delay_max: Duration::from_secs(30), + subtree_round1_responder_cooldown: Duration::from_millis(1), + ..ReplicationConfig::default() + }; + let mut config = TestNetworkConfig { + node_count: nodes, + payment_enforcement: true, + replication_config: Some(replication_config), + ..TestNetworkConfig::default() + }; + config.evm_network = Some(anvil.to_network()); + let mut network = TestNetwork::new(config).await.expect("build network"); + network.start().await.expect("start network"); + network.warmup_dht().await.expect("warmup dht"); + for node in network.nodes() { + wire_like_production(node); + } + + let client_index = nodes.saturating_sub(1); + let client = network + .nodes() + .get(client_index) + .and_then(|n| n.p2p_node.as_ref()) + .map(Arc::clone) + .expect("client node p2p"); + + // ---- Seed: write synthetic chunks straight into the stores of the seven + // XOR-closest nodes (each node's own close group in a full routing table), + // then rotate commitments, so every node commits to a production-sized key + // set without thousands of on-chain payments. + let peers: Vec<[u8; 32]> = network.nodes().iter().map(peer_bytes).collect(); + let seed_chunks = keys_per_node * nodes / CLOSE_GROUP_SIZE; + let mut rng = rand::thread_rng(); + let mut assignments: Vec)>> = vec![Vec::new(); nodes]; + for i in 0..seed_chunks { + let mut content = vec![0u8; SEED_CHUNK_BYTES]; + rng.fill(&mut content[..]); + if let Some(prefix) = content.get_mut(..8) { + prefix.copy_from_slice(&(i as u64).to_le_bytes()); + } + let address = compute_address(&content); + for index in closest_indices(&address, &peers) + .into_iter() + .take(CLOSE_GROUP_SIZE) + { + if let Some(slot) = assignments.get_mut(index) { + slot.push((address, content.clone())); + } + } + } + let mut writers = Vec::new(); + for (node, chunks) in network.nodes().iter().zip(assignments) { + if let Some(protocol) = node.ant_protocol.as_ref() { + let storage = protocol.storage(); + writers.push(tokio::spawn(async move { + for (address, content) in chunks { + storage.put(&address, &content).await.expect("seed put"); + } + })); + } + } + for writer in writers { + writer.await.expect("seed writer"); + } + for node in network.nodes() { + if let Some(engine) = node.replication_engine.as_ref() { + engine.rebuild_commitment_now().await.expect("rebuild"); + } + } + let counts: Vec = network + .nodes() + .iter() + .filter_map(|n| n.replication_engine.as_ref()) + .filter_map(|e| e.commitment_state().current()) + .map(|c| c.commitment().key_count as usize) + .collect(); + eprintln!( + "PF-METRIC phase=seed chunks={seed_chunks} committed_nodes={} key_count_min={} \ + key_count_median={} key_count_max={}", + counts.len(), + counts.iter().min().copied().unwrap_or(0), + median(counts.clone()), + counts.iter().max().copied().unwrap_or(0) + ); + + deliver_commitments(&network).await; + let sizes = cache_sizes(&network).await; + eprintln!( + "PF-METRIC phase=gossip mode=delivered cache_min={} cache_median={} cache_max={}", + sizes.iter().min().copied().unwrap_or(0), + median(sizes.clone()), + sizes.iter().max().copied().unwrap_or(0) + ); + assert!( + median(sizes.clone()) >= FLOOR_MIN_NEIGHBOURS, + "precondition unmet: the median node knows fewer than {FLOOR_MIN_NEIGHBOURS} neighbour \ + commitments, so the floor would skip every admission" + ); + + // ---- Honest uploads. + eprintln!("PF-METRIC phase=build tolerance_percent={PRICE_FLOOR_TOLERANCE_PERCENT}"); + let mut honest_keys = Vec::new(); + let (mut honest_stored, mut honest_floor, mut honest_other) = (0usize, 0usize, 0usize); + for i in 0..honest_uploads { + let mut data = vec![0u8; 4096]; + rng.fill(&mut data[..]); + let address = compute_address(&data); + let storers: Vec = closest_indices(&address, &peers) + .into_iter() + .filter(|&j| j != client_index) + .take(CLOSE_GROUP_SIZE) + .filter_map(|j| peers.get(j).map(|p| PeerId::from_bytes(*p))) + .collect(); + let mut quotes = Vec::new(); + for peer in &storers { + if let Ok(q) = request_quote(&client, peer, address, data.len() as u64).await { + quotes.push((*peer, q)); + } + } + quotes.sort_by_key(|(_, q)| q.price); + let Some((issuer, median_quote)) = quotes.get(quotes.len() / 2).cloned() else { + eprintln!("PF-WARN honest {i}: no quotes"); + continue; + }; + let amount = median_quote.price * Amount::from(PAYMENT_MULTIPLIER); + let proof = pay(&wallet, &issuer, &median_quote, amount).await; + let (mut stored, mut floor, mut other) = (0usize, 0usize, 0usize); + for peer in &storers { + match paid_put(&client, peer, address, &data, proof.clone()).await { + (PutOutcome::Stored, _) => stored += 1, + (PutOutcome::FloorRefused, _) => floor += 1, + (PutOutcome::OtherError, e) => { + other += 1; + eprintln!("PF-WARN honest {i} put to {peer}: {e}"); + } + } + } + eprintln!( + "PF-METRIC phase=honest upload={i} key={} quotes={} paid_price={} stored={stored} \ + floor_refused={floor} other_error={other}", + hex::encode(address), + quotes.len(), + median_quote.price + ); + assert!( + stored >= STORE_QUORUM && floor == 0, + "honest upload {i} must reach a storage quorum with no floor refusal" + ); + honest_stored += stored; + honest_floor += floor; + honest_other += other; + honest_keys.push(address); + } + + // ---- A node joins after seeding: no data, no commitment, baseline quotes. + let before = network.node_count(); + let fresh_index = match network.add_node().await { + Ok(index) => index, + // The node is started and registered before the harness waits on its + // replication bootstrap, which can outlast the harness's fixed timeout + // on a seeded network. The node is still usable: it holds no + // commitment either way, which is all this phase needs. + Err(e) if network.node_count() > before => { + eprintln!("PF-WARN fresh node bootstrap: {e}; continuing with it"); + before + } + Err(e) => panic!("add fresh node: {e}"), + }; + if let Some(node) = network.nodes().get(fresh_index) { + wire_like_production(node); + } + let fresh_peer_bytes = network + .nodes() + .get(fresh_index) + .map(peer_bytes) + .expect("fresh node"); + let fresh_peer = PeerId::from_bytes(fresh_peer_bytes); + sync_rounds(&network, 3).await; + let mut all_peers = peers.clone(); + all_peers.push(fresh_peer_bytes); + let fresh_p2p = network + .nodes() + .get(fresh_index) + .and_then(|n| n.p2p_node.as_ref()) + .map(Arc::clone) + .expect("fresh node p2p"); + let trust_before: HashMap<[u8; 32], f64> = peers + .iter() + .map(|p| (*p, fresh_p2p.peer_trust(&PeerId::from_bytes(*p)))) + .collect(); + let mut refusers: BTreeSet<[u8; 32]> = BTreeSet::new(); + let mut fan_out_targets: BTreeSet<[u8; 32]> = BTreeSet::new(); + let fan_out_group_size = ReplicationConfig::default().close_group_size; + + // ---- Near-empty quoter uploads: pay the fresh node's baseline quote 3x. + let mut near_empty_keys = Vec::new(); + let (mut ne_mature_stored, mut ne_mature_floor, mut ne_mature_other, mut ne_fresh_stored) = + (0usize, 0usize, 0usize, 0usize); + let mut attempts = 0usize; + while near_empty_keys.len() < near_empty_uploads && attempts < near_empty_uploads * 20 { + attempts += 1; + let mut data = vec![0u8; 4096]; + rng.fill(&mut data[..]); + let address = compute_address(&data); + let order = closest_indices(&address, &all_peers); + // Only addresses where the fresh node is among the eight closest, so + // every receiver counts it K-close and the floor is what decides. + let Some(rank) = order.iter().position(|&j| j == fresh_index) else { + continue; + }; + if rank >= 8 { + continue; + } + let quote = match request_quote(&client, &fresh_peer, address, data.len() as u64).await { + Ok(q) => q, + Err(e) => { + eprintln!("PF-WARN near-empty quote: {e}"); + continue; + } + }; + let amount = quote.price * Amount::from(PAYMENT_MULTIPLIER); + let proof = pay(&wallet, &fresh_peer, "e, amount).await; + let mature: Vec = order + .iter() + .copied() + .filter(|&j| j != client_index && j != fresh_index) + .take(CLOSE_GROUP_SIZE) + .filter_map(|j| all_peers.get(j).map(|p| PeerId::from_bytes(*p))) + .collect(); + // The quoter first, as a client paying it would: its store is what + // triggers the fan-out whose possession check is under test. + let (fresh_outcome, fresh_err) = + paid_put(&client, &fresh_peer, address, &data, proof.clone()).await; + // The fan-out goes to the quoter's own view of the close group, which + // can differ from the order above and can include the client node. + fan_out_targets.extend( + fresh_p2p + .dht_manager() + .find_closest_nodes_local_with_self(&address, fan_out_group_size) + .await + .iter() + .filter(|n| n.peer_id != fresh_peer) + .map(|n| *n.peer_id.as_bytes()), + ); + let (mut stored, mut floor, mut other) = (0usize, 0usize, 0usize); + for peer in &mature { + match paid_put(&client, peer, address, &data, proof.clone()).await { + (PutOutcome::Stored, _) => stored += 1, + (PutOutcome::FloorRefused, _) => { + floor += 1; + refusers.insert(*peer.as_bytes()); + } + (PutOutcome::OtherError, e) => { + other += 1; + eprintln!("PF-WARN near-empty put to {peer}: {e}"); + } + } + } + assert_eq!( + fresh_outcome, + PutOutcome::Stored, + "the near-empty quoter must store its own write: {fresh_err}" + ); + // A mature storer whose own price is at most the payment over + // 3 x the tolerance accepts by design (the cap), so a few may store it + // in a low-count region. What the floor promises is that the write + // does not reach a storage quorum, and that every refusal is the + // floor's. The quoter's own store counts toward the quorum too. + assert!( + other == 0 && floor + stored == mature.len() && stored + 1 < STORE_QUORUM, + "the floor must keep a near-empty payment off a storage quorum, refusing it \ + on the floor (stored={stored} floor={floor} other={other})" + ); + eprintln!( + "PF-METRIC phase=near_empty upload={} key={} fresh_rank={rank} paid_price={} \ + mature_stored={stored} mature_floor_refused={floor} mature_other_error={other} \ + fresh_node={fresh_outcome:?}", + near_empty_keys.len(), + hex::encode(address), + quote.price + ); + ne_mature_stored += stored; + ne_mature_floor += floor; + ne_mature_other += other; + ne_fresh_stored += usize::from(fresh_outcome == PutOutcome::Stored); + near_empty_keys.push(address); + } + + assert_eq!( + honest_keys.len(), + honest_uploads, + "every requested honest upload must complete" + ); + assert_eq!( + near_empty_keys.len(), + near_empty_uploads, + "precondition unmet: not every requested near-empty upload was attempted" + ); + + // ---- Tail: fresh offers, PaidNotify and possession checks play out. + eprintln!("PF-METRIC phase=tail sleeping_secs={tail_secs}"); + tokio::time::sleep(Duration::from_secs(tail_secs as u64)).await; + + let mut honest_holders = Vec::new(); + let mut honest_votes = Vec::new(); + for key in &honest_keys { + let (h, v) = holders_and_votes(network.nodes(), key); + eprintln!( + "PF-METRIC phase=after kind=honest key={} holders={h} paid_votes={v}", + hex::encode(key) + ); + honest_holders.push(h); + honest_votes.push(v); + } + let mut ne_holders = Vec::new(); + let mut ne_votes = Vec::new(); + for key in &near_empty_keys { + let (h, v) = holders_and_votes(network.nodes(), key); + assert!( + h < STORE_QUORUM, + "the floor must keep a near-empty write off a storage quorum" + ); + eprintln!( + "PF-METRIC phase=after kind=near_empty key={} holders={h} paid_votes={v}", + hex::encode(key) + ); + ne_holders.push(h); + ne_votes.push(v); + } + + // The node that stored the near-empty writes fanned them out; it must not + // have penalised any peer that refused them, nor any peer it fanned them + // out to. + let mut min_ratio = f64::INFINITY; + let mut penalty_sized = 0usize; + let judged: Vec<&[u8; 32]> = refusers.union(&fan_out_targets).collect(); + for &peer in &judged { + let before = trust_before.get(peer).copied().unwrap_or(0.5); + let after = fresh_p2p.peer_trust(&PeerId::from_bytes(*peer)); + let ratio = if before > 0.0 { after / before } else { 1.0 }; + min_ratio = min_ratio.min(ratio); + if ratio <= POSSESSION_PENALTY_TRUST_RATIO { + penalty_sized += 1; + } + } + eprintln!( + "PF-METRIC phase=trust refusers={} fan_out_targets={} judged={} \ + min_trust_ratio={min_ratio:.3} penalty_sized_drops={penalty_sized}", + refusers.len(), + fan_out_targets.len(), + judged.len() + ); + assert_eq!( + penalty_sized, 0, + "no refuser or fan-out target may be penalised for a write under the price floor" + ); + + eprintln!( + "PF-SUMMARY honest_uploads={} honest_stored={honest_stored} honest_floor_refused={honest_floor} \ + honest_other_error={honest_other} honest_holders_median={} honest_paid_votes_median={} \ + near_empty_uploads={} near_empty_mature_stored={ne_mature_stored} \ + near_empty_mature_floor_refused={ne_mature_floor} near_empty_mature_other_error={ne_mature_other} \ + near_empty_fresh_stored={ne_fresh_stored} near_empty_holders_median={} \ + near_empty_paid_votes_median={}", + honest_keys.len(), + median(honest_holders), + median(honest_votes), + near_empty_keys.len(), + median(ne_holders), + median(ne_votes), + ); + + network.shutdown().await.expect("shutdown network"); +} diff --git a/tests/e2e/replication.rs b/tests/e2e/replication.rs index 6ae3fb5b..9cb1b01f 100644 --- a/tests/e2e/replication.rs +++ b/tests/e2e/replication.rs @@ -5,6 +5,7 @@ #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use super::harness::{parseable_single_node_proof, POSSESSION_PENALTY_TRUST_RATIO}; use super::testnet::{TestNetworkConfig, DEFAULT_CHUNK_OPERATION_TIMEOUT_SECS}; use super::TestHarness; use ant_node::ant_protocol::{ChunkMessage, ChunkMessageBody, ChunkPutRequest, ChunkPutResponse}; @@ -16,10 +17,11 @@ use ant_node::replication::config::{ REPLICATION_PROTOCOL_ID, }; use ant_node::replication::fresh::FreshWriteEvent; +use ant_node::replication::neighbor_sync; use ant_node::replication::protocol::{ compute_audit_digest, AuditChallenge, AuditResponse, FetchRequest, FetchResponse, - FreshReplicationOffer, FreshReplicationResponse, NeighborSyncRequest, ReplicationMessage, - ReplicationMessageBody, VerificationRequest, ABSENT_KEY_DIGEST, + FreshReplicationOffer, FreshReplicationResponse, NeighborSyncRequest, PaidNotify, + ReplicationMessage, ReplicationMessageBody, VerificationRequest, ABSENT_KEY_DIGEST, }; use ant_node::replication::pruning; use ant_node::replication::scheduling::ReplicationQueues; @@ -30,7 +32,7 @@ use bytes::Bytes; use saorsa_core::identity::PeerId; use saorsa_core::{P2PNode, TrustEvent}; use serial_test::serial; -use std::collections::HashSet; +use std::collections::{HashMap, HashSet}; use std::fs; use std::sync::Arc; use std::time::Duration; @@ -551,6 +553,330 @@ async fn fresh_write_pipeline_never_offers_a_corrupt_chunk() { harness.teardown().await.expect("teardown"); } +/// ADR-0006: a fresh write this node cannot price is not used to judge its +/// peers. Node A's price floor is wired (a gossip cache is attached) but A has +/// no commitment of its own, so it cannot price the single-node proof it +/// forwards. +/// +/// - When A's verification of the payment skipped the floor too (stood in for +/// here, as these tests settle nothing on chain), the scheduled possession +/// check must not run, and the key is marked in A's paid list, so A hints it +/// to its peers as paid only, never as a replica they could later be +/// audited on. +/// - When A's verification was not contested but the proof A forwards is, the +/// check must not run either, and the key is recorded plainly. +/// - When a marked record of the key won before the write recorded its own +/// plain decision, as a concurrent `PaidNotify` or fresh offer can, the +/// winning mark keeps the check from running even with a proof the floor +/// never prices. +/// - The control write that follows forwards a proof the floor never prices, +/// on a key with no mark: its check does penalise the same kind of absent +/// peer and its key is hinted as a replica, so the other results are not +/// vacuous. +#[tokio::test] +#[serial] +async fn possession_check_skips_a_write_the_floor_cannot_price() { + let mut net_config = TestNetworkConfig::small(); + net_config.replication_config = Some(ReplicationConfig { + possession_check_delay_min: Duration::from_millis(200), + possession_check_delay_max: Duration::from_millis(500), + ..ReplicationConfig::default() + }); + let harness = TestHarness::setup_with_config(net_config) + .await + .expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let a_idx = 3; + let a = harness.test_node(a_idx).expect("node a"); + let p2p_a = a.p2p_node.as_ref().expect("p2p a"); + let engine_a = a.replication_engine.as_ref().expect("engine a"); + let protocol_a = a.ant_protocol.as_ref().expect("proto a"); + protocol_a + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + let self_a = *p2p_a.peer_id(); + let close_group_size = ReplicationConfig::default().close_group_size; + + let paid_list_a = engine_a.paid_list(); + let mut outcomes = Vec::new(); + for (content, parseable, admitted_contested, recorded_contested) in [ + ( + &b"adr-0006 contested fan-out payload"[..], + true, + true, + false, + ), + ( + &b"adr-0006 contested proof fan-out payload"[..], + true, + false, + false, + ), + ( + &b"adr-0006 marked record fan-out payload"[..], + false, + false, + true, + ), + ( + &b"adr-0006 control fan-out payload"[..], + false, + false, + false, + ), + ] { + let address = compute_address(content); + let proof = if parseable { + parseable_single_node_proof(address) + } else { + dummy_payment_proof() + }; + if admitted_contested { + protocol_a + .payment_verifier_arc() + .note_contested_admission_for_tests(&address); + } + if recorded_contested { + paid_list_a + .insert_price_contested(&address) + .await + .expect("record the winning mark"); + } + let close_group: Vec = p2p_a + .dht_manager() + .find_closest_nodes_local_with_self(&address, close_group_size) + .await + .iter() + .filter(|n| n.peer_id != self_a) + .map(|n| n.peer_id) + .collect(); + let close_peer = *close_group + .first() + .expect("expected a non-empty close group"); + let trust_before: Vec = close_group.iter().map(|p| p2p_a.peer_trust(p)).collect(); + + // A client PUTs the chunk to A, through A's handler and fresh-write + // pipeline. Only A holds the payment in its cache, so the close group + // refuses A's offers and is absent when, and only if, A probes it. + harness.prepopulate_payment_cache_for_peer(&self_a, &address); + put_cached_chunk(&harness, a_idx, content, proof).await; + + let recorded_by = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + while !paid_list_a.contains(&address).unwrap_or(false) + && tokio::time::Instant::now() < recorded_by + { + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + let marked = paid_list_a + .is_price_contested(&address) + .expect("read the mark"); + let replica_hinted = neighbor_sync::build_replica_hints_for_peer( + &close_peer, + &protocol_a.storage(), + paid_list_a, + p2p_a, + close_group_size, + ) + .await + .contains(&address); + let paid_hinted = neighbor_sync::build_paid_hints_for_peer( + &close_peer, + paid_list_a, + p2p_a, + ReplicationConfig::default().paid_list_close_group_size, + ) + .await + .contains(&address); + + let deadline = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + let mut penalised = false; + while tokio::time::Instant::now() < deadline { + penalised = close_group + .iter() + .zip(trust_before.iter()) + .any(|(peer, &before)| { + p2p_a.peer_trust(peer) <= before * POSSESSION_PENALTY_TRUST_RATIO + }); + if penalised { + break; + } + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + outcomes.push((penalised, marked, replica_hinted, paid_hinted)); + } + + assert_eq!( + outcomes, + vec![ + (false, true, false, true), + (false, false, true, true), + (false, true, false, true), + (true, false, true, true), + ], + "a write the floor cannot price, or whose record is marked, must not be used \ + to judge peers (no penalty; a marked key is hinted as paid only), while the \ + control write must" + ); + + harness.teardown().await.expect("teardown"); +} + +/// How a write reaches the node that records its paid key. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum RecordPath { + /// A client PUT, recorded by the storer's fresh-write pipeline. + Put, + /// A fresh offer on its own, with no `PaidNotify` before it. + FreshOffer, + /// A `PaidNotify` on its own. + PaidNotify, +} + +/// ADR-0006: every path that records a paid key marks it exactly when this +/// node's own verification of the payment was contested, and never from the +/// proof that came with the message, which a cache hit leaves unchecked. +/// +/// Each node's floor is wired but cannot price (it has no commitment of its +/// own), so every proof below would be contested if it were priced. Payments +/// are admitted from the payment cache, and the verified contested admission +/// is stood in for on the keys that have one. Each path records one key with +/// that admission, which must be marked, and one without, which must not be. +#[tokio::test] +#[serial] +async fn every_record_path_marks_a_key_from_its_own_verified_decision() { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + for i in 0..harness.node_count() { + if let Some(protocol) = harness.test_node(i).and_then(|n| n.ant_protocol.as_ref()) { + protocol + .payment_verifier_arc() + .attach_commitment_cache(Arc::new(RwLock::new(HashMap::new()))); + } + } + + let (source_idx, receiver_idx) = (3, 1); + let source_p2p = harness + .test_node(source_idx) + .and_then(|n| n.p2p_node.as_ref()) + .expect("source p2p"); + let receiver_peer = *harness + .test_node(receiver_idx) + .and_then(|n| n.p2p_node.as_ref()) + .expect("receiver p2p") + .peer_id(); + + let mut outcomes = Vec::new(); + for (path, admitted_contested) in [ + (RecordPath::Put, true), + (RecordPath::Put, false), + (RecordPath::FreshOffer, true), + (RecordPath::FreshOffer, false), + (RecordPath::PaidNotify, true), + (RecordPath::PaidNotify, false), + ] { + let content = format!("adr-0006 record path {path:?} {admitted_contested}").into_bytes(); + let address = compute_address(&content); + let forged_proof = parseable_single_node_proof(address); + harness.prepopulate_payment_cache_everywhere(&address); + let recorder_idx = if path == RecordPath::Put { + source_idx + } else { + receiver_idx + }; + let recorder = harness.test_node(recorder_idx).expect("recorder"); + if admitted_contested { + recorder + .ant_protocol + .as_ref() + .expect("recorder protocol") + .payment_verifier_arc() + .note_contested_admission_for_tests(&address); + } + + match path { + RecordPath::Put => { + put_cached_chunk(&harness, source_idx, &content, forged_proof).await; + } + RecordPath::FreshOffer => { + let response = send_replication_request( + source_p2p, + &receiver_peer, + ReplicationMessage { + request_id: rand::random(), + body: ReplicationMessageBody::FreshReplicationOffer( + FreshReplicationOffer { + key: address, + data: content.clone(), + proof_of_payment: forged_proof, + }, + ), + }, + PROPAGATION_TIMEOUT, + ) + .await; + assert!( + matches!( + response.body, + ReplicationMessageBody::FreshReplicationResponse( + FreshReplicationResponse::Accepted { .. } + ) + ), + "the receiver must accept a cached fresh offer: {:?}", + response.body + ); + } + RecordPath::PaidNotify => { + let notify = ReplicationMessage { + request_id: rand::random(), + body: ReplicationMessageBody::PaidNotify(PaidNotify { + key: address, + proof_of_payment: forged_proof, + }), + }; + source_p2p + .send_message( + &receiver_peer, + REPLICATION_PROTOCOL_ID, + Bytes::from(notify.encode().expect("encode PaidNotify")), + &[], + ) + .await + .expect("send PaidNotify"); + } + } + + let paid_list = recorder + .replication_engine + .as_ref() + .expect("recorder engine") + .paid_list(); + let deadline = tokio::time::Instant::now() + PROPAGATION_TIMEOUT; + while !paid_list.contains(&address).unwrap_or(false) + && tokio::time::Instant::now() < deadline + { + tokio::time::sleep(PROPAGATION_POLL_INTERVAL).await; + } + assert!( + paid_list.contains(&address).unwrap_or(false), + "{path:?} never recorded the key" + ); + let marked = paid_list + .is_price_contested(&address) + .expect("read the mark"); + outcomes.push((path, admitted_contested, marked)); + } + + for (path, admitted_contested, marked) in outcomes { + assert_eq!( + marked, admitted_contested, + "{path:?}: the mark must follow this node's verified decision, never the attached proof" + ); + } + + harness.teardown().await.expect("teardown"); +} + /// Proof bytes for tests that only need to reach the pre-payment gates. fn dummy_payment_proof() -> Vec { vec![DUMMY_PAYMENT_PROOF_BYTE; DUMMY_PAYMENT_PROOF_LEN] @@ -578,14 +904,26 @@ async fn store_paid_chunk(harness: &TestHarness, source_idx: usize, content: &[u /// handler stores it and emits the fresh-write event itself, carrying the /// dummy proof it was paid with. async fn put_paid_chunk(harness: &TestHarness, source_idx: usize, content: &[u8]) -> XorName { + harness.prepopulate_payment_cache_everywhere(&compute_address(content)); + put_cached_chunk(harness, source_idx, content, dummy_payment_proof()).await +} + +/// PUT a chunk through the source node's handler, attaching `proof`. The +/// caller has put the payment in the source's cache, so the proof itself is +/// never checked. +async fn put_cached_chunk( + harness: &TestHarness, + source_idx: usize, + content: &[u8], + proof: Vec, +) -> XorName { let address = compute_address(content); - harness.prepopulate_payment_cache_everywhere(&address); let request = ChunkMessage { request_id: rand::random(), body: ChunkMessageBody::PutRequest(ChunkPutRequest::with_payment( address, Bytes::copy_from_slice(content), - dummy_payment_proof(), + proof, )), }; let protocol = harness @@ -1516,6 +1854,91 @@ async fn test_prune_pass_requires_remote_confirmation_before_delete() { harness.teardown().await.expect("teardown"); } +/// ADR-0006: a peer missing a key whose payment the price floor contested on +/// the pruner may have refused it under its own floor, so the prune audit +/// keeps the record (its proofs are still missing) but scores no one. The +/// control pass, the same shape with an unmarked key, does penalise the +/// absent peers, so the first result is not vacuous. +#[tokio::test] +#[serial] +async fn prune_audit_scores_no_peer_for_a_price_contested_key() { + let harness = TestHarness::setup_minimal().await.expect("setup"); + harness.warmup_dht().await.expect("warmup"); + + let pruner_idx = 3; + let close_group_size = 2; + let config = prune_test_config(close_group_size); + let sync_state = Arc::new(RwLock::new(NeighborSyncState::new_cycle(vec![]))); + let repair_proofs = Arc::new(RwLock::new(RepairProofs::new())); + let audit_challenge_coordinator = Arc::new(AuditChallengeCoordinator::new()); + + let pruner = harness.test_node(pruner_idx).expect("pruner"); + let pruner_p2p = Arc::clone(pruner.p2p_node.as_ref().expect("pruner p2p")); + let pruner_storage = pruner.ant_protocol.as_ref().expect("protocol").storage(); + let pruner_paid_list = Arc::clone( + pruner + .replication_engine + .as_ref() + .expect("pruner replication engine") + .paid_list(), + ); + let pruner_peer = *pruner_p2p.peer_id(); + + let mut outcomes = Vec::new(); + for (label, contested) in [("price-contested", true), ("unmarked", false)] { + let (content, address, targets) = + find_remote_prune_candidate(&harness, pruner_idx, close_group_size, label).await; + pruner_storage + .put(&address, &content) + .await + .expect("put record on pruner"); + if contested { + pruner_paid_list + .insert_price_contested(&address) + .await + .expect("mark"); + } + let trust_before: Vec = targets.iter().map(|p| pruner_p2p.peer_trust(p)).collect(); + + // No target holds the record: every one is absent when audited. + let result = pruning::run_prune_pass_with_context(pruning::PrunePassContext { + self_id: &pruner_peer, + storage: &pruner_storage, + paid_list: &pruner_paid_list, + p2p_node: &pruner_p2p, + config: &config, + sync_state: &sync_state, + repair_proofs: &repair_proofs, + allow_remote_prune_audits: true, + commitment_state: None, + audit_challenge_coordinator: &audit_challenge_coordinator, + }) + .await; + assert!( + pruner_storage.exists(&address).expect("exists"), + "a record no target proves must stay local" + ); + assert_eq!(result.records_pruned, 0); + + let penalised = targets + .iter() + .zip(trust_before.iter()) + .any(|(peer, &before)| { + pruner_p2p.peer_trust(peer) <= before * POSSESSION_PENALTY_TRUST_RATIO + }); + outcomes.push(penalised); + } + + assert_eq!( + outcomes, + vec![false, true], + "an absent peer must not be scored for a price-contested key, and must be for an \ + unmarked one" + ); + + harness.teardown().await.expect("teardown"); +} + /// Pruner-retention veto (ADR-0002): a chunk the pruner is no longer responsible /// for, but which is still committed under a recently-gossiped commitment, must /// NOT be deleted — the storage-commitment audit's round-2 byte challenge could diff --git a/tests/e2e/testnet.rs b/tests/e2e/testnet.rs index ca33a901..ec0d11a3 100644 --- a/tests/e2e/testnet.rs +++ b/tests/e2e/testnet.rs @@ -1174,7 +1174,6 @@ impl TestNetwork { cache_capacity: TEST_PAYMENT_CACHE_CAPACITY, close_group_size: replication_config.close_group_size, local_rewards_address: rewards_address, - price_floor: ant_node::payment::PriceFloorConfig::default(), }; let payment_verifier = PaymentVerifier::new(payment_config); diff --git a/tests/poc_price_floor_live.rs b/tests/poc_price_floor_live.rs index 3ba4d877..545a4be4 100644 --- a/tests/poc_price_floor_live.rs +++ b/tests/poc_price_floor_live.rs @@ -6,74 +6,84 @@ //! //! - a **real deployed payment vault** on a local Anvil chain, paid with a //! **real on-chain `pay_for_quotes` transaction**, read back through the -//! production `completedPayments` contract call (NOT the test override); and -//! - the **real `ResponderCommitmentState`** as the floor's commitment source, -//! so the local price is derived through the same production -//! `current_binding_snapshot` → `calculate_price(committed_key_count)` path -//! this node's own `QuoteGenerator` prices from. +//! production `completedPayments` contract call (NOT the test override); +//! - the **real `ResponderCommitmentState`** as this node's own commitment +//! source, read through the same production `current_binding_snapshot` → +//! `calculate_price(committed_key_count)` path this node's own +//! `QuoteGenerator` prices from; and +//! - a gossip cache of **real signed neighbour commitments**, the close-group +//! view the floor prices against. //! //! What each test asserts maps directly to a PR claim: -//! - shadow mode never rejects, even when the settlement is below floor; -//! - enforcement rejects the cheapest-of-K one-quote settlement; -//! - enforcement accepts an honest at-floor overpayment; +//! - the floor refuses a real 3x-baseline settlement; +//! - an honest at-floor overpayment is admitted, and one atto less is not; //! - reading the floor does not mutate commitment answerability. //! //! Run: `cargo test --test poc_price_floor_live --features test-utils`. #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] +use std::collections::HashMap; use std::sync::Arc; -use std::time::SystemTime; +use std::time::{Instant, SystemTime}; +use ant_node::payment::quote::CommitmentSource; use ant_node::payment::{ calculate_price, serialize_single_node_proof, EvmVerifierConfig, PaymentProof, PaymentStatus, - PaymentVerifier, PaymentVerifierConfig, PriceFloorConfig, VerificationContext, + PaymentVerifier, PaymentVerifierConfig, VerificationContext, PRICE_FLOOR_TOLERANCE_PERCENT, +}; +use ant_node::replication::commitment_state::{ + BuiltCommitment, PeerCommitmentRecord, ResponderCommitmentState, }; -use ant_node::replication::commitment_state::{BuiltCommitment, ResponderCommitmentState}; use ant_node::CLOSE_GROUP_SIZE; use evmlib::common::Amount; use evmlib::testnet::Testnet; use evmlib::wallet::Wallet; -use evmlib::{EncodedPeerId, PaymentQuote, ProofOfPayment, RewardsAddress}; +use evmlib::{EncodedPeerId, Network, PaymentQuote, ProofOfPayment, RewardsAddress}; +use saorsa_core::identity::PeerId; use saorsa_pqc::api::sig::{ml_dsa_65, MlDsaPublicKey, MlDsaSecretKey}; +use tokio::sync::RwLock; -/// Committed key count for the receiver in the floor tests. Chosen large -/// enough that `calculate_price(count)` sits well above twice the baseline, so -/// a baseline-priced (cheapest-of-K) settlement is genuinely below a 50% -/// floor. Asserted in-test so a pricing-curve change fails loudly. -const RECEIVER_COMMITTED_KEYS: u32 = 500_000; +/// Committed key count for the receiver. Above every neighbour, so the +/// own-price cap does not bind and the reference is the group median. +const RECEIVER_COMMITTED_KEYS: u32 = 8_000; -/// The receiver's own commitment-derived local price, via the real pricing -/// curve — the exact value the production floor computes. -fn receiver_local_price() -> Amount { - calculate_price(RECEIVER_COMMITTED_KEYS as usize) -} +/// The receiver's close group: fifteen neighbours, exactly the floor's minimum +/// sample, shaped like a production group around a ~4,000-key median. +const NEIGHBOUR_KEY_COUNTS: [u32; 15] = [ + 3_000, 3_200, 3_400, 3_600, 3_800, 4_000, 4_200, 4_400, 4_600, 4_800, 5_000, 5_200, 5_400, + 5_600, 5_800, +]; + +/// Lower median of the neighbours plus the receiver's own (larger) count: the +/// eighth smallest of sixteen. +const GROUP_MEDIAN_KEYS: u32 = 4_400; /// 3x baseline: an honest 3x settlement of the *cheapest* possible quote -/// (committed count 0). This is the cheapest-of-K amount a modified client can -/// settle while still paying an honest 3x of the quote it chose. -fn cheapest_of_k_settlement() -> Amount { +/// (committed count 0). This is the near-empty-quoter amount a modified client +/// can settle while still paying an honest 3x of the quote it chose. +fn baseline_settlement() -> Amount { calculate_price(0) * Amount::from(3u64) } -/// The settled amount an at-floor honest overpayment must reach under a 50% -/// floor: `3 x 50% x local_price`. +/// The settled amount an at-floor honest overpayment must reach: +/// `3 x tolerance x` the group median price. fn at_floor_settlement() -> Amount { - receiver_local_price() * Amount::from(50u64) / Amount::from(100u64) * Amount::from(3u64) + calculate_price(GROUP_MEDIAN_KEYS as usize) * Amount::from(PRICE_FLOOR_TOLERANCE_PERCENT) + / Amount::from(100u64) + * Amount::from(3u64) } fn keypair() -> (MlDsaPublicKey, MlDsaSecretKey) { ml_dsa_65().generate_keypair().unwrap() } -/// Build the receiver's real commitment source, rotated to a commitment over -/// `RECEIVER_COMMITTED_KEYS` keys so `current_binding_snapshot` reports that -/// count — the same object type production wires into both the quote generator -/// and (via this PR) the verifier's floor. -fn receiver_commitment_source() -> Arc { +/// A real signed commitment over `key_count` keys, and its issuer's peer id +/// (BLAKE3 of the public key, matching production binding). +fn signed_commitment(key_count: u32) -> ([u8; 32], BuiltCommitment) { let (public_key, secret_key) = keypair(); let peer_id_bytes = *blake3::hash(&public_key.to_bytes()).as_bytes(); - let entries: Vec<([u8; 32], [u8; 32])> = (0..RECEIVER_COMMITTED_KEYS) + let entries: Vec<([u8; 32], [u8; 32])> = (0..key_count) .map(|i| { let mut k = [0u8; 32]; k[..4].copy_from_slice(&i.to_be_bytes()); @@ -83,7 +93,16 @@ fn receiver_commitment_source() -> Arc { let built = BuiltCommitment::build(entries, &peer_id_bytes, &secret_key, &public_key.to_bytes()) .unwrap(); - assert_eq!(built.commitment().key_count, RECEIVER_COMMITTED_KEYS); + assert_eq!(built.commitment().key_count, key_count); + (peer_id_bytes, built) +} + +/// Build the receiver's real commitment source, rotated to a commitment over +/// `RECEIVER_COMMITTED_KEYS` keys so `current_binding_snapshot` reports that +/// count — the same object type production wires into both the quote generator +/// and the verifier's floor. +fn receiver_commitment_source() -> Arc { + let (_, built) = signed_commitment(RECEIVER_COMMITTED_KEYS); let state = ResponderCommitmentState::new(); state.rotate(built); Arc::new(state) @@ -125,19 +144,38 @@ fn serialize_proof(peer_quotes: Vec<(EncodedPeerId, PaymentQuote)>) -> Vec { serialize_single_node_proof(&proof).unwrap() } -/// Build a verifier wired to the live Anvil vault and the real commitment -/// source, with the given floor policy. No `P2PNode` is attached, so the -/// issuer K-closest check fail-opens under `test-utils` — that isolates the -/// on-chain settlement read and the floor as the load-bearing checks. -fn live_verifier(network: evmlib::Network, floor: PriceFloorConfig) -> PaymentVerifier { +/// Build a verifier wired to the live Anvil vault, the real commitment source +/// and a close-group view of real signed neighbour commitments. No `P2PNode` +/// is attached, so the K-closest set comes from the test override: the +/// neighbours plus every quote issuer in `issuers`. +fn live_verifier( + network: Network, + source: Arc, + issuers: &[&EncodedPeerId], +) -> PaymentVerifier { let verifier = PaymentVerifier::new(PaymentVerifierConfig { evm: EvmVerifierConfig { network }, cache_capacity: 128, close_group_size: CLOSE_GROUP_SIZE, local_rewards_address: RewardsAddress::new([0x11; 20]), - price_floor: floor, }); - verifier.attach_local_commitment_source(receiver_commitment_source()); + let dyn_source: Arc = source; + verifier.attach_local_commitment_source(dyn_source); + + let received_at = Instant::now(); + let mut cache = HashMap::new(); + let mut k_closest = Vec::with_capacity(NEIGHBOUR_KEY_COUNTS.len() + issuers.len()); + for count in NEIGHBOUR_KEY_COUNTS { + let (peer, built) = signed_commitment(count); + cache.insert( + PeerId::from_bytes(peer), + PeerCommitmentRecord::from_verified(built.commitment().clone(), received_at), + ); + k_closest.push(peer); + } + k_closest.extend(issuers.iter().map(|issuer| *issuer.as_bytes())); + verifier.attach_commitment_cache(Arc::new(RwLock::new(cache))); + verifier.set_paid_quote_k_closest_for_tests(k_closest); verifier } @@ -158,103 +196,82 @@ async fn pay_and_build_proof( serialize_proof(vec![(peer_id, quote)]) } -/// Shadow mode (the shipping default) must NEVER reject on the floor, even when -/// the real on-chain settlement is genuinely below the receiver's floor. -#[tokio::test(flavor = "multi_thread")] -async fn shadow_mode_accepts_below_floor_real_settlement() { - let testnet = Testnet::new().await.unwrap(); +fn anvil_wallet(testnet: &Testnet) -> (Network, Wallet) { let network = testnet.to_network(); let wallet = Wallet::new_from_private_key( network.clone(), &testnet.default_wallet_private_key().unwrap(), ) .unwrap(); - let verifier = live_verifier(network, PriceFloorConfig::default()); - assert!(!verifier.price_floor_config().enforce, "default is shadow"); - assert!( - cheapest_of_k_settlement() < at_floor_settlement(), - "test fixture must keep the cheap settlement below the floor" - ); - - let xorname = [0xA1u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([2; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, cheapest_of_k_settlement()).await; - - let status = verifier - .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) - .await - .expect("shadow mode must accept a real below-floor settlement"); - assert_eq!(status, PaymentStatus::PaymentVerified); + (network, wallet) } -/// Enforcement rejects the cheapest-of-K one-quote settlement: a real 3x-baseline -/// on-chain payment against a fuller receiver, read from the real vault. +/// The floor refuses a real 3x-baseline settlement, paid to a near-empty +/// quoter, at a receiver that knows its close group. #[tokio::test(flavor = "multi_thread")] -async fn enforced_floor_rejects_cheapest_of_k_real_settlement() { +async fn rejects_a_real_below_floor_settlement() { let testnet = Testnet::new().await.unwrap(); - let network = testnet.to_network(); - let wallet = Wallet::new_from_private_key( - network.clone(), - &testnet.default_wallet_private_key().unwrap(), - ) - .unwrap(); - let verifier = live_verifier( - network, - PriceFloorConfig { - enforce: true, - tolerance_percent: 50, - }, + let (network, wallet) = anvil_wallet(&testnet); + assert!( + baseline_settlement() < at_floor_settlement(), + "test fixture must keep the baseline settlement below the floor" ); - let xorname = [0xA2u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([3; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, cheapest_of_k_settlement()).await; + let xorname = [0xA1u8; 32]; + let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([2; 20])); + let verifier = live_verifier(network, receiver_commitment_source(), &[&peer_id]); + let proof = pay_and_build_proof(&wallet, peer_id, quote, baseline_settlement()).await; let err = verifier .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) .await - .expect_err("a real 3x-baseline settlement must not clear a fuller receiver's floor"); + .expect_err("a real 3x-baseline settlement must not clear the floor"); assert!( - err.to_string().contains("below this node's price floor"), + err.to_string() + .contains("below the close-group price floor"), "unexpected rejection reason: {err}" ); } -/// Enforcement accepts an honest at-floor overpayment: the client overpays the -/// cheap quote to `3 x 50% x local_price`, settled for real on-chain. +/// Enforcement accepts an honest at-floor overpayment and refuses one atto +/// less: the client overpays the cheap quote to exactly `3 x tolerance x` the +/// group median, settled for real on-chain. #[tokio::test(flavor = "multi_thread")] async fn enforced_floor_accepts_at_floor_real_overpayment() { let testnet = Testnet::new().await.unwrap(); - let network = testnet.to_network(); - let wallet = Wallet::new_from_private_key( - network.clone(), - &testnet.default_wallet_private_key().unwrap(), - ) - .unwrap(); + let (network, wallet) = anvil_wallet(&testnet); let source = receiver_commitment_source(); + assert!( + calculate_price(RECEIVER_COMMITTED_KEYS as usize) + > calculate_price(GROUP_MEDIAN_KEYS as usize), + "the receiver must sit above its group so the reference is the group median" + ); // Snapshot the commitment hash BEFORE verification so we can prove the // floor read did not rotate/refresh/age retention (claim: non-mutating). let hash_before = source.current().map(|c| c.hash()); - let verifier = PaymentVerifier::new(PaymentVerifierConfig { - evm: EvmVerifierConfig { - network: network.clone(), - }, - cache_capacity: 128, - close_group_size: CLOSE_GROUP_SIZE, - local_rewards_address: RewardsAddress::new([0x11; 20]), - price_floor: PriceFloorConfig { - enforce: true, - tolerance_percent: 50, - }, - }); - let concrete = Arc::clone(&source); - let dyn_source: Arc = concrete; - verifier.attach_local_commitment_source(dyn_source); - let xorname = [0xA3u8; 32]; - let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([4; 20])); - let proof = pay_and_build_proof(&wallet, peer_id, quote, at_floor_settlement()).await; + let short_name = [0xA3u8; 32]; + let (short_peer, short_quote) = + signed_quote(short_name, calculate_price(0), RewardsAddress::new([4; 20])); + let xorname = [0xA4u8; 32]; + let (peer_id, quote) = signed_quote(xorname, calculate_price(0), RewardsAddress::new([5; 20])); + let verifier = live_verifier(network, Arc::clone(&source), &[&short_peer, &peer_id]); + + let one_short = at_floor_settlement() - Amount::from(1u64); + let short_proof = pay_and_build_proof(&wallet, short_peer, short_quote, one_short).await; + let err = verifier + .verify_payment( + &short_name, + Some(&short_proof), + VerificationContext::ClientPut, + ) + .await + .expect_err("one atto below the floor must be refused"); + assert!(err + .to_string() + .contains("below the close-group price floor")); + let proof = pay_and_build_proof(&wallet, peer_id, quote, at_floor_settlement()).await; let status = verifier .verify_payment(&xorname, Some(&proof), VerificationContext::ClientPut) .await