Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions app/LAUNCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,28 @@ cluster with `fly ssh console -a <app> -C "sh -c 'PGPASSWORD=\$OPERATOR_PASSWORD
-U postgres -d basebid -c ...'"` (port 5433, not the socket). Destroy the rehearsal app afterwards.
For a real recovery: restore to a new app, `fly postgres detach`/`attach` it to `basebid` (DATABASE_URL),
redeploy, run `scripts/smoke.sh`. Point-in-time: `--restore-target-time <RFC3339>` instead of a name.

## Trade-time USD backfill (2026-11-10)

Every trade and fee carries its USD value **at the time it happened** (`bb_launch_swaps.usd`, `bb_launch_fee_events.usd`,
per-launch sums in `bb_launches.volume_usd` / `fees_collected_usd` / `fees_burned_usd`). The indexer prices new ones from
the hourly ledger `bb_quote_usd_hourly`, writing the live price into the hour in progress (`lib/launchpad/usd-history.ts`,
`usdLedger.ts`). Trades from before that went live, and any hour the indexer could not price (a price source down, the
indexer catching up), are filled by the backfill. It is safe to run again at any time: it only prices rows with no USD
value, never replaces an hour the indexer captured live, and stops at the last completed hour.

```bash
cd app
# 1. generate (public price APIs only, no database access): dry run first
npx tsx --tsconfig tsconfig.test.json scripts/usd-backfill.ts --mode dry-run --out /tmp/usd-dry.sql
# 2. run it on the database machine: it prints the report (old method vs trade-time, per chain and quote, totals, fees)
# and ROLLS BACK, so nothing changes
fly ssh sftp shell -a basebid-db <<< "put /tmp/usd-dry.sql /tmp/usd-dry.sql"
fly ssh console -a basebid-db -C "sh -c 'PGPASSWORD=\$OPERATOR_PASSWORD psql -h localhost -p 5433 -U postgres -d basebid -f /tmp/usd-dry.sql'"
# 3. once the report looks right: generate and run --mode apply the same way (it writes, then recomputes every launch's
# USD totals under the same locks as scripts/rebuild-launch-totals.sql; the indexer waits a few seconds)
```

History sources: ETH-USD and HYPE-USD hourly closes from Coinbase Exchange (HYPE gaps from Hyperliquid), GITLAWB from its
GITLAWB/WETH pool on Base via GeckoTerminal (price in WETH between trades) × ETH-USD, TWIG = GITLAWB, stables $1. Stocks,
community pairs and unlisted quotes have no history there: their older trades stay without a USD value (`usd IS NULL`).
3 changes: 3 additions & 0 deletions app/db/concurrent-indexes.sql
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
-- Indexes on the big, busy tables, built by scripts/migrate.mjs AFTER the schema transaction commits, one at a time
-- with CREATE INDEX CONCURRENTLY: reads and writes on the table never wait for the build. (Inside the schema
-- transaction a build would hold the exclusive lock an ALTER TABLE on that table already took, and every query on
Expand All @@ -17,3 +17,6 @@
CREATE INDEX CONCURRENTLY IF NOT EXISTS bb_token_transfers_from_idx ON bb_token_transfers (chain_id, token, from_addr, block_number);
-- boosts (2026-10-15): a boost names its launch by position id; the lane joins on it
CREATE INDEX CONCURRENTLY IF NOT EXISTS bb_launches_chain_token_id_idx ON bb_launches (chain_id, token_id);
-- trade-time USD (2026-11-10): the trades still without a USD value (every one until the first backfill, then only
-- those whose hour had no price yet), so a repeat backfill and the "partial" check never scan every swap
CREATE INDEX CONCURRENTLY IF NOT EXISTS bb_launch_swaps_usd_missing_idx ON bb_launch_swaps (chain_id, block_time) WHERE usd IS NULL;
24 changes: 24 additions & 0 deletions app/db/schema.sql
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
-- basebid.lol · free token launchpad on Base.
--
-- Applied by scripts/migrate.mjs (Fly release_command) against the DEDICATED
Expand Down Expand Up @@ -624,3 +624,27 @@
revoked_at timestamptz
);
CREATE INDEX IF NOT EXISTS bb_admin_sessions_wallet_idx ON bb_admin_sessions (wallet, created_at DESC);

-- ── trade-time USD (2026-11-10, lib/launchpad/usd-history.ts) ───────────────
-- Every trade's and fee's USD value at the time it happened, instead of its quote volume at today's price. One USD price
-- per quote per UTC hour: the live price the indexer saw during that hour (source 'live', updated until the hour ends,
-- never replaced afterwards), or a historical close filled in by scripts/usd-backfill.ts ('coinbase', 'geckoterminal',
-- 'fixed'). `decimals` is the quote's, so a raw amount converts without a second lookup.
CREATE TABLE IF NOT EXISTS bb_quote_usd_hourly (
chain_id integer NOT NULL,
quote text NOT NULL, -- lowercase; 0x000… = the chain's native asset
hour timestamptz NOT NULL, -- start of the UTC hour
usd double precision NOT NULL CHECK (usd > 0), -- per whole quote unit
decimals integer NOT NULL CHECK (decimals BETWEEN 0 AND 36),
source text NOT NULL,
updated_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (chain_id, quote, hour)
);
-- the value at the trade's own hour, fixed once written; NULL = no price for that hour yet (the backfill fills it)
ALTER TABLE bb_launch_swaps ADD COLUMN IF NOT EXISTS usd double precision;
-- quote side only, as the USD fee totals always were: 'collected' (quote_amount) and a 'burned' row in the launch's quote
ALTER TABLE bb_launch_fee_events ADD COLUMN IF NOT EXISTS usd double precision;
-- each launch's sums of those values, added to in the event's own transaction and recomputed by the backfill
ALTER TABLE bb_launches ADD COLUMN IF NOT EXISTS volume_usd double precision NOT NULL DEFAULT 0;
ALTER TABLE bb_launches ADD COLUMN IF NOT EXISTS fees_collected_usd double precision NOT NULL DEFAULT 0;
ALTER TABLE bb_launches ADD COLUMN IF NOT EXISTS fees_burned_usd double precision NOT NULL DEFAULT 0;
125 changes: 125 additions & 0 deletions app/scripts/usd-backfill.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
/**
* Trade-time USD backfill (lib/launchpad/usd-history.ts states the rule). Fetches hourly prices from launch day to now and
* writes ONE SQL file that fills bb_quote_usd_hourly, prices every trade and fee that has no USD value yet at its own
* hour, and recomputes each launch's USD totals. Nothing here touches a database: the file is reviewed, then run on the
* database machine exactly like scripts/rebuild-launch-totals.sql.
*
* npx tsx --tsconfig tsconfig.test.json scripts/usd-backfill.ts --mode dry-run [--from 2026-09-01] [--out file.sql]
* npx tsx --tsconfig tsconfig.test.json scripts/usd-backfill.ts --mode apply [--from 2026-09-01] [--out file.sql]
*
* Dry run: the SQL fills the ledger inside a transaction, prints the report (old method vs trade-time, per chain and
* quote, plus totals) and ROLLS BACK. Apply: same report, then writes the values and recomputes the totals. Both are safe
* to run again: only rows with no USD value are priced, and an hour the indexer captured live is never replaced.
*
* Sources (all public, no key): ETH-USD and HYPE-USD hourly closes from Coinbase Exchange (the live site prices ETH from
* Coinbase too), HYPE hours Coinbase lacks from Hyperliquid; GITLAWB from its GITLAWB/WETH Uniswap v4 pool on Base via
* GeckoTerminal (price in WETH, forward-filled between trades: a pool's price only moves when it trades) times ETH-USD,
* the live method (gitlawb.ts). TWIG takes GITLAWB's price (1:1 wrapper). Stables are $1. Stocks, community pairs and
* unlisted quotes have no history here: their past trades stay without a USD value and count as partial.
*/
import { writeFileSync } from "node:fs";
import { GITLAWB_POOL_ID } from "../src/lib/launchpad/gitlawb.ts";
import {
GITLAWB_HISTORY, HOUR_MS, backfillHours, buildBackfillSql, fillHours, multiplySeries, parseCoinbaseCandles, parseGeckoOhlcv,
parseHyperliquidCandles, quoteSeriesRows, type BackfillMode, type HourPoint, type SeriesKey,
} from "../src/lib/launchpad/usd-history.ts";

const args = new Map<string, string>();
for (let i = 2; i < process.argv.length; i += 2) args.set(process.argv[i].replace(/^--/, ""), process.argv[i + 1] ?? "");
const mode = args.get("mode") as BackfillMode;
if (mode !== "dry-run" && mode !== "apply") throw new Error("--mode dry-run | apply");
// up to the last COMPLETED hour: the hour in progress belongs to the indexer's live capture (usd-history.backfillHours)
const { from, to } = backfillHours(args.get("from") || "2026-09-01T00:00:00Z", Date.now());
const out = args.get("out") || `usd-backfill.${mode}.sql`;
const UA = { "user-agent": "openlaunch-usd-backfill" };
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

async function getJson(url: string, init?: RequestInit, tries = 4): Promise<unknown> {
for (let i = 0; ; i++) {
const res = await fetch(url, { ...init, headers: { ...UA, ...(init?.headers ?? {}) }, signal: AbortSignal.timeout(20_000) });
if (res.ok) return res.json();
if (i + 1 >= tries || (res.status !== 429 && res.status < 500)) throw new Error(`${url.split("?")[0]}: HTTP ${res.status}`);
await sleep(2_000 * (i + 1));
}
}

/** Coinbase Exchange: at most 300 candles per request. */
async function coinbaseHourly(product: string): Promise<HourPoint[]> {
const pts: HourPoint[] = [];
for (let start = from; start <= to; start += 300 * HOUR_MS) {
const end = Math.min(to, start + 299 * HOUR_MS);
const url = `https://api.exchange.coinbase.com/products/${product}/candles?granularity=3600&start=${new Date(start).toISOString()}&end=${new Date(end).toISOString()}`;

Check notice on line 51 in app/scripts/usd-backfill.ts

View workflow job for this annotation

GitHub Actions / pr-guard

pr-guard: A host the repository has not used before

app/scripts/usd-backfill.ts: api.exchange.coinbase.com
pts.push(...parseCoinbaseCandles(await getJson(url)));
await sleep(250);
}
return pts;
}

/** Hyperliquid candleSnapshot: up to 5000 candles per request. */
async function hyperliquidHourly(coin: string): Promise<HourPoint[]> {
const pts: HourPoint[] = [];
for (let start = from; start <= to; start += 4_000 * HOUR_MS) {
const end = Math.min(to + HOUR_MS - 1, start + 4_000 * HOUR_MS - 1);
const body = { type: "candleSnapshot", req: { coin, interval: "1h", startTime: start, endTime: end } };
pts.push(...parseHyperliquidCandles(await getJson("https://api.hyperliquid.xyz/info", { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body) })));
await sleep(250);
}
return pts;
}

/** GeckoTerminal pool OHLCV, newest first, paged backwards with before_timestamp (1000 per page). Price in the pool's other token. */
async function geckoHourlyInPair(network: string, pool: string, token: string): Promise<HourPoint[]> {
const pts: HourPoint[] = [];
let before = Math.floor((to + HOUR_MS) / 1000);
for (let page = 0; page < 20; page++) {
const url = `https://api.geckoterminal.com/api/v2/networks/${network}/pools/${pool}/ohlcv/hour?aggregate=1&limit=1000&currency=token&token=${token}&before_timestamp=${before}`;
const got = parseGeckoOhlcv(await getJson(url, { headers: { accept: "application/json" } }));
if (got.length === 0) break;
pts.push(...got);
const oldest = got[0].hour;
if (oldest <= from) break;
before = Math.floor(oldest / 1000);
await sleep(2_500); // GeckoTerminal's public API allows about 30 calls a minute
}
return pts;
}

function summary(name: string, m: ReadonlyMap<number, number>): string {
const v = [...m.values()];
const hours = (to - from) / HOUR_MS + 1;
return `${name.padEnd(12)} ${String(m.size).padStart(5)}/${hours} hours min ${Math.min(...v).toPrecision(4)} max ${Math.max(...v).toPrecision(4)}`;
}

async function main(): Promise<void> {
const quotes = quoteSeriesRows();
for (const q of quotes) if (q.series === "USD" && q.fixedUsd !== 1) throw new Error(`fixed quote ${q.chain}:${q.quote} is not $1 (${q.fixedUsd}); give it its own series`);

console.error(`hours ${new Date(from).toISOString()} .. ${new Date(to).toISOString()}`);
const eth = fillHours(await coinbaseHourly("ETH-USD"), from, to, 6);
const hypeCb = await coinbaseHourly("HYPE-USD");
const hypeHl = await hyperliquidHourly("HYPE");
// Coinbase first (the live HYPE price's primary source); Hyperliquid only for the hours Coinbase has no candle for
const cbHours = new Set(hypeCb.map((p) => p.hour));
const hype = fillHours([...hypeHl.filter((p) => !cbHours.has(p.hour)), ...hypeCb], from, to, 6);
const gitlawbInWeth = fillHours(await geckoHourlyInPair(GITLAWB_HISTORY.network, GITLAWB_POOL_ID, GITLAWB_HISTORY.token), from, to, 24 * 365);
const gitlawb = multiplySeries(gitlawbInWeth, eth);
const usd = new Map<number, number>();
for (let h = from; h <= to; h += HOUR_MS) usd.set(h, 1);

// sanity bounds: a unit mix-up (a price in wei, a series in the wrong currency) fails here, before anything is written
const bounds: Record<SeriesKey, [number, number]> = { USD: [1, 1], "ETH-USD": [100, 100_000], "HYPE-USD": [1, 10_000], "GITLAWB-USD": [1e-12, 1] };
const series = new Map<SeriesKey, Map<number, number>>([["USD", usd], ["ETH-USD", eth], ["HYPE-USD", hype], ["GITLAWB-USD", gitlawb]]);
for (const [k, m] of series) {
console.error(summary(k, m));
for (const v of m.values()) if (v < bounds[k][0] || v > bounds[k][1]) throw new Error(`${k}: ${v} is outside ${bounds[k].join("..")}`);
}

const sql = buildBackfillSql({ mode, generatedAt: new Date().toISOString(), from, to, quotes, series });
writeFileSync(out, sql);
console.error(`wrote ${out} (${(sql.length / 1024).toFixed(0)} KiB, ${quotes.length} quotes). Run it on the database machine as LAUNCH.md describes.`);
}

main().catch((err) => {
console.error(err instanceof Error ? err.message : err);
process.exit(1);
});
Loading
Loading