Fix ad reporting and throttle crawlers through x402 gateway - #259
Merged
Merged
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| try { | ||
| const client = connection(); | ||
| if (!client) throw new Error("unavailable"); | ||
| const retryMs = Number(await client.eval(REQUEST_LIMIT_LUA, keys.length, ...keys, ...args)); |
| redis = new Redis(`redis://${mapping}`, { maxRetriesPerRequest: 1, connectTimeout: 2000 }); | ||
| const args = ['semrushbot:ad', 3600, 12, 60_000, 600, 3_600_000]; | ||
| const keys = ['test:ip', 'test:family', 'test:metrics']; | ||
| const claim = () => redis.eval(REQUEST_LIMIT_LUA, keys.length, ...keys, ...args); |
| await claim(); | ||
| assert.ok(await redis.pttl('test:ip') <= before, 'rejects must not extend the deadline'); | ||
| await redis.set('test:family', '600', 'PX', 3_600_000); | ||
| const acrossIps = await redis.eval(REQUEST_LIMIT_LUA, 3, 'test:other-ip', 'test:family', 'test:metrics', ...args); |
ThreatCrush Security Scan48 finding(s) HIGH/CRITICAL: 2 | MEDIUM: 31 | LOW: 15
Snippets are redacted; ThreatCrush never prints matched credential material. |
| expect(model.campaigns[0]).toMatchObject({ clicks: 14, impressions: 110, spentCents: 40 }); | ||
| expect(model.slots[0]).toMatchObject({ clicks: 14, impressions: 110, earnedCents: 20 }); | ||
| expect(model.daily).toHaveLength(7); | ||
| for (const [table, field, value] of filters.filter(([, field]) => field === "owner_id")) expect(value, table).toBe("owner"); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Ads TUI showed billed clicks as all accepted clicks and mislabeled free clicks as invalid. API-token reports lacked
auth.uid(), then fell back to slow lifetime views that could return incomplete HTTP-200 responses and zero domain rows.This replaces that fallback with an owner-scoped, windowed reporting RPC over rollups and live records. Billed, free and rejected clicks remain separate; failed reads return retryable 503 without fabricated totals. The dashboard includes domain traffic/CoinPay attribution, potential scoring, loading/retry feedback and hqtui 0.6.2, and uses matching periods for targets and paid CPC.
Commercial crawlers, including Semrush, use the existing x402-gateway $1/day pass. Redis enforces 12 requests/minute/IP and 600/hour/crawler family even for paid passes, ahead of ad queries. Browser ad requests have a 60/minute/IP ceiling. Robots excludes ad redirects; authorized crawler redirects never write or charge ad clicks. Administrators can inspect separate aggregate crawler counters.
Validation: 2,366 unit/contract tests passed (9 optional integrations skipped), root typecheck and production webpack build passed. Disposable PostgreSQL tests cover owner isolation, execution permissions and click/date semantics; Redis tests cover concurrent admission, retry expiry and IP rotation. Those integration checks are now in CI. A rolled-back production validation exactly matched raw seven-day records; the new query took 132–394 ms for 7/30/365-day reports.
Deployment requires
20260913170000_ad_token_earnings.sqlfirst. It has been applied to production with service-role-only execution. No historical click records were changed. Seedocs/crawler-access.mdfor policy and operations details.