Skip to content

fix(search): retry Keenable fetch on 429 using Retry-After - #670

Closed
ilya-bogin-keenable wants to merge 1 commit into
LibreChat-AI:mainfrom
keenableai:fix/keenable-scraper-429-retry
Closed

ilya-bogin-keenable wants to merge 1 commit into
LibreChat-AI:mainfrom
keenableai:fix/keenable-scraper-429-retry

Conversation

@ilya-bogin-keenable

Copy link
Copy Markdown
Contributor

Keenable's fetch API has a per-second rate limit (10 rps on the keyless endpoint, per org when keyed) and answers a burst with HTTP 429 and Retry-After: 1. KeenableScraper.scrapeUrls fetches every link at once and never retries, so a burst loses pages. A self-hosted LibreChat user hit this and moved page fetching to Firecrawl (maxexcloo/kubelab#619):

[web_search] scrape links=5 ok=3 chars=27.0k failed=2 reasons={http_429:2}

This PR retries a 429 up to twice per URL. It waits Retry-After seconds (1s if the header is missing, capped at 3s) plus up to 500 ms of jitter so the retries don't land together. Any other error fails at once, as before, and no other scraper, default or config changes. New unit tests cover a 429 followed by a 200, three 429s in a row (reported as a failure, as today), Retry-After being honored, capped and falling back (fake timers), and a 500 not being retried.

Live check through KeenableScraper, keyless, 15 Wikipedia URLs at once: before the change 10/15 succeeded and 5 failed with 429. After it, 15/15 succeeded in about 2.3s, with 5 requests retried once.

Keenable enforces a per-second rate limit (10 rps keyless, per org when keyed) and answers bursts with 429 and Retry-After. scrapeUrls fetches every link at once, so a burst lost pages. Retry a 429 up to twice, waiting Retry-After seconds (1s fallback, 3s cap) plus jitter. Other errors are not retried.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant