fix: honor BASE_TIMEOUT/BASE_MAX_RETRIES in scrape_as_html and extract - #172
Open
karaposu wants to merge 1 commit into
Open
fix: honor BASE_TIMEOUT/BASE_MAX_RETRIES in scrape_as_html and extract#172karaposu wants to merge 1 commit into
karaposu wants to merge 1 commit into
Conversation
BASE_TIMEOUT and BASE_MAX_RETRIES are applied by the base_request wrapper, but scrape_as_html and extract's scrape stage POSTed to the same unlocker /request endpoint with raw axios -- so the documented knobs were honored by scrape_as_markdown and the batch tools while these two silently ignored them. An operator setting BASE_TIMEOUT=30 saw it "work sometimes": markdown scrapes timed out properly, HTML scrapes and extract hung indefinitely. Route both calls through base_request. With the env vars unset this is byte-identical (one attempt, timeout 0 = axios default, same error instances); with them set, both tools now time out and retry exactly as their siblings do on the same endpoint. Worst case per scrape becomes (retries+1) x timeout, and a retried scrape re-issues the request, matching the existing contract of the four already-wrapped tools. Adds test/base-request-coverage.test.js: a source-level invariant that every api.brightdata.com/request call site uses base_request -- it fails on the pre-fix code (2 of 6 sites raw) and guards against a new tool being added with a raw axios /request call. Note: on this base a malformed BASE_MAX_RETRIES still NaN-poisons base_request's retry loop for all wrapped tools; that parsing fix is part of PR brightdata#163, which is best merged before or alongside this.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
BASE_TIMEOUTandBASE_MAX_RETRIESare documented, operator-facing settings applied by thebase_requestwrapper — butscrape_as_htmlandextract's scrape stage POST to the same unlocker/requestendpoint with rawaxios, so the two knobs are silently ignored there. An operator whosets
BASE_TIMEOUT=30sees it "work sometimes": markdown scrapes time out properly while HTML scrapesand
extracthang indefinitely, with nothing marking the difference.Fix
Route both calls through
base_request— a one-token swap at each site; both configs are alreadyfield-identical to
scrape_as_markdown's wrapped call. With the env vars unset, behaviour isbyte-identical (one attempt,
timeout: 0= axios's own default, same error instances rethrown). Withthem set, both tools now time out and retry exactly as their siblings do on the same endpoint.
Behaviour change (only when the knobs are configured)
Worst case per scrape becomes (retries+1) × timeout, and a retried scrape re-issues the request —
matching the existing contract of the four already-wrapped tools. A timeout surfaces as a clear error
instead of an indefinite hang.
Tests
test/base-request-coverage.test.js: a source-level invariant that everyapi.brightdata.com/requestcall site usesbase_request. It fails on the pre-fix code (2 of 6 sitesraw) and guards against a future tool being added with a raw
axios/requestcall.Notes
BASE_MAX_RETRIESparsing this wrapper depends on(on current
main, a malformed value NaN-poisons the retry loop for all wrapped tools); the fixesare otherwise independent.
extractfunction — whichever merges second is a trivialrebase.