Skip to content

fix: honor BASE_TIMEOUT/BASE_MAX_RETRIES in scrape_as_html and extract - #172

Open
karaposu wants to merge 1 commit into
brightdata:mainfrom
karaposu:fix/base-request-coverage
Open

fix: honor BASE_TIMEOUT/BASE_MAX_RETRIES in scrape_as_html and extract#172
karaposu wants to merge 1 commit into
brightdata:mainfrom
karaposu:fix/base-request-coverage

Conversation

@karaposu

Copy link
Copy Markdown

Problem

BASE_TIMEOUT and BASE_MAX_RETRIES are documented, operator-facing settings applied by the
base_request wrapper — but scrape_as_html and extract's scrape stage POST to the same unlocker
/request endpoint with raw axios, so the two knobs are silently ignored there. An operator who
sets BASE_TIMEOUT=30 sees it "work sometimes": markdown scrapes time out properly while HTML scrapes
and extract hang indefinitely, with nothing marking the difference.

Fix

Route both calls through base_request — a one-token swap at each site; both configs are already
field-identical to scrape_as_markdown's wrapped call. With the env vars unset, behaviour is
byte-identical (one attempt, timeout: 0 = axios's own default, same error instances rethrown). With
them set, both tools now time out and retry exactly as their siblings do on the same endpoint.

Behaviour change (only when the knobs are configured)

Worst case per scrape becomes (retries+1) × timeout, and a retried scrape re-issues the request —
matching the existing contract of the four already-wrapped tools. A timeout surfaces as a clear error
instead of an indefinite hang.

Tests

test/base-request-coverage.test.js: a source-level invariant that every
api.brightdata.com/request call site uses base_request. It fails on the pre-fix code (2 of 6 sites
raw) and guards against a future tool being added with a raw axios /request call.

Notes

BASE_TIMEOUT and BASE_MAX_RETRIES are applied by the base_request wrapper,
but scrape_as_html and extract's scrape stage POSTed to the same unlocker
/request endpoint with raw axios -- so the documented knobs were honored by
scrape_as_markdown and the batch tools while these two silently ignored
them. An operator setting BASE_TIMEOUT=30 saw it "work sometimes": markdown
scrapes timed out properly, HTML scrapes and extract hung indefinitely.

Route both calls through base_request. With the env vars unset this is
byte-identical (one attempt, timeout 0 = axios default, same error
instances); with them set, both tools now time out and retry exactly as
their siblings do on the same endpoint. Worst case per scrape becomes
(retries+1) x timeout, and a retried scrape re-issues the request, matching
the existing contract of the four already-wrapped tools.

Adds test/base-request-coverage.test.js: a source-level invariant that every
api.brightdata.com/request call site uses base_request -- it fails on the
pre-fix code (2 of 6 sites raw) and guards against a new tool being added
with a raw axios /request call.

Note: on this base a malformed BASE_MAX_RETRIES still NaN-poisons
base_request's retry loop for all wrapped tools; that parsing fix is part of
PR brightdata#163, which is best merged before or alongside this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant