Skip to content

browser-throughput: pause 15s between sessions so Wikipedia stops 429-ing provider egress - #535

Merged
HeyGarrison merged 1 commit into
masterfrom
devin/1792702691-browser-throughput-stagger
Oct 22, 2026
Merged

HeyGarrison merged 1 commit into
masterfrom
devin/1792702691-browser-throughput-stagger

Conversation

@HeyGarrison

Copy link
Copy Markdown
Collaborator

Summary

Browser providers report Wikipedia answering 429 during our runs. The browser runs on the provider's network, so every page load comes from that provider's egress IPs: the CI default of 100 sessions back to back (sessions take ~3–5s) is ~300 Wikipedia loads in ~6 minutes from one range, which the loop then records as the provider's failure (Oct 16 daily: kernel 60 actions, hyperbrowser 36, steel 4 all status=429).

browser-throughput.bench.ts now sets staggerDelayMs: 15_000 in its config. The bench runs with groupBy: 'round', where the runner sleeps staggerDelayMs between rounds — with --provider X in CI each round is one session, so this is a fixed gap between sessions. The sleep is outside every timed action, so actionsPerSecond / taskMs / screenshotMs are unchanged; a scheduled run goes from ~6 min to ~32 min per provider (within the 60-min job timeout and the daily 90-min deadline). --stagger-delay-ms still overrides it per invocation.

Not changed: browser.bench.ts (navigates example.com only) and the stealth concurrency sweep (its c-level bursts are the thing being measured; spacing them would change the metric).

Link to Devin session: https://app.devin.ai/sessions/504ed394c05242fca787c1737e55fe48
Open in Devin Desktop: https://app.devin.ai/desktop/session/504ed394c05242fca787c1737e55fe48?variant=devin
Requested by: @HeyGarrison

@devin-ai-integration

Copy link
Copy Markdown
Contributor

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@HeyGarrison
HeyGarrison marked this pull request as ready for review October 22, 2026 21:00
@HeyGarrison
HeyGarrison merged commit b07042f into master Oct 22, 2026
2 of 3 checks passed
@open-cla

open-cla Bot commented Oct 22, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@computesdk

computesdk Bot commented Oct 22, 2026

Copy link
Copy Markdown

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Kernel 97.3 0.04s 0.04s 0.16s 0.05s 0.32s 10/10
2 Browseruse 96.7 0.19s 0.03s 0.06s 0.03s 0.32s 10/10
3 Tilion 94.5 0.04s 0.07s 0.07s 0.02s 0.36s 10/10
4 Browserbase 93.6 0.15s 0.23s 0.17s 0.13s 0.75s 10/10
5 Hyperbrowser 93.1 0.22s 0.16s 0.22s 0.09s 0.72s 10/10
6 Steel 87.0 0.39s 0.68s 0.16s 0.24s 1.48s 10/10
7 Notte 76.4 0.37s 0.41s 0.75s 0.10s 1.70s 9/10

View full run · SVG available as build artifact

@computesdk

computesdk Bot commented Oct 22, 2026

Copy link
Copy Markdown

Browser Throughput Benchmark Results

# Provider Score APS (med) Task (med) Task (p95) Screenshot Status
1 Tilion 86.5 7.27/s 1.38s 2.06s 103ms 3/3
2 Browserbase 80.4 5.80/s 1.73s 1.91s 155ms 3/3
3 Steel 68.3 3.27/s 3.06s 3.19s 297ms 3/3
4 Browseruse 46.7 3.60/s 2.41s 3.29s 325ms 2/3
5 Hyperbrowser 45.6 3.28/s 2.84s 3.28s 342ms 2/3
6 Kernel 31.0 8.48/s 0.14s 1.18s 184ms 1/3
7 Notte 0.0 0.00/s 0.17s 0.77s 0ms 0/3

View full run · SVG available as build artifact

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant