[0x19]

Implements: RFC 0163

What changes

atlas crawl had a fixed pace of one request a second. The owner asked for it to be
configurable, 10 a second by default, and for a crawl to show a progress bar as soon as it
knows how much there is. This is the agent half; the CLI half (iohr init --crawl) follows
in the sdk repo.

• --rate <n>: requests a second per host. Default 10, at least 1, at most 50.
  A robots.txt Crawl-delay that is slower still wins, per host, and never speeds a crawl up
  (the existing slow_to keeps the slower of the two).
• --progress: one JSON line per event on standard error, version 1:
  {"event":"progress","v":1,"scope":"<registrable domain>","phase":"pages","read":12,"failed":1,"queued":40,"total":120,"cap":2000}.
  Written after every page, when a sitemap gives the total, and when the listings start and
  end. total is the start page plus the sitemap entries, never below read + failed + queued,
  capped by --max-pages, absent until a sitemap has been read, then sticky. No page URL,
  title or text is ever in a line; only the scope (the host's registrable domain) and counts.
  Fields are only added; readers ignore unknown ones.
• The crawl state file keeps the sitemap count so a resumed run reports the same total.
• docs/docs-connectors.md: Politeness, the new Progress paragraph, the flags table, the tests line.

Backing off

A 429 or a 503 from a host halves that host's rate (never below 1 a second, never raised again in the same crawl), after the existing Retry-After wait and retry. The halved pace applies from the request after the retry. The client is shared, so the other connectors that use it back off the same way. Test: a_429_or_a_503_halves_the_pace_and_it_stays_halved, which reads the arrival times on the site's side (it failed before the change).

The ceiling of 50 a second

One agent, one request in flight per host. Above about 50 a second the agent stops being a
polite reader and starts to look like a load test to a small site, and a rate that high only
helps on a site that would not need a crawl at all. The floor of 1 keeps the old behaviour
reachable. Out-of-range values are refused by clap (exit 2), not silently clamped on the
command line; the library clamps (per_minute_for_rate) so a caller cannot get round it.
The ceiling of 50 is a judgement, not a measured limit; the default of 10 is the owner's (2026-11-08). One const to change either.

Tests (crates/iohr-agent/tests/crawl_rate.rs, 14 tests)

Failing before: on origin/main the file does not compile (progress, DEFAULT_RATE,
MAX_RATE, MIN_RATE, crawl_reporting, per_minute_for_rate and the flags do not exist).
Passing after: 13 passed. The pace tests stamp each request as the server receives it, so
gaps are server-side arrival times, not the crawler's own clock.

• requests_are_spaced_to_the_rate_asked, the_default_rate_is_ten_a_second,
  the_slowest_rate_is_one_request_a_second, the_rate_is_per_second_and_bounded
• a_slower_crawl_delay_wins_over_the_rate
• progress_gives_a_total_once_a_sitemap_has_said, a_sitemap_counts_the_start_page_once,
  a_site_without_a_sitemap_has_no_total_only_a_count, the_total_is_never_more_than_the_page_budget,
  a_progress_line_holds_the_scope_and_counts_and_no_url
• a_rate_out_of_range_is_refused, the_binary_keeps_the_rate_and_reports_progress,
  the_binary_says_nothing_extra_without_progress (the real iohr-agent binary against a
  small local server)

Checks

Built and run on the build host (the Mac is under the build floor), against this head.

cargo +1.99.0 fmt --all --check                      exit 0
cargo +1.99.0 clippy -p iohr-agent --all-targets -- -D warnings   exit 0
cargo +1.99.0 nextest run -p iohr-agent --locked --all-features   484 passed, 2 skipped
cargo +1.99.0 test -p iohr-agent --doc --locked      0 failed
repo:check (typos, shellcheck, actionlint, zizmor)   exit 0
bash tools/ci-changed.sh --plan                      repo:check check

Not run here: cargo deny and capture:ebpf-check (the eBPF crate needs bpf-linker, which
the build host lacks; this change does not touch it or any dependency). A workspace-wide
clippy stops on that crate for the same reason, so clippy ran on iohr-agent.

Compatibility and release

Additive: no flag, field or output changed. An older agent refuses --rate and --progress
as unknown flags (clap usage error, exit 2); the sdk change runs the agent once more without them when it sees exactly that refusal (it does not probe atlas crawl --help). Pre-1.0:
a feat is a patch bump. Roll order: release the agent first, then the CLI.
No monitor applies (a command-line flag, no endpoint).