Repository navigation
benchsdk: benchmark result filters on api + cli - #537
Merged
Merged
Conversation
@benchsdk/api: getBenchmarkResults and listRuns accept participants /
since / until / timeZone (+ steps on results) and sourceOrg (routes via
/benchmarks/~/{sourceOrg}/{slug}). listRuns now returns
{ items, filter?, upgrade? }; both responses carry the server's filter
echo. New types: BenchmarkRunFilterInput, BenchmarkResultsFilterEcho,
BenchmarkRunStepOverview, BenchmarkUpgradePayload, BenchmarkRunList,
ListRunsOptions.
@benchsdk/cli: bench runs list + bench results gain --participant,
--steps (results only), --since/--until (YYYY-MM-DD, ISO, or relative
24h/7d/2w resolved client-side), --day, --tz (defaults to machine zone),
--source-org. --json prints the raw response including the filter echo;
a filter flag against a platform without the feature exits non-zero.
Results table gains one row per run x participant x step with --steps.
Includes a changeset for a minor bump of both packages. Callers updated:
benchsdk tests and benchmarks/scale scripts (listRuns returns a page).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
…n minting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
dtice25
marked this pull request as ready for review
October 23, 2026 02:13
Contributor License AgreementAll contributors are covered by a CLA. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
✅ Devin resolved the finding on e52436a
Fixed by Devin (1)
- Empty filter lists return unfiltered results
View all findings in Devin Review
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Client/CLI half of the benchmark result filters feature (platform half: computesdk/benchmarks-platform#848, spec:
docs/2026-10-23-benchmark-result-filters-mcp-tools.mdin that PR). Makes "how did e2b'screatestep do over the last 7 days?" a three-command answer.@benchsdk/api:getBenchmarkResults+listRunsacceptparticipants/since/until/timeZone(+stepson results) andsourceOrg, which routes through/benchmarks/~/{sourceOrg}/{slug}for shared-slug disambiguation.listRunsnow returnsBenchmarkRunList = { items, filter?, upgrade? }instead of a bare array — breaking, but the only in-repo consumers (benchsdk tests,benchmarks/scalescripts) are updated here.BenchmarkResultsOverviewgainsfilter(server echo) andupgrade;items[]gainssteps?: BenchmarkRunStepOverview[](with realfirstStartedAt/lastCompletedAtnow that the platform populates them).participants/stepsas comma lists;steps: 'all'passes through.@benchsdk/cli(bench runs list+bench results):--participant <list>,--steps <list>|all(results only — the/runsroute rejects it),--since/--until(YYYY-MM-DD, ISO, or relative24h/7d/2wresolved client-side to ISO instants),--day YYYY-MM-DD(sugar for since=until=that day; conflicts with--since/--until),--tz(IANA zone, defaults to the machine's),--source-org.--jsonprints the raw response including thefilterecho. When a filter flag was sent but the response carries nofilter, the command fails loudly ("platform does not support result filters yet") instead of printing unfiltered results under a filtered-looking command.resultstable: one row per run×participant as before, plus a Steps table (run×participant×step) when--stepsis sent.runs list(the all-benchmarks listing has no filter) and are rejected with--run(single-run results)..agents/skills/benchsdk-cli/SKILL.md("Investigating one provider") updated.Changeset for a minor bump of both packages.
pnpm typecheckclean;benchsdk(128 tests) andbenchsdk-cli(32 tests) pass — one env-gated live integration test (integration.test.ts) still fails against the live platform atupdateRun(untouched path, pre-existing).Link to Devin session: https://app.devin.ai/sessions/d90b25244b0649fba4f9f0f0a9648365
Open in Devin Desktop: https://app.devin.ai/desktop/session/d90b25244b0649fba4f9f0f0a9648365?variant=devin
Requested by: @dtice25