Improve benchmark UX, publish PR results, and release 0.2.0

README.md

@@ -49,8 +49,8 @@ abench /path/to/model
 
 Abench looks for `.yaml` and `.yml` files directly inside `/path/to/model/.abench/`
 (no recursive search). In a terminal it lists them alphabetically and asks which
-experiment to run, then prompts for that experiment's inputs. Enter chooses the
-first file; a single file still gets a selection prompt. Only the selected file
+experiment to run, then prompts for that experiment's inputs. Use ↑/↓ to move the highlighted selection and Enter to confirm. The first file
+is initially selected; a single file still gets a selection prompt. Only the selected file
 is loaded. `run`, `validate`, and `prepare` all support directory selection.
 
 `abench /path/to/model --help` lists available files without prompting or running
@@ -196,7 +196,10 @@ vars:
 ```
 
 In a terminal, run, validate, and prepare prompt for each input in YAML order.
-Press Enter to accept a displayed default. Required inputs have no default and
+Inputs with `choices` use the same arrow-key menu, with the declared default
+initially highlighted (or the first option if required). Enter confirms the
+selection and preserves its declared type. Other inputs use text prompts.
+Press Enter to accept a displayed default. Required text inputs have no default and
 must be entered; empty or invalid answers prompt again with an explanation.
 Ctrl-C cancels before source resolution or downloads. Selected values are printed
 and saved in `suite.json` as `input_values`.
@@ -266,13 +269,119 @@ This experiment file describes **which tests to run**. A model profile such as
 across suites.
 
 Terminal progress identifies the experiment number, image build, warmup, and
-measured attempts/retries. Builds and model phases print elapsed time every
-15 seconds; model phases also show current and peak cgroup memory when samples
-are available. Full console output stays in the printed log paths. These are
-status updates, not an estimated completion percentage.
+measured attempts/retries. During builds and model phases, an interactive terminal
+shows live elapsed time, current and peak cgroup memory (when available), and a
+panel with the last 12 lines of the run log. Ctrl-C cancels the command. Full
+console output stays in the printed log paths; elapsed time and memory status
+are saved every 15 seconds in a sibling `console.progress.log` (or
+`build.progress.log` for builds). Redirected output and non-interactive terminals
+only print start and finish summaries. These are status updates, not an estimated
+completion percentage.
+
+## Publishing benchmark results to a PR
+
+A suite can post one results comment after its final comparison report. Add an
+explicit target (the repository is always on github.com):
+
+```yaml
+inputs:
+  pr:
+    type: integer
+    required: true
+    minimum: 1
+publish:
+  github:
+    repository: ActivitySim/activitysim
+    pr: ${pr}
+    baseline: main
+```
+
+`baseline` names a run in `runs` and defaults to the first run. The posting target
+is independent of source selection: use `pr: ${pr}` in an ActivitySim source
+mapping to benchmark that PR's pinned head as well. See
+[`examples/sandag-pr.yaml`](examples/sandag-pr.yaml) for a complete example with a
+pinned baseline; adjust its model/data paths and baseline commit for your model.
+
+```bash
+abench examples/sandag-pr.yaml --set pr=1110
+# Run benchmarks, but only generate the local comment and charts:
+abench examples/sandag-pr.yaml --set pr=1110 --publish-dry-run
+# Inspect or publish retained results without rerunning benchmarks:
+abench publish /path/to/suite-output --dry-run
+abench publish /path/to/suite-output
+# Read local status, or verify the comment still exists on GitHub:
+abench publish /path/to/suite-output --status
+abench publish /path/to/suite-output --verify
+# The permanent launcher works from any working directory:
+/path/to/suite-output/publication/publish.sh
+/path/to/suite-output/publication/publish.sh --dry-run
+```
+
+Publication requires GitHub CLI 2.99 or newer with `gh pr comment --attach`, an
+authenticated account, and repository write access for image uploads. Use
+`gh auth login --hostname github.com` (OAuth), or a classic personal access token
+through `GH_TOKEN`. Other token types and GitHub Enterprise Server image uploads
+are not supported by this first release. `gh` remains optional for ordinary runs
+and publication dry runs. Authentication and PR access are checked before asset
+preparation or benchmark execution when publication is enabled; validation and
+prepare-only commands do not publish. Tokens are never written to experiment
+metadata or passed into benchmark containers.
+
+The comment contains elapsed time and peak memory, percentage changes against the
+baseline for valid runs with matching comparison settings, run settings, source
+commits, memory traces, and component-runtime charts. Invalid runs have no change
+claims and are excluded from the runtime chart; unstarted runs are identified.
+PR head/base commits are recorded at startup, and a head change during execution
+is noted when posting. These are observations, not statistical significance tests.
+
+The `publication/` directory retains `comment.md`, `memory.svg`, `runtimes.svg`,
+and `state.json` with the comment ID/URL and publication status. `STATUS.md` shows
+whether publication is pending, failed, uncertain, or published, with attempt time,
+errors, a retry command, and the comment link. It reflects local knowledge;
+`--verify` checks GitHub without uploading or editing anything. Verification
+failures preserve the previous publication record. A missing previously published
+comment is flagged and is not automatically recreated.
+
+The executable `publish.sh` is permanent: it is safe to call repeatedly and passes
+through `--dry-run`, `--status`, or `--verify`. It prefers `abench` on PATH, then
+uses `uvx --refresh --from <recorded-checkout>` when a development checkout was
+available when the bundle was prepared. It reports an actionable error if neither
+is available and never stores credentials. Output paths with spaces are supported.
+Existing suites can create these helpers with `abench publish OUTPUT --status`
+without reading measurement data or contacting GitHub.
+
+The final console message explicitly says whether results were published, gives
+the comment URL on success, or gives the launcher command when action is needed.
+A connection failure after posting begins is marked as an uncertain outcome;
+retrying reconciles the comment marker before attempting another post.
+
+The exact upload body is retained as `upload.md`. A retry searches for the suite's
+unique comment
+marker before posting; a recorded successful publication is a no-op. Independent
+suite executions get separate comments. Simultaneous publication of the same
+output directory is blocked. Partial uploads can leave unused GitHub attachments,
+but the local reports are retained and a retry can complete the comment.
+
+A posting error fails the command and prints the retry command. If the benchmark
+also failed, its failure remains primary and the publication error is printed
+separately. The full HTML/JSON comparison remains local; this release uploads only
+the summary and SVG figures, not raw logs, model data, or the interactive HTML.
+Images must fit GitHub's 10 MB attachment limit, and summaries are limited to
+60,000 characters. Publication transport tests simulate GitHub; a live attachment
+smoke test should use a designated test PR, never a production PR by default.
 
 ## Run controls
 
+On macOS, benchmark commands automatically run `/usr/bin/caffeinate -i` to prevent
+idle system sleep for the entire invocation, including preparation, builds,
+warmup, measured attempts, reporting, and PR publication. The display can still
+sleep. The assertion is released on completion or cancellation; it also expires
+if the abench process exits unexpectedly. Other platforms are unchanged.
+Use `--allow-sleep` to opt out, for example `abench experiments.yaml --allow-sleep`.
+Report-only, publish-only, validation, preparation-only, and help commands do not
+start caffeinate. This prevents idle sleep; it does not override explicit sleep
+or closing a laptop lid.
+
 - `--single-process` (default), or `--multiprocess --processes N`. The count applies
   to every sliced stage; coordinators are additional processes.
 - `--sharrow` (default) or `--no-sharrow`. Sharrow enabled requires its source pin.

examples/sandag-pr.yaml

@@ -0,0 +1,58 @@
+# Run from any directory: abench /path/to/abench/examples/sandag-pr.yaml
+schema_version: 1
+# Publishing is opt-in through this block. Use --publish-dry-run to preview locally.
+publish:
+  github:
+    repository: ActivitySim/activitysim
+    pr: ${pr}
+    baseline: main
+inputs:
+  pr:
+    type: integer
+    required: true
+    minimum: 1
+    description: ActivitySim PR to benchmark and comment on
+  households:
+    type: integer
+    default: 28365
+    minimum: 0
+    description: Households to sample from benchmarking-data; 0 uses all
+  processes:
+    type: integer
+    default: 4
+    minimum: 1
+  warmup_households:
+    type: integer
+    default: 5000
+    minimum: 1
+vars:
+  model: ../../sandag-abm3-example
+  activitysim_main: 5c6fae24a91a57a2d6dfc2e1dbe062a61d94545a
+output_root: ../experiments/sandag-pr${pr}-${timestamp}
+defaults:
+  model_dir: ${model}
+  profile: sandag
+  data_dir: ${model}/benchmarking-data
+  config_overlay: ["${model}/configs_explicit_chunk"]
+  multiprocess: true
+  processes: ${processes}
+  sharrow: true
+  households: ${households}
+  # Cache misses in the measured run trigger a retry using the compiled flows.
+  warmup_households: ${warmup_households}
+  memory: 80g
+  shm_size: 8g
+  platform: linux/arm64
+  sources:
+    - sharrow=ActivitySim/sharrow@fc175b27d8e0c5d202721c67d96b050e6117b235
+runs:
+  main:
+    label: SANDAG main — explicit chunking
+    sources:
+      - activitysim=ActivitySim/activitysim@${activitysim_main}
+  candidate:
+    label: SANDAG PR${pr} — explicit chunking
+    sources:
+      - name: activitysim
+        repository: ActivitySim/activitysim
+        pr: ${pr}

pyproject.toml

@@ -10,7 +10,7 @@ readme = "README.md"
 requires-python = ">=3.10"
 license = "BSD-3-Clause"
 license-files = ["LICENSE"]
-dependencies = ["PyYAML>=6", "platformdirs>=3", "zstandard>=0.21"]
+dependencies = ["PyYAML>=6", "platformdirs>=3", "zstandard>=0.21", "prompt_toolkit>=3.0.36,<4"]
 keywords = ["activitysim", "benchmark", "transportation", "memory", "sharrow"]
 classifiers = [
     "Development Status :: 3 - Alpha",
@@ -40,6 +40,17 @@ abench = "abench.cli:entrypoint"
 [tool.setuptools.package-data]
 abench = ["Dockerfile", "profiles/*.yaml"]
 
+# uvx --from <checkout> must notice source edits, not just metadata changes.
+[tool.uv]
+cache-keys = [
+    { file = "pyproject.toml" },
+    { file = "README.md" },
+    { file = "MANIFEST.in" },
+    { file = "src/abench/**/*.py" },
+    { file = "src/abench/Dockerfile" },
+    { file = "src/abench/profiles/*.yaml" },
+]
+
 [tool.ruff]
 line-length = 88
 

src/abench/__init__.py

@@ -1,3 +1,3 @@
 """Reproducible ActivitySim experiments in Linux containers."""
 
-__version__ = "0.1.2"
+__version__ = "0.2.0"

src/abench/awake.py

@@ -0,0 +1,58 @@
+"""Keep macOS hosts awake for the lifetime of a benchmark CLI invocation."""
+
+import os
+import subprocess
+import sys
+from contextlib import contextmanager
+
+
+def benchmark_invocation(argv):
+    """Exclude reporting, validation, data preparation, and informational commands."""
+    return not (
+        (argv and argv[0] in {"report", "publish", "validate", "prepare"})
+        or any(
+            flag in argv
+            for flag in ("--help", "-h", "--version", "--report-only", "--allow-sleep")
+        )
+    )
+
+
+@contextmanager
+def keep_awake(enabled=True):
+    if not enabled or sys.platform != "darwin":
+        yield
+        return
+    # -i prevents idle system sleep without keeping the display on. -w also
+    # releases the assertion if abench dies without running Python cleanup.
+    try:
+        process = subprocess.Popen(
+            ["/usr/bin/caffeinate", "-i", "-w", str(os.getpid())],
+            stdin=subprocess.DEVNULL,
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+        )
+    except OSError as error:
+        raise ValueError(
+            "Cannot prevent macOS idle sleep with caffeinate. "
+            "Use --allow-sleep to run without sleep prevention."
+        ) from error
+    try:
+        # Detect launch failures before committing to an expensive run.
+        try:
+            code = process.wait(timeout=0.05)
+        except subprocess.TimeoutExpired:
+            print("macOS idle sleep prevention enabled (caffeinate).", flush=True)
+        else:
+            raise ValueError(
+                f"caffeinate exited unexpectedly (exit {code}); "
+                "use --allow-sleep to run without sleep prevention."
+            )
+        yield
+    finally:
+        if process.poll() is None:
+            process.terminate()
+            try:
+                process.wait(timeout=5)
+            except subprocess.TimeoutExpired:
+                process.kill()
+                process.wait()

src/abench/cli.py

@@ -17,6 +17,7 @@
 
 from . import __version__
 from .attempts import measured_attempts
+from .awake import benchmark_invocation, keep_awake
 from .common import read_json, write_json
 from .failures import BenchmarkFailure, describe_failure
 from .flow_cache import publish_flows, reuse_flows
@@ -60,7 +61,12 @@ def positive(value):
 def parser():
     p = argparse.ArgumentParser(
         description=__doc__,
-        epilog="Named experiments: abench experiments.yaml [--set NAME=VALUE]; preflight: abench validate experiments.yaml [--set NAME=VALUE]; data only: abench prepare experiments.yaml",
+        epilog="Named experiments: abench experiments.yaml [--set NAME=VALUE]; preflight: abench validate experiments.yaml [--set NAME=VALUE]; data only: abench prepare experiments.yaml; publish retained results: abench publish OUTPUT_DIRECTORY [--dry-run]",
+    )
+    p.add_argument(
+        "--allow-sleep",
+        action="store_true",
+        help="disable automatic macOS idle sleep prevention",
     )
     p.add_argument("--version", action="version", version=f"abench {__version__}")
     p.add_argument("--model-dir", type=Path, default=Path.cwd())
@@ -233,6 +239,35 @@ def container_phase(spec, output, data, image, phase_name):
 def main(argv=None):
     p = parser()
     argv = list(sys.argv[1:] if argv is None else argv)
+    if argv and argv[0] == "publish":
+        from .publishing import publish
+
+        publish_parser = argparse.ArgumentParser(prog="abench publish")
+        publish_parser.add_argument("output_directory", type=Path)
+        publish_mode = publish_parser.add_mutually_exclusive_group()
+        publish_mode.add_argument(
+            "--dry-run",
+            action="store_true",
+            help="generate the local bundle without GitHub access",
+        )
+        publish_mode.add_argument(
+            "--status",
+            action="store_true",
+            help="show recorded publication status without contacting GitHub",
+        )
+        publish_mode.add_argument(
+            "--verify",
+            action="store_true",
+            help="check publication status on GitHub without posting",
+        )
+        options = publish_parser.parse_args(argv[1:])
+        publish(
+            options.output_directory,
+            dry_run=options.dry_run,
+            status_only=options.status,
+            verify=options.verify,
+        )
+        return 0
     # A file invocation stays separate from model profiles and ordinary flags.
     candidate = argv[1:] if argv and argv[0] in ("run", "validate", "prepare") else argv
     if (
@@ -272,6 +307,16 @@ def main(argv=None):
             action="store_true",
             help="use defaults and --set values without prompting (required inputs must be supplied)",
         )
+        suite_parser.add_argument(
+            "--publish-dry-run",
+            action="store_true",
+            help="run benchmarks and prepare the comment and charts without publishing",
+        )
+        suite_parser.add_argument(
+            "--allow-sleep",
+            action="store_true",
+            help="disable automatic macOS idle sleep prevention",
+        )
         suite_args = suite_parser.parse_args(candidate)
         interactive = not suite_args.non_interactive and sys.stdin.isatty()
         selected = select_experiment(suite_args.experiment_file, interactive)
@@ -282,6 +327,7 @@ def main(argv=None):
             validate_only=argv[0] == "validate",
             assignments=suite_args.set,
             interactive=interactive,
+            **({"publish_dry_run": True} if suite_args.publish_dry_run else {}),
         )
     action = argv.pop(0) if argv and argv[0] in ("run", "report", "validate") else "run"
     args = p.parse_args(argv)
@@ -599,7 +645,9 @@ def publish_cache():
 def entrypoint():
     """Expose CLI errors without an unnecessary Python traceback."""
     try:
-        sys.exit(main())
+        with keep_awake(enabled=benchmark_invocation(sys.argv[1:])):
+            code = main()
+        sys.exit(code)
     except KeyboardInterrupt:
         print("\nBenchmark cancelled.", file=sys.stderr)
         sys.exit(130)