chore: repository cleanup and tilebench package layout

[bowencui123]

The repository-cleanup half of the former #312. The README rewrite is #314, stacked on this branch. Since the split this PR has also gained the artifact-storage work (items 2 and 4 below).

What this does

1. Task descriptions get a directory of their own. benchmarks/problems/current/<op>_current.md is a prompt input of the LLM pipeline, not part of an operator's benchmark definition, so the 45 files move to tilebench/problems/, a data directory beside benchmarks/, core/ and data/, reached through PROBLEMS_ROOT in tilebench/paths.py; the 16 unused problems/original files go. Rebuilt iter-0 prompts keep a description block identical to the archived campaign prompts.
2. Raw logs leave version control. The raw logs were force-added despite .gitignore. They are generated artifacts: results/<gpu>/logs/ is Git-ignored, a benchmark run writes its logs there, and that is all it does. run_bench.py performs no Git operation and has no archive hook or flag. The paper's frozen raw logs are preserved on archive/raw-logs-2026-09-18 (item 10).
3. data/peak_performance. .gitkeep dropped, check_peak_specs.py moved in from the repo root, and it now reads measured peaks from B200.json instead of hardcoding 8000 GB/s.
4. LLM generation artifacts leave the main source tree. The generation and evaluation pipeline remains version-controlled under tilebench/llm/, while generated trajectories under tilebench/benchmarks/llm_generated/ are now Git-ignored experimental artifacts. The complete frozen AACL 2026 campaign (45 operators × {gpt-5.5, claude-opus-4-7}, 732 iterations) is preserved on the archive branch and published as a downloadable artifact, with scripts/fetch_artifacts.py restoring it to the expected runtime path. artifacts/manifest.json declares downloadable artifacts (URL, SHA256, archive name, provided directories); scripts/package_artifacts.py builds the reproducible tarball. LLM_GENERATED_ROOT and generate.py are unchanged: same runtime path, different storage policy. The branch history still contains the commits that restored the campaign from exp/llm_and_analysis before it was untracked, so squash-merging keeps those 6,140 files out of main's history.
5. skills/ is tracked. The Triton and cuTile API references were read from the untracked .claude/skills/, so a fresh clone built prompts without them.
6. Sources consolidate under one package. core/, data/, benchmarks/, tools/llm_codegen/ and tilebench_run/ become tilebench/{core,data,benchmarks,llm,profiling}/, with the task descriptions in tilebench/problems/; generated NCU reports move to outputs/, and the profiling command-line tools to scripts/profiling/ (item 13). New tilebench/paths.py resolves every package resource from __file__, replacing relative paths, parent-count repo-root guesses and ten scripts that embedded one machine's absolute checkout path. scripts/ put the repo root on sys.path themselves, so PYTHONPATH=. is no longer needed and they work from any directory. A minimal pyproject.toml makes pip install -e . work.
7. results/ tracks only CSVs. Figures, aggregates and runtime_summary.md are regenerated by visualize.py and aggregate_results.py, so they leave tracking; .gitignore had been ignoring the summary CSVs, the one thing that should be versioned. With item 9 the tracked set is results/<hardware>/csv/**/*.csv for any hardware label, and aggregate CSVs stay ignored.
8. Missing kernel-count metadata is an error. The NCU harness validates every capture against probed kernel counts and names. A missing file now raises, naming the command that produces it, instead of silently assuming one kernel per launch. Where that metadata lives is item 12.
9. Results are scoped by hardware. TileBench will hold results for several GPUs (B200 today; GH200 and an AMD part to follow), and a global results/{csv,logs,figures,aggregate,runs} lets the same operator overwrite itself across machines. The layout is now results/<gpu>/{csv,logs,figures,aggregate,runs}/, built only through helpers in tilebench/paths.py (results_root, results_csv_dir, ...). hardware_label() accepts any single safe path component and rejects traversal; there is no device list, so a new GPU needs a label, not a patch. The 90 committed CSVs are B200 data and move to results/B200/csv/ as a pure rename. run_bench.py, run_bench_all.py, visualize.py, plot_sweep_max.py, aggregate_results.py and ncu_catalogue.py take --gpu, required and with no default, so a run on another machine cannot land in B200's namespace. The PyTorch, Triton, cuTile and TileLang columns of a namespace were measured on that GPU, and the TileLang merge is confined to the CSV of the current --gpu. NKI keeps its original organisation inside the new layout: run_bench.py --gpu B200 --tile-language nki merges torch_nki_ms, nki_ms and speedup_nki into results/B200/csv/, exactly as on main, and its logs and Neuron profiles go to results/B200/logs/ (nki_profiles/, nki_neff_manifest.jsonl). --gpu names the campaign, not the device NKI ran on. These are cross-hardware measurements: speedup_nki = torch_nki_ms / nki_ms, never torch_ms / nki_ms, and NKI latencies get no B200 drift scaling.
10. The archive branch is this branch plus archive-only material, and owns its own tooling. archive/raw-logs-2026-09-18 absorbs this branch by merge and adds what the public tree does not keep: the paper's 272 raw logs, under results/B200/logs/ (a4245d86 moved them there from the flat results/logs/), both LLM snapshots, the B200 NCU metadata, the retired profiling files of item 13, and the archive tooling. That tooling, a maintenance script and its tests, lives on the archive branch only: backing results up targets one dated branch and is a maintainer task, not a feature of the framework, so this PR ships none of it. There the script carries itself and its tests forward, so an archive run on a newer main cannot delete them.
11. Raw JSON is collision-free. results/<gpu>/logs/ held one <op>_results.json and one <op>_autotune.json, so a default run, an autotune run, a TileLang-only run and an NKI run of one operator overwrote each other. Default names are now <op>_<mode>_<backends>.json, for example mul2_default_triton-cutile.json, mul2_autotune_triton-cutile.json, mul2_default_tilelang.json, mul2_default_nki.json, built only by timing_log_path / autotune_log_path. The new tilebench/backends.py owns the backend names, their canonical order and the --tile-language parser, so cutile,triton and triton,cutile are one tag. Explicit --output / --autotune-log still win. Readers name the run they want: visualize.py takes --mode and --tile-language, and ncu_catalogue.py reads exactly <op>_autotune_<backends>.json (default triton,cutile, both required). Nothing is chosen by glob or modification time, and there is no latest.json. With one file per selection the autotune-log merge is removed: it existed to stop a TileLang run clobbering the Triton/cuTile winners, and would now refuse a rerun with different cases. run_bench.py is the only owner of these names: nothing renames or copies results afterwards.
12. Profiling metadata and NCU reports are scoped by hardware, and are generated data. Autotune winners, kernel launch counts and kernel names are measured on one GPU, so the single global pair becomes one directory per GPU. They are written by scripts/profiling/ncu_catalogue.py and scripts/profiling/probe_kernel_count.py and read only by the NCU tools, never by a benchmark code path, so they are experiment data rather than a package resource: they live under the Git-ignored outputs/profiling/<gpu>/{ncu_catalogue,kernel_counts}.json, next to the reports in outputs/ncu/<gpu>/, and nothing is committed for any GPU. The B200 files are kept on the archive branch. NCU_CATALOGUE and KERNEL_COUNTS are removed in favour of ncu_catalogue_path(gpu), kernel_counts_path(gpu) and ncu_output_dir(gpu), which use the same label validation as results_root. ncu_catalogue, probe_kernel_count, ncu_one, ncu_driver, ncu_writeup, hf_upload and plot_sweep_max take --gpu, required and without a default. A GPU without metadata is an error that names the command to produce it; another GPU's files are never used. hf_upload uploads under ncu_report_main/<gpu>/, so it cannot overwrite the released B200 reports, which were not touched. plot_sweep_max.py no longer needs this metadata: the sweep-max case comes from each operator's config.yaml through the rule the catalogue uses, so the README figure is reproducible from the configs and the committed CSVs alone.
13. tilebench/profiling/ is a library; programs live under scripts/. A file is not package source because it ends in .py, so the 17 Python files there were classified by what they do and who imports them. Library, stays (3): __init__.py; ncu_kernel_select.py, kernel selection, capture validation and metadata loading, imported by four tools; ncu_catalogue.py, sweep-max cases and catalogue entries, imported by plot_sweep_max.py, now without its argparse main and sys.path hack. Command-line tools, moved (8): scripts/aggregate_results.py, which aggregates result CSVs and so sits with visualize.py; scripts/profiling/{ncu_catalogue,ncu_driver,ncu_one,ncu_writeup,probe_kernel_count,hf_upload}.py, none of which had an importer outside the tests; and scripts/profiling/ncu_generic_harness.py, which is never imported because it is the process NCU profiles. They put the repository root on sys.path like the other scripts and run from any directory without PYTHONPATH; the drivers locate the harness next to themselves, and the report path rule they duplicated is now tilebench.paths.ncu_report_path. Campaign-specific, removed and kept on the archive (12): five cluster launch scripts (.sh, .sbatch), probe_hung_cfgs.py and six ncu_harness_*.py: hard-coded operators, cases, configs and cluster paths, code that runs at import, the methodology the generic harness replaced, and no reference from any driver, test or document. They are not moved to scripts/. The tilebench.profiling package-data entry is dropped.

Behaviour

Measurement behaviour is unchanged: operator code, case grids, dtypes, tolerances, autotune spaces, timing protocol, CSV values and the peak-performance specs are untouched. For the package move, 6429 of 6507 moved files are byte-identical and the 78 modified carry import/path edits only.

Item 9 deliberately changes the command line:

• --gpu <label> is required by every result-producing script. run_bench.py prints the namespace next to the detected device and warns when the label does not appear in the device name.
• The default backend set, and all, is triton, cutile, tilelang. It used to include nki, which a GPU host only ever recorded as nan. NKI still runs whenever it is named.
• NKI profiling artifacts move from results/logs/nki_profiles/ to results/<gpu>/logs/nki_profiles/. The engine takes a logs_dir and hands the paths to the orchestrator, which no longer has a default location. NKI execution, timing and the CSV merge are unchanged, and its logs are archived with the rest of results/<gpu>/logs/.
• Default raw JSON names change to <op>_<mode>_<backends>.json (item 11). visualize.py gains --mode and --tile-language to name the run it reads, and ncu_catalogue.py gains --tile-language. The archived paper logs keep their original file names.
• The NCU tools require --gpu (item 12), and hf_upload.py uploads under a per-GPU prefix.
• Profiling tools are run as python scripts/profiling/<tool>.py, and the aggregator as python scripts/aggregate_results.py, instead of python -m tilebench.profiling.<tool> (item 13). Cluster launch scripts and campaign-specific harnesses are gone from the public tree.
• run_bench.py writes results and performs no Git operation. Backing results up is not part of a run, and there is no flag for it.
• plot_sweep_max.py writes to results/<gpu>/figures/ unless --output is given, so plotting another GPU cannot overwrite the README figure. Its title names the --gpu label instead of a hard-coded "NVIDIA B200".

Verification

• pytest: 166 passed, 1 skipped. compileall clean. Tests that only fingerprinted one migration (tests/data/b200_csv.sha256, the B200 metadata hashes) or exercised removed launch scripts were deleted, and the 19 archive-script tests moved to the archive branch with the script, where they pass. The CSV schema test, the namespace and isolation tests and the NKI merge tests stay.
• The B200 CSV move is byte-identical: 90 files before and after, SHA256 recorded before git mv and matched after, all 90 git blobs unchanged, 90 renames at 100% similarity with 0 lines added or removed. 56 of the files use CRLF and were not normalised; the move was committed with --no-verify so the whitespace hooks could not touch the data. The hashes were checked at migration time and are recorded here; no fingerprint file is kept in the repository.
• The committed CSVs have the 10 frozen columns and no gpu/device/hardware column: the directory names the hardware. The only extension the guard accepts is the three NKI columns, checked row by row against torch_nki_ms / nki_ms.
• A real run on a B200 with default flags, run_bench.py --gpu SMOKE --operator mul2 --tile-language triton --case-indices 0, started from /tmp, wrote only under results/SMOKE/, printed the namespace with the detected device, raised the label warning, and left the local archive ref untouched. visualize.py --gpu SMOKE read and wrote inside the same namespace. The 90 B200 CSVs were re-hashed afterwards and were unchanged. Omitting --gpu is rejected before anything runs. A test stubs the engine and asserts that run_bench.py itself starts no process.
• The new tests were mutation-checked: restoring the old .gitignore rules, computing speedup_nki from the B200 torch_ms, or drift-scaling nki_ms each makes the corresponding test fail.
• .gitignore was exercised with git check-ignore: results/{B200,GH200,MI300X}/csv/x.csv are trackable; logs, figures, runs, aggregate/x.csv, non-CSV files under csv/ and the legacy results/logs/ are ignored; no hardware name appears in the rules.
• The published LLM artifact contains 6,140 files.
• A real download, SHA256 verification, extraction, and tree comparison reproduced the local frozen campaign exactly.
• tilebench/benchmarks/llm_generated/ remains the canonical runtime output path but is no longer tracked on the main source branch: git ls-files is empty there, git check-ignore matches, and the pipeline still creates the directory on demand.
• The archive branch (tip 734bbd56, no force push) absorbs this branch by git merge --no-ff, five times so far. First merge (5df592a7): 272 rename/delete conflicts, the already-migrated B200 logs, kept; the merge had silently deleted the 284-file legacy LLM snapshot, restored. The later merges had no conflicts; whatever this branch had dropped, seven campaign-specific profiling files and then the three archive tooling files, was restored byte for byte from the last commits that had them (3ba7c669, b6cc83cb). Now the source part of the archive tree is identical to this branch (429 files, 0 differing, 0 missing) and the archive adds 6,713 files: 6,140 LLM, 284 legacy LLM, 272 raw logs, 2 B200 metadata files, 12 retired profiling files and 3 archive tooling files; 429 + 6,713 = 7,142, the size of the archive tree. No previously archived file is missing or changed, and the four data subtrees keep their tree hashes. The archive branch's own copy of its script, run from a clean checkout of this branch, is a no-op, and its 19 tests pass in a worktree of that branch.
• fetch_artifacts.py is tested for checksum mismatch, refusal to overwrite without --force, and rejection of traversal, links and members outside the declared directories.
• 45 operators discovered; all 45 config.yaml parse to identical hashes; all 180 impl_* modules import.
• The 45 rebuilt LLM prompts are byte-identical to the pre-refactor build, and were re-hashed before and after the task descriptions moved to tilebench/problems/ and the pipeline to tilebench/llm/: 45 of 45 unchanged. That move is 55 renames at 100% similarity, committed with --no-verify so the whitespace hooks could not touch prompt inputs. A wheel and an sdist carry the 45 descriptions and the tilebench.llm package, and a test asserts that every operator has a description under PROBLEMS_ROOT.
• The B200 profiling metadata (ncu_catalogue.json sha256 814bb30d…, kernel_counts.json sha256 cfe0687e…) left the public tree without a content change and is on the archive branch with the same blob ids (6db68b1a, 8d8311bb). Peak-performance loads match their pre-move hashes.
• Raw JSON: a sequence of B200 default, B200 autotune, B200 TileLang-only, B200 NKI and GH200 default leaves ten distinct files, each byte-identical to what its own run wrote. On a real B200, --tile-language triton and cutile,triton produced mul2_default_triton.json and mul2_default_triton-cutile.json, and visualize.py, started from /tmp, read the second by name.
• Metadata isolation: building the GH200 catalogue or probing GH200 kernel counts leaves the B200 files byte-identical, and the reverse. The catalogue reads the one named autotune log even when newer decoy logs of other modes and selections sit beside it; adding a fallback to B200 makes a test fail. The sweep-max cases re-derived from config.yaml alone match the former committed catalogue for 45 of 45 operators.
• Packaging: a wheel and an sdist built from a source copy contain exactly three files under tilebench/profiling/ (__init__.py, ncu_catalogue.py, ncu_kernel_select.py), no scripts/, no harness or command-line tool, no .sh, .sbatch or metadata, and still ship the peak-performance JSON, the 45 operator configs and the prompt files. A test installs the package with pip install --target and asserts the same from outside the repository. Another asserts, by parsing the package, that no module under tilebench/profiling/ has a __main__ guard, imports argparse or runs code at import.
• The relocated tools were run for real on a B200, started from /tmp with no PYTHONPATH: ncu_catalogue.py and probe_kernel_count.py for a throwaway label, then ncu_one.py under the ncu binary, which executed the relocated generic harness and validated 1/1 launches. Tests start every tool from outside the repository up to argument parsing, and check that both drivers point at the harness next to them; removing a script's sys.path bootstrap, pointing a driver back into the package, or dropping a program into the package each makes a test fail.
• git grep finds no /projects/kzhou6, no PYTHONPATH=., no _summary.csv consumer, no global metadata constant, no python -m tilebench.profiling command and no reference to a removed launch script or harness outside skills/, which is the frozen prompt input of the paper campaign and is left byte-identical on purpose.
• Smoke benchmarks on B200 for mul2, vector_add, mean_reduction and matmul_int8, plus one autotuned case; all backends discovered.
• pip install -e . resolves package data from outside the repo.

Open PRs

Every open PR was test-merged. 25 merge cleanly; 22 NKI PRs conflict only on their own results/logs/autotune_logs/<op>_autotune.json; none has a semantic conflict. 46 add files under pre-refactor paths and are migrated with:

git -c merge.directoryRenames=false merge origin/main
python scripts/migrate_pr_paths.py

Disabling directory-rename detection matters: with it on, git offers the wrong destination for impl_nki.py.

The 23 open NKI PRs carry over without a data fix. Each edits two results/csv/<op>_{default,autotune}.csv files to append torch_nki_ms, nki_ms and speedup_nki. #307 was test-merged onto this branch: git follows the rename and applies the edit to results/B200/csv/weight_dequant_{default,autotune}.csv; the three columns are kept; all 60 rows of each file keep their frozen PyTorch, Triton, cuTile and TileLang values; speedup_nki equals torch_nki_ms / nki_ms in 60 of 60 rows; and the full suite passes on the merged tree, B200 guard included. The only conflict is the known one on the PR's own results/logs/autotune_logs/<op>_autotune.json, a raw log that main no longer tracks and that belongs on the archive branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA