Skip to content

infra: camera-ready repository cleanup and README refresh - #312

Closed
bowencui123 wants to merge 19 commits into
mainfrom
bowen/infra/camera_ready
Closed

infra: camera-ready repository cleanup and README refresh#312
bowencui123 wants to merge 19 commits into
mainfrom
bowen/infra/camera_ready

Conversation

@bowencui123

@bowencui123 bowencui123 commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Repository cleanup and README refresh for the camera-ready release. One commit per concern.

  1. benchmarks/problems removed. problems/current/<op>_current.md is a prompt input of the LLM codegen pipeline, so the 45 descriptions move to tools/llm_codegen/problems/ and prompt_builder.py / generate_descriptions.py are repointed. The 16 unused problems/original files are deleted.
  2. Raw logs untracked, archived automatically. results/logs/ (272 JSON files, force-added despite .gitignore) leaves main; the summary CSVs stay. run_bench.py, the only writer of results/logs/, now calls scripts/archive_logs.sh after every run. The snapshot is a local commit on archive/raw-logs-2026-09-18; nothing is checked out or pushed, a failure only prints a warning, and --no-archive skips it. Publish with scripts/archive_logs.sh --push. The archive tree is origin/main + the logs already archived + results/logs/ from disk, so a machine holding only some of the logs adds to the archive instead of replacing it.
  3. data/peak_performance. .gitkeep dropped; check_peak_specs.py moved in from the repo root and now reads the measured peaks from B200.json instead of hardcoding the 8000 GB/s datasheet value.
  4. benchmarks/llm_generated restored. PR Code-only cherry-pick from exp/llm_and_analysis: operator fixes, llm_codegen updates, sweep tooling #98 cherry-picked the llm_codegen code without its data, leaving two pilot operators. This restores the campaign from exp/llm_and_analysis: 45 operators × {gpt-5.5, claude-opus-4-7}/high, identical to the tree on that branch. The pipeline-development runs from llm_codegen: iterative LLM kernel-generation pipeline (Triton + cuTile) #95 (high_CHEAT, high_no_skill, high_pre_*, xhigh) are removed.
  5. skills/ added. prompt_builder read the Triton and cuTile API references from the untracked .claude/skills/, so a fresh clone built prompts without them. skills/{triton,cutile}-guide/SKILL.md are the versions the paper's campaign used.
  6. README. Restructured around the paper: headline results (pairwise win/loss matrix, suite geomeans, autotune gains) with the sweep-max latency figure from scripts/plot_sweep_max.py, the overview diagram, backend scope, the paper's environment next to the current development environment, a quick start, the evaluation methodology, the LLM track, and a collapsible developer reference holding the CLI tables and the former OPERATOR_AUTHORING_GUIDE.md. Stale content is brought up to date (project structure, 2× L2 eviction, CLI flags, Triton/cuTile examples, peak values, FP8 status re-checked on the current stack).

Verification

  • pytest tests: 75 passed, 1 skipped.
  • README: the win/loss matrix (37/8, 36/9, 33/12; 14 operators within 5%) recomputes exactly from results/csv; every operator sweeps 20 cases per dtype; section, appendix and table references were checked against the paper sources. On GitHub's render of this branch all in-page links resolve, both images load, and the three collapsible blocks render. The quick-start package pins exist on their indexes; cuda-tile-experimental was dropped from the install line because it is not on PyPI and nothing imports it.
  • Rebuilt iter-0 prompts reproduce the API-reference block and the description block of all 90 archived prompts byte for byte.
  • Token counts of the restored campaign match the archived RQ4 trajectory data for 731 of 732 iterations; the remaining one has no llm_usage.json in the source branch.
  • archive_logs.sh exercised on temporary refs: first snapshot differs from the base only under results/logs/; a rerun is a no-op; a fresh worktree holding one new log adds it and keeps the other 316; a clone with no results/logs/ is a no-op; a remote tip ahead of the local one is adopted; a changed log on disk wins. The working tree and index are untouched throughout.
  • End to end: a one-case run_bench.py --operator mul2 run created the snapshot, and --no-archive did not.
  • plot_sweep_max.py matches exactly one CSV row per operator to the NCU-catalogue sweep-max case (it asserts this).
  • The pre-commit hooks pass without rewriting any file, so the restored LLM artifacts and skills/ stay byte-identical to their sources.

Effect on the other open PRs

Every open PR was test-merged against this branch. 25 merge cleanly. 22 NKI PRs conflict, each in exactly one file, results/logs/autotune_logs/<op>_autotune.json (modified there, deleted here); none has any other conflict. After this PR lands, each is fixed by merging main and git rm-ing that file, which leaves only impl_nki.py and the two CSVs in the PR. The dropped log versions go to the archive branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA

bowencui123 and others added 19 commits September 18, 2026 10:33
@bowencui123 @claude
chore(llm_codegen): move task descriptions next to the pipeline; drop…
bbd8b61 
… benchmarks/problems

benchmarks/problems/current/<op>_current.md is a prompt input of the LLM
codegen pipeline, not a benchmark asset. Move the 45 descriptions to
tools/llm_codegen/problems/ and repoint prompt_builder and
generate_descriptions. The 16 unused files under problems/original are
removed. Rebuilt iter-0 prompts keep a description section identical to the
archived campaign prompts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
chore(results): stop tracking raw logs; add archive_logs.sh
fec244e 
results/logs/ was force-added despite being git-ignored. Untrack the 272
raw timing/autotune JSON files; the summary CSVs stay tracked.
scripts/archive_logs.sh snapshots the local results/logs/ onto
archive/raw-logs-2026-09-18, building each commit as <origin/main tree> +
results/logs/ in a throwaway index so that branch always equals main plus
the raw logs. Nothing is checked out.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
chore(peak_performance): move check_peak_specs.py beside the peak files
4b5f139 
Drop the .gitkeep placeholder (the directory holds B200.json and
Trainium2.json). check_peak_specs.py now reads the measured peak bandwidth
and per-dtype peak TFLOPS from B200.json instead of hardcoding the
8000 GB/s datasheet value.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
data(llm_generated): restore the full two-model LLM campaign
30ff150 
PR #98 cherry-picked the llm_codegen code without its data, leaving main
with two pilot operators from #95. Restore the campaign from
exp/llm_and_analysis: 44 operators x {gpt-5.5, claude-opus-4-7}/high, each
with iter_N/ prompts, responses, kernels, feedback and token usage plus
final/ and run_summary.json. Token counts match the archived RQ4 trajectory
data iteration by iteration.

Removed: the pipeline-development runs kept from #95
(flash_attention high_CHEAT / high_no_skill) and the vector_add directory.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
docs(readme): camera-ready README
24e28f5 
- New tagline (accelerator framework; cuTile on CUDA 13.2, Triton, TileLang,
  AWS Neuron NKI, PyTorch); NKI added to the backend list.
- Figures under assets/: the paper's overview diagram and a new sweep-max
  latency chart (scripts/plot_sweep_max.py; PyTorch / Triton / cuTile at the
  NCU-catalogue case of each operator).
- Software Versions section with the stack the results were measured on.
- OPERATOR_AUTHORING_GUIDE.md merged in as a section; the duplicated
  Operator Config / Autotune / Adding a New Operator sections are dropped.
- Stale content brought up to date: project structure, 2x-L2 eviction size,
  CLI flags (--autotune, --tile-language, --gpu), Triton and cuTile examples
  (best_config / CutileAutotuner, no global state), peaks read from
  data/peak_performance/<GPU>.json, FP8 status re-verified on this stack.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
data(llm_generated): restore the vector_add campaign runs
d33b2fc 
Bring back vector_add/{gpt-5.5,claude-opus-4-7}/high so the archive covers
all 45 operators of the paper's LLM track. benchmarks/llm_generated is now
identical to the tree on exp/llm_and_analysis.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
feat(llm_codegen): ship the Triton and cuTile API references under sk…
a00dfa0 
…ills/

prompt_builder read the API references from .claude/skills/, which is not
tracked, so a fresh clone built prompts without them. Add
skills/{triton,cutile}-guide/SKILL.md and read them from there.

The files are the versions the paper's campaign used (the cuTile guide
predates the later ct.static_iter section). Rebuilt iter-0 prompts now
reproduce the API-reference and description blocks of all 90 archived
prompts byte for byte.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
feat(results): archive raw logs automatically after every run_bench
3ebae04 
run_bench.py is the only writer of results/logs/ (run_bench_all.py writes to
results/runs/), so it now calls scripts/archive_logs.sh once the timing and
autotune JSON are written. The snapshot is a local commit on the archive
branch; nothing is checked out or pushed, a failure only prints a warning,
and --no-archive skips it.

archive_logs.sh now accumulates: the tree is origin/main + the logs already
archived (remote and local tips) + results/logs/ from disk, with the disk
winning. A machine holding only some of the logs therefore adds to the
archive instead of replacing it. Offline runs fall back to the last fetched
refs, and the ref update is compare-and-swap.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
docs(readme): corrected overview figure, version note, auto-archive, …
483bc3f 
…skills/

- assets/overview.png: corrected figure ("Verify").
- Software Versions: the TileBench paper was measured with cuda-tile 1.3.0;
  the TileBench++ work (TileLang, NKI) uses 1.5.0.
- Document the automatic raw-log archive, --no-archive, and skills/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
docs(readme): restructure around the paper results
2a85c11 
Lead with the paper's headline results (pairwise win/loss matrix, suite
geomeans, autotune gains) and the sweep-max latency figure, then the
benchmark overview, backend scope, paper vs. development environments,
quick start, evaluation methodology, the LLM track, and a collapsible
developer reference (CLI tables and the operator authoring guide).

Checked against the paper sources and the recorded CSVs: the win/loss
matrix (37/8, 36/9, 33/12; 14 operators within 5%) recomputes exactly from
results/csv, every operator sweeps 20 cases per dtype, and the section,
appendix and table references resolve.

The quick-start install no longer lists cuda-tile-experimental: it is not
on PyPI and nothing in the repository imports it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
refactor: consolidate TileBench sources under the tilebench package
07c8540 
Move the framework sources out of the repository root into one importable
package, so imports are unambiguous and package resources resolve from the
package location instead of the current working directory.

  core/              -> tilebench/core/
  data/              -> tilebench/data/
  benchmarks/        -> tilebench/benchmarks/
  tools/llm_codegen/ -> tilebench/llm_codegen/
  tilebench_run/     -> tilebench/profiling/   (source, catalogue)
  tilebench_run/ncu/ -> outputs/ncu/           (generated reports)

New tilebench/paths.py resolves every package resource from __file__:
operator configs, peak-performance JSON, LLM prompt inputs and the NCU
catalogue. Callers that hardcoded relative paths (engine.py's
benchmarks/operators/<op>/config.yaml), recomputed the repo root by parent
count, or embedded this machine's absolute checkout path (ten profiling
scripts) now go through it. scripts/ stay the user entry points and put the
repository root on sys.path themselves, so they no longer need PYTHONPATH=.
and work from any directory.

Peak performance is split by kind: the framework-consumed device specs stay
tracked at tilebench/data/peak_performance/<GPU>.json with unchanged
contents, while the detailed measure_peak.py sweep becomes a generated
output under the git-ignored outputs/peak_performance/ and is no longer
tracked under results/.

tilebench/profiling/hf_upload.py gains a __main__ guard: it uploads at
import time, which was harmless for a loose script but not for a module of
an importable package.

Behaviour is unchanged. Verified: 45 operators discovered, all 45 config.yaml
parse to identical hashes, all 180 impl_* modules import, the 45 rebuilt LLM
prompts are byte-identical to the pre-move build, peak-performance and NCU
catalogue loads match, and 6429 of 6507 moved files are byte-identical (the
78 modified carry import/path edits only). Committed with --no-verify so the
whitespace hooks cannot rewrite 65 pre-existing files during a pure move.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
tools: add scripts/migrate_pr_paths.py for rebasing pre-refactor bran…
0fa58d1 
…ches

The open pull requests were written against the old source layout. This
moves any file still under a pre-refactor source directory onto the
tilebench package and rewrites the imports inside it, so each branch is a
merge plus one command:

    git merge -c merge.directoryRenames=false origin/main
    python scripts/migrate_pr_paths.py

Disabling directory-rename detection is load-bearing. With it on, git
guesses a destination for a file added inside a renamed directory and picks
the wrong one for the NKI operator PRs, offering
tilebench/benchmarks/llm_generated/<op>/<model>/high/final/impl_nki.py
instead of tilebench/benchmarks/operators/<op>/impl_nki.py.

Verified on #294 (single operator), #303 (cross-operator import) and #259
(45 operators at once). Temporary: delete it once those branches land.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
chore: keep only CSV benchmark results tracked
b645387 
results/ now tracks only the summary CSVs. The per-operator figures, the
aggregate tables and runtime_summary.md are regenerated by visualize.py and
aggregate_results.py, so they leave version control; outputs/ was already
declared the generated-artifact tree by the layout refactor but 49 NCU
writeups and probe results were still tracked inside it.

Nothing is deleted from the working tree: the 12 GB of .ncu-rep reports, the
NCU writeups, the aggregates and the figures all stay on disk, now ignored.

.gitignore previously ignored results/csv/ — exactly the one thing that
should be version controlled — while the generated siblings were tracked by
force-add. It now ignores results/* and re-includes results/csv/, verified
with git check-ignore on existing, new and nested CSV paths.

probe_kernel_count.py and ncu_writeup.py gained the mkdir their siblings
already had: with outputs/ untracked, a fresh clone has no outputs/ncu/ to
write into, and probe_kernel_count.py would otherwise fail after completing
the whole probe.

CSV contents are untouched: the 90 tracked files and their checksums are
identical before and after.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123 @claude
refactor: move kernel-count metadata into the profiling package
c153413 
kernel_counts.json records how many kernels one impl.run() launches per
(op, dtype, backend) and under which names. The NCU harness validates every
capture against it, so it is profiling input rather than a profiling report
and belongs beside ncu_catalogue.json, not in the git-ignored outputs/ tree
it shared with the 12 GB of .ncu-rep files.

  outputs/ncu/kernel_counts.json -> tilebench/profiling/kernel_counts.json

The file is copied byte for byte (sha256
cfe0687e48665968e09e62ac3def2e86ae3bed14aa4a12a46175dcce79b666a6, 118850
bytes, 224 entries covering all 45 operators); no probe was re-run and no
value was touched. tilebench.paths gains KERNEL_COUNTS, so every reader
resolves it from the package instead of the working directory.

Missing metadata is no longer silent. Both drivers used to skip the file
when absent and then assume one kernel per launch, which disables
wrong-kernel detection without saying so. ncu_kernel_select now owns the
loader: an absent file raises MissingKernelCountsError, which the two CLIs
turn into a clear message naming probe_kernel_count.py, while an individual
unprobed pair still falls back to 1 and warns on stderr, since a single
pair that errored during probing should not abort a whole sweep.

probe_kernel_count.py writes the canonical path. A full run rewrites it as
before; an ONLY_OP run now merges that operator's rows into the existing
file, which previously truncated it to the probed operator alone. The
probing algorithm, operator set, launch-count detection and JSON schema are
unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
@bowencui123
docs: add developer guide
14198b3
@bowencui123
docs: rewrite README for benchmark repository
29ca688
@bowencui123
docs: add TileBench icon
2a807b9
@bowencui123
docs: add TileBench icon to README
4c9cb6c
@bowencui123
Clean up README.md formatting
ddad492 
Removed extra blank lines in README.md
This was referenced Sep 19, 2026
Open
Open
@bowencui123

bowencui123 commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by the split Keren asked for:

Together those two reproduce this branch's tree byte for byte: bowen/docs/readme and bowen/infra/camera_ready have identical trees (231beba6). Closing this one in favour of the pair; bowen/infra/camera_ready is left untouched as a reference.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bowencui123