Skip to content

docs: rewrite README and add the developer guide - #314

Open
bowencui123 wants to merge 17 commits into
bowen/infra/repo-cleanupfrom
bowen/docs/readme
Open

bowencui123 wants to merge 17 commits into
bowen/infra/repo-cleanupfrom
bowen/docs/readme

Conversation

@bowencui123

@bowencui123 bowencui123 commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

The documentation half of PR #312. Stacked on #313, so the diff below is only the 11 documentation and asset files; retarget the base to main once #313 lands.

What this does

  • README rewritten around the results a reader wants first: the pairwise win/loss matrix, the suite geometric means, the sweep-max latency figure and the overview diagram, then the suite, backends, installation, quick start, methodology and the LLM track.
  • docs/developer_guide.md added: the CLI reference plus the former OPERATOR_AUTHORING_GUIDE.md, which is removed from the repository root.
  • assets/: the TileBench logo as the page title (cropped to its content; a transparent variant for light mode and the original cream card for dark mode, so the black wordmark stays readable), the overview diagram with a transparent background, the sweep-max latency figure, and four link icons. Link icons appear only in the top resource bar and the Recorded Results table; inline body links are plain Markdown. Each ## section heading carries an emoji, in the style of the SWE-bench README, and the table-of-contents links use the matching #-name anchors.
  • Artifact access. The README links the three places results live outside main: the 220 raw Nsight Compute reports on Hugging Face (free login required), the raw logs on archive/raw-logs-2026-09-18, and the LLM campaign on Google Drive. It states that tilebench/benchmarks/llm_generated/ is the Git-ignored runtime output path, that a fresh clone does not ship the 6,140 campaign files, and that python scripts/fetch_artifacts.py --artifact llm-aacl2026 restores them.
  • Hardware-scoped results. Follows item 9 of chore: repository cleanup and tilebench package layout #313. Every command passes --gpu B200, and result paths read results/<hardware>/{csv,logs,figures,aggregate,runs}/. The text says that this is a general layout, that results/B200/csv/ holds the paper results committed today, and that other hardware sits beside it without a code change. The developer guide documents the label rules, what a namespace holds, the scope of the TileLang merge, and the per-GPU archive command. Both documents state that NKI results are stored with the B200 campaign as cross-hardware measurements taken on AWS Trainium: three columns in results/B200/csv/, logs and profiles under results/B200/logs/, with speedup_nki = torch_nki_ms / nki_ms and never torch_ms / nki_ms. The headline B200 numbers are unchanged, and no empty GH200 or AMD directory is created.
  • Multi-hardware consistency. Follows items 10 to 13 of chore: repository cleanup and tilebench package layout #313. Profiling metadata is documented as tilebench/profiling/metadata/<hardware>/{ncu_catalogue,kernel_counts}.json, with B200 as the directory committed today, and NCU reports as outputs/ncu/<hardware>/. The 220 released reports are described as the paper's B200 profiles; the count is unchanged. Raw JSON names carry the mode and the backend selection: the README shows the two files its quick start produces and passes --tile-language to visualize.py, and the developer guide adds the naming rule, --mode, the ncu_catalogue log contract and the --gpu commands of the NCU tools. The paper's B200 raw logs are stated to live under results/B200/logs/ on the archive branch; the sentence that described results/logs/ as their current location is gone.

Verification

  • The win/loss matrix (37/8, 36/9, 33/12) and the "14 operators within 5%" claim recompute exactly from results/B200/csv/; the geometric means (2.02×, 1.58×, 1.18×, 1.22×), the suite composition (26 from TritonBench, 19 from LeetGPU) and the category counts match the paper sources.
  • Every operator sweeps 20 cases per dtype, as stated.
  • All 22 CLI flags cited across both documents exist in the corresponding --help; the commands no longer need PYTHONPATH= and were run as written. The flags added for the results layout (--gpu on four scripts, --output on plot_sweep_max.py, --results-root) were checked against --help as well, and the README quick-start form of run_bench.py was run for real on a B200 under a throwaway label.
  • On GitHub's render of this branch all 38 in-page links resolve, the logo, both figures and the seven link icons load, and the cross-links between README and the developer guide work both ways.
  • Every path reflects the layout from chore: repository cleanup and tilebench package layout #313; no pre-refactor path remains.
  • The documented fetch command was run for real against the published Google Drive file: download, SHA256 match, extraction, and a tree comparison that reproduced the 6,140-file campaign exactly.
  • The four external targets (Hugging Face dataset, Google Drive file, archive branch, developer guide) return HTTP 200. pytest on this branch: 183 passed, 1 skipped.

Two things to decide before merge

  1. Decided: the backend table keeps TileLang "Results: N/A". The table is unchanged in this PR. For reference, all 45 committed CSVs under results/B200/csv/ carry tilelang_ms and speedup_tilelang (suite geometric mean 1.72× over PyTorch); they are not part of the headline comparison.
  2. The citation section and BibTeX entry were dropped in the rewrite. Intentional while the proceedings entry is pending, or should they come back?

Separately, the repository still has no LICENSE file while the README tells people to pip install -e ..

🤖 Generated with Claude Code

https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA

bowencui123 and others added 17 commits September 19, 2026 06:34
- New tagline (accelerator framework; cuTile on CUDA 13.2, Triton, TileLang,
  AWS Neuron NKI, PyTorch); NKI added to the backend list.
- Figures under assets/: the paper's overview diagram and a new sweep-max
  latency chart (scripts/plot_sweep_max.py; PyTorch / Triton / cuTile at the
  NCU-catalogue case of each operator).
- Software Versions section with the stack the results were measured on.
- OPERATOR_AUTHORING_GUIDE.md merged in as a section; the duplicated
  Operator Config / Autotune / Adding a New Operator sections are dropped.
- Stale content brought up to date: project structure, 2x-L2 eviction size,
  CLI flags (--autotune, --tile-language, --gpu), Triton and cuTile examples
  (best_config / CutileAutotuner, no global state), peaks read from
  data/peak_performance/<GPU>.json, FP8 status re-verified on this stack.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
(cherry picked from commit 24e28f5)
…skills/

- assets/overview.png: corrected figure ("Verify").
- Software Versions: the TileBench paper was measured with cuda-tile 1.3.0;
  the TileBench++ work (TileLang, NKI) uses 1.5.0.
- Document the automatic raw-log archive, --no-archive, and skills/.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
(cherry picked from commit 483bc3f)
Lead with the paper's headline results (pairwise win/loss matrix, suite
geomeans, autotune gains) and the sweep-max latency figure, then the
benchmark overview, backend scope, paper vs. development environments,
quick start, evaluation methodology, the LLM track, and a collapsible
developer reference (CLI tables and the operator authoring guide).

Checked against the paper sources and the recorded CSVs: the win/loss
matrix (37/8, 36/9, 33/12; 14 operators within 5%) recomputes exactly from
results/csv, every operator sweeps 20 cases per dtype, and the section,
appendix and table references resolve.

The quick-start install no longer lists cuda-tile-experimental: it is not
on PyPI and nothing in the repository imports it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
(cherry picked from commit 2a85c11)
(cherry picked from commit 14198b3)
(cherry picked from commit 2a807b9)
(cherry picked from commit 4c9cb6c)
Removed extra blank lines in README.md

(cherry picked from commit ddad492)
Separate the three things that used to read as one: the version-controlled
pipeline under tilebench/llm_codegen/, the git-ignored local output under
tilebench/benchmarks/llm_generated/, and the paper's frozen snapshot restored
with fetch_artifacts.py. The download link is marked as not yet published.
Recorded Results now lists where each kind of generated artifact is kept, and
the developer guide documents archive_artifacts.sh and the publishing flow.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Add a resource row under the description, in the spirit of KernelBench's
arXiv | blog | dataset line: NCU reports on Hugging Face, the LLM artifact on
Google Drive, the raw-log archive branch and the developer guide, each led by
its service's icon. The same icons mark those links where they appear in the
body and in the Recorded Results table, which gains a row for the Nsight
Compute reports.

The 220 reports (45 operators, Triton and cuTile, every profiled dtype,
11.8 GB) are already public at huggingface.co/datasets/bcui2/NCU_report; the
README now says so, and that downloading needs a free Hugging Face login
because the dataset is gated with automatic approval.

Icons are vendored under assets/icons/ rather than hot-linked: the Hugging
Face logo from its brand-assets repository (viewBox cropped to the glyph),
the Google Drive product icon from Wikimedia Commons, the GitHub mark from
Simple Icons and the book glyph from Octicons, the last two in a neutral grey
that stays visible in GitHub's dark theme. Each icon sits inside its link's
text, so GitHub does not wrap it in a link to the image file.

Also fixes the tagline, which followed </div> without a blank line and was
therefore rendered with its literal ** markers.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
- Put the TileBench logo at the top as the page title. The supplied artwork
  is cropped to its content (top and bottom margins removed). The light
  variant has the page background removed; the dark variant keeps the
  original cream card so the black wordmark stays readable.
- Replace the overview figure with the transparent-background version. It is
  pixel-identical to the previous one when flattened on white.
- Keep icons in the top resource bar and the Recorded Results table only;
  inline body links are plain Markdown. Section anchors are unchanged.
- NCU wording: "The released artifact contains 220 raw Nsight Compute
  reports."
- State that tilebench/benchmarks/llm_generated/ is the Git-ignored runtime
  output path, that a fresh clone does not ship the 6140 campaign files, and
  that fetch_artifacts.py restores them automatically.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Updated the overview section to provide a clearer description of TileBench and its supported backends.
Emoji prefix on the 14 `##` headings, in the style of the SWE-bench README.
GitHub drops the emoji from the heading slug but keeps the following space,
so the 12 table-of-contents links move from `#name` to `#-name`. No other
content changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Follows the results layout change on the infra branch. Every command in the
README and the developer guide passes --gpu B200, and the result paths read
results/<hardware>/{csv,logs,figures,aggregate,runs}/. The text states that
this is a general layout, that results/B200/csv/ holds the paper results
committed today, and that other hardware sits beside it without a code
change. The developer guide documents the label rules, the one-namespace-
one-hardware rule, TileLang merge scope, NKI output under outputs/nki/, and
the per-GPU archive command. Headline B200 numbers are unchanged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
…data

Follows the correction on the infra branch. The previous text said NKI was
kept out of the GPU namespaces, under outputs/nki/. NKI keeps its original
organisation instead: its columns (torch_nki_ms, nki_ms, speedup_nki) are
merged into results/B200/csv/, and its logs and profiles live under
results/B200/logs/. The README and the developer guide now say that these
are measurements taken on AWS Trainium, stored beside the B200 results for
a unified per-operator record, that speedup_nki = torch_nki_ms / nki_ms and
never torch_ms / nki_ms, and that --gpu names the campaign rather than the
device NKI ran on.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
…chive logs

Follows the multi-hardware consistency work on the infra branch.

- Profiling metadata is documented as
  tilebench/profiling/metadata/<hardware>/{ncu_catalogue,kernel_counts}.json,
  with B200 as the directory committed today and other hardware beside it.
  NCU reports are written to outputs/ncu/<hardware>/. The 220 released
  reports are described as the paper's B200 profiles; the count is unchanged.
- Raw JSON names carry the mode and the backend selection. The README shows
  the two files its quick start produces and passes --tile-language to
  visualize.py, which now names the run it reads. The developer guide adds
  the naming rule, --mode, the ncu_catalogue log contract and the --gpu
  commands of the NCU tools.
- The paper's B200 raw logs are stated to live under results/B200/logs/ on
  the archive branch. The sentence describing results/logs/ as their current
  location is gone.

The Backend Support table is unchanged, TileLang "N/A" included.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant