docs: rewrite README and add the developer guide - #314
Open
bowencui123 wants to merge 17 commits into
Open
bowencui123 wants to merge 17 commits into
bowencui123 wants to merge 17 commits into
Conversation
This was referenced Sep 19, 2026
bowencui123
force-pushed
the
bowen/docs/readme
branch
3 times, most recently
from
September 19, 2026 09:41
985957e to
cdcedaa
Compare
- New tagline (accelerator framework; cuTile on CUDA 13.2, Triton, TileLang, AWS Neuron NKI, PyTorch); NKI added to the backend list. - Figures under assets/: the paper's overview diagram and a new sweep-max latency chart (scripts/plot_sweep_max.py; PyTorch / Triton / cuTile at the NCU-catalogue case of each operator). - Software Versions section with the stack the results were measured on. - OPERATOR_AUTHORING_GUIDE.md merged in as a section; the duplicated Operator Config / Autotune / Adding a New Operator sections are dropped. - Stale content brought up to date: project structure, 2x-L2 eviction size, CLI flags (--autotune, --tile-language, --gpu), Triton and cuTile examples (best_config / CutileAutotuner, no global state), peaks read from data/peak_performance/<GPU>.json, FP8 status re-verified on this stack. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA (cherry picked from commit 24e28f5)
…skills/
- assets/overview.png: corrected figure ("Verify").
- Software Versions: the TileBench paper was measured with cuda-tile 1.3.0;
the TileBench++ work (TileLang, NKI) uses 1.5.0.
- Document the automatic raw-log archive, --no-archive, and skills/.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
(cherry picked from commit 483bc3f)
Lead with the paper's headline results (pairwise win/loss matrix, suite geomeans, autotune gains) and the sweep-max latency figure, then the benchmark overview, backend scope, paper vs. development environments, quick start, evaluation methodology, the LLM track, and a collapsible developer reference (CLI tables and the operator authoring guide). Checked against the paper sources and the recorded CSVs: the win/loss matrix (37/8, 36/9, 33/12; 14 operators within 5%) recomputes exactly from results/csv, every operator sweeps 20 cases per dtype, and the section, appendix and table references resolve. The quick-start install no longer lists cuda-tile-experimental: it is not on PyPI and nothing in the repository imports it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA (cherry picked from commit 2a85c11)
(cherry picked from commit 14198b3)
(cherry picked from commit 29ca688)
(cherry picked from commit 2a807b9)
(cherry picked from commit 4c9cb6c)
Removed extra blank lines in README.md (cherry picked from commit ddad492)
Separate the three things that used to read as one: the version-controlled pipeline under tilebench/llm_codegen/, the git-ignored local output under tilebench/benchmarks/llm_generated/, and the paper's frozen snapshot restored with fetch_artifacts.py. The download link is marked as not yet published. Recorded Results now lists where each kind of generated artifact is kept, and the developer guide documents archive_artifacts.sh and the publishing flow. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Add a resource row under the description, in the spirit of KernelBench's arXiv | blog | dataset line: NCU reports on Hugging Face, the LLM artifact on Google Drive, the raw-log archive branch and the developer guide, each led by its service's icon. The same icons mark those links where they appear in the body and in the Recorded Results table, which gains a row for the Nsight Compute reports. The 220 reports (45 operators, Triton and cuTile, every profiled dtype, 11.8 GB) are already public at huggingface.co/datasets/bcui2/NCU_report; the README now says so, and that downloading needs a free Hugging Face login because the dataset is gated with automatic approval. Icons are vendored under assets/icons/ rather than hot-linked: the Hugging Face logo from its brand-assets repository (viewBox cropped to the glyph), the Google Drive product icon from Wikimedia Commons, the GitHub mark from Simple Icons and the book glyph from Octicons, the last two in a neutral grey that stays visible in GitHub's dark theme. Each icon sits inside its link's text, so GitHub does not wrap it in a link to the image file. Also fixes the tagline, which followed </div> without a blank line and was therefore rendered with its literal ** markers. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
- Put the TileBench logo at the top as the page title. The supplied artwork is cropped to its content (top and bottom margins removed). The light variant has the page background removed; the dark variant keeps the original cream card so the black wordmark stays readable. - Replace the overview figure with the transparent-background version. It is pixel-identical to the previous one when flattened on white. - Keep icons in the top resource bar and the Recorded Results table only; inline body links are plain Markdown. Section anchors are unchanged. - NCU wording: "The released artifact contains 220 raw Nsight Compute reports." - State that tilebench/benchmarks/llm_generated/ is the Git-ignored runtime output path, that a fresh clone does not ship the 6140 campaign files, and that fetch_artifacts.py restores them automatically. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Updated the overview section to provide a clearer description of TileBench and its supported backends.
Emoji prefix on the 14 `##` headings, in the style of the SWE-bench README. GitHub drops the emoji from the heading slug but keeps the following space, so the 12 table-of-contents links move from `#name` to `#-name`. No other content changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
Follows the results layout change on the infra branch. Every command in the
README and the developer guide passes --gpu B200, and the result paths read
results/<hardware>/{csv,logs,figures,aggregate,runs}/. The text states that
this is a general layout, that results/B200/csv/ holds the paper results
committed today, and that other hardware sits beside it without a code
change. The developer guide documents the label rules, the one-namespace-
one-hardware rule, TileLang merge scope, NKI output under outputs/nki/, and
the per-GPU archive command. Headline B200 numbers are unchanged.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
…data Follows the correction on the infra branch. The previous text said NKI was kept out of the GPU namespaces, under outputs/nki/. NKI keeps its original organisation instead: its columns (torch_nki_ms, nki_ms, speedup_nki) are merged into results/B200/csv/, and its logs and profiles live under results/B200/logs/. The README and the developer guide now say that these are measurements taken on AWS Trainium, stored beside the B200 results for a unified per-operator record, that speedup_nki = torch_nki_ms / nki_ms and never torch_ms / nki_ms, and that --gpu names the campaign rather than the device NKI ran on. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
…chive logs
Follows the multi-hardware consistency work on the infra branch.
- Profiling metadata is documented as
tilebench/profiling/metadata/<hardware>/{ncu_catalogue,kernel_counts}.json,
with B200 as the directory committed today and other hardware beside it.
NCU reports are written to outputs/ncu/<hardware>/. The 220 released
reports are described as the paper's B200 profiles; the count is unchanged.
- Raw JSON names carry the mode and the backend selection. The README shows
the two files its quick start produces and passes --tile-language to
visualize.py, which now names the run it reads. The developer guide adds
the naming rule, --mode, the ncu_catalogue log contract and the --gpu
commands of the NCU tools.
- The paper's B200 raw logs are stated to live under results/B200/logs/ on
the archive branch. The sentence describing results/logs/ as their current
location is gone.
The Backend Support table is unchanged, TileLang "N/A" included.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA
bowencui123
force-pushed
the
bowen/docs/readme
branch
from
September 19, 2026 10:36
cdcedaa to
d211e23
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The documentation half of PR #312. Stacked on #313, so the diff below is only the 11 documentation and asset files; retarget the base to
mainonce #313 lands.What this does
docs/developer_guide.mdadded: the CLI reference plus the formerOPERATOR_AUTHORING_GUIDE.md, which is removed from the repository root.assets/: the TileBench logo as the page title (cropped to its content; a transparent variant for light mode and the original cream card for dark mode, so the black wordmark stays readable), the overview diagram with a transparent background, the sweep-max latency figure, and four link icons. Link icons appear only in the top resource bar and the Recorded Results table; inline body links are plain Markdown. Each##section heading carries an emoji, in the style of the SWE-bench README, and the table-of-contents links use the matching#-nameanchors.main: the 220 raw Nsight Compute reports on Hugging Face (free login required), the raw logs onarchive/raw-logs-2026-09-18, and the LLM campaign on Google Drive. It states thattilebench/benchmarks/llm_generated/is the Git-ignored runtime output path, that a fresh clone does not ship the 6,140 campaign files, and thatpython scripts/fetch_artifacts.py --artifact llm-aacl2026restores them.--gpu B200, and result paths readresults/<hardware>/{csv,logs,figures,aggregate,runs}/. The text says that this is a general layout, thatresults/B200/csv/holds the paper results committed today, and that other hardware sits beside it without a code change. The developer guide documents the label rules, what a namespace holds, the scope of the TileLang merge, and the per-GPU archive command. Both documents state that NKI results are stored with the B200 campaign as cross-hardware measurements taken on AWS Trainium: three columns inresults/B200/csv/, logs and profiles underresults/B200/logs/, withspeedup_nki = torch_nki_ms / nki_msand nevertorch_ms / nki_ms. The headline B200 numbers are unchanged, and no empty GH200 or AMD directory is created.tilebench/profiling/metadata/<hardware>/{ncu_catalogue,kernel_counts}.json, with B200 as the directory committed today, and NCU reports asoutputs/ncu/<hardware>/. The 220 released reports are described as the paper's B200 profiles; the count is unchanged. Raw JSON names carry the mode and the backend selection: the README shows the two files its quick start produces and passes--tile-languagetovisualize.py, and the developer guide adds the naming rule,--mode, thencu_cataloguelog contract and the--gpucommands of the NCU tools. The paper's B200 raw logs are stated to live underresults/B200/logs/on the archive branch; the sentence that describedresults/logs/as their current location is gone.Verification
results/B200/csv/; the geometric means (2.02×, 1.58×, 1.18×, 1.22×), the suite composition (26 from TritonBench, 19 from LeetGPU) and the category counts match the paper sources.--help; the commands no longer needPYTHONPATH=and were run as written. The flags added for the results layout (--gpuon four scripts,--outputonplot_sweep_max.py,--results-root) were checked against--helpas well, and the README quick-start form ofrun_bench.pywas run for real on a B200 under a throwaway label.pyteston this branch: 183 passed, 1 skipped.Two things to decide before merge
results/B200/csv/carrytilelang_msandspeedup_tilelang(suite geometric mean 1.72× over PyTorch); they are not part of the headline comparison.Separately, the repository still has no LICENSE file while the README tells people to
pip install -e ..🤖 Generated with Claude Code
https://claude.ai/code/session_01EdNBRsQvET5PUJ3Nw1hVCA