Support hosted NCU profiling through the Popcorn API

[msaroufim]

Default CLI profiling should use the same Popcorn identity and hosted submission service as other modes. Add POST /profile/{leaderboard}/{gpu} with validated benchmark and NCU capture options, routed through the existing permissions, rate limits, submission preparation, and backend. Provider credentials stay on KernelBot; users need no Modal installation or account.

The GPU runner selects benchmarks[N] (or all entries), follows evaluator child processes, captures custom_kernel/, and returns .ncu-rep plus text/CSV details and shape/configuration provenance. Failed or empty captures report errors. No provider failover is added. The default CLI uses this route; --profile-brev is a separate explicit command.

Pin Modal's NCU to 2025.2.1: CUDA 13.3's bundled 2026.2 produced many NaN hardware counters in the B200 check. Repeating the same capture with 2025.2.1 restored valid occupancy, throughput, cache, and instruction counters without extra container privileges.

Validation:

• 316 CPU tests passed (pytest -m 'not integration' tests); 14 GPU integration tests deselected. Ruff passed for new profiling code, run_eval, the Modal image, and profiling tests.
• Isolated end-to-end test exercised the real API, submission preparation, KernelBackend, GPU run_config, and release CLI with a fake database and ephemeral B200 launcher. The CLI's PATH was empty and MODAL_* variables absent.
• problems/linalg/qr_v2, benchmark 0 (batch=20, n=32, cond=1, seed=43214), reference-kernels 51e22db671d36c1c76091c43c36a44546ba324a1; filter regex:geqr2, demangled names, launch count 1.
• NCU 2025.2.1: 39 passes, 7,593,257-byte report successfully reopened for both exports; 12.51% achieved occupancy, 0.42% SM throughput, 70.09% L2 hit rate, and 1,565,428 executed instructions. Six ctc__ metrics unavailable. Other problems/GPU types are not covered by this GPU check.

Deploy the API and updated GPU runners before releasing gpu-mode/popcorn-cli#80. New-problem guide: gpu-mode/reference-kernels#172. No production deployment is included in this PR; rollout details are in docs/hosted-ncu-profiling.md.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
src/libkernelbot
backend.py					261
profiling.py					33
submission.py					115-122
utils.py
Project Total

This report was generated by python-coverage-comment-action