cuda-kernel-refine_skill

Overview

This skill implements an iterative CUDA kernel refinement loop focused on improving throughput, reducing bandwidth, and lowering latency using NVIDIA tooling (nsys, ncu, ptxas, SASS). It guides you from establishing a reproducible benchmark baseline through targeted profiling, bottleneck classification, single-change optimization cycles, correctness checks, and quantitative comparison. The process enforces one change per cycle so improvements are attributable and regressions are reversible.

How this skill works

Start by recording a stable baseline benchmark with controlled workload parameters and iterations. Capture an nsys trace for kernel timing and API/transfer patterns, export to SQLite for targeted queries, then drill into register pressure with ptxas/cubin output and micro-architecture counters with ncu. Classify kernels as memory-, compute-, or latency-bound, apply focused optimizations (fusion, vectorized loads, tensor cores, fast-math, shared memory reuse, ILP), verify correctness with compute-sanitizer and tests, and re-run the same benchmark to compare results. Repeat the loop with a single, isolated change each iteration.

When to use it

• You need measurable kernel speedups or bandwidth reduction
• High launch count or many small kernels per step
• Profiling shows high memory traffic or low SM utilization
• You suspect register spills, occupancy limits, or launch overhead
• Comparing alternative implementations or validating fusion impact

Best practices

• Make a reproducible baseline: same workload, iterations, and GPU state for all comparisons
• One change per cycle and revert on regression to isolate cause/effect
• Use nsys→SQLite for rapid hotspot and launch/grid inspection before deep dives
• Use ptxas/cubin to check register usage and spills; lower registers to raise occupancy only when safe
• Prefer fusion and elimination of D2H syncs for latency-bound cases; prefer vectorized loads and reduced precision for memory-bound kernels
• Run compute-sanitizer and project tests after every functional change before benchmarking

Example use cases

• Fuse elementwise postprocessing (residual_add + norm) across layers to cut launches and bandwidth
• Use ncu SpeedOfLight to decide if a custom matmul should target tensor cores or wider loads
• Diagnose and eliminate per-step cudaMallocs shown by nsys API summary to reduce allocator overhead
• Convert FP32 weights to FP8/FP4 and validate accuracy tradeoffs while measuring bandwidth improvement
• Investigate a single slow kernel with cuobjdump SASS to verify vectorized LDG.E.128 loads and absence of register spills