cuda-kernels

Load the pipeline, move it to CUDA, then monkey-patch target modules by matching type(module).__name__ before enabling CPU offloading. The included ltx_kernel_injection_example.py demonstrates the complete pattern in about 150 lines for LTX-Video.

Use the kernels library's get_kernel function with a repository ID like kernels-community/activation, optionally pinning a version. Check availability first with has_kernel, and pre-allocate output tensors since most kernels require them.

torch.compile cannot trace unregistered custom kernels and raises a dynamo Unsupported error. Register the kernel as a PyTorch custom op using torch.library.custom_op with a fake implementation, or run kernels and compile as mutually exclusive configurations.

Some diffusers RMSNorm modules use elementwise_affine=False, so module.weight is None. Detect this case and substitute a weight tensor of ones matching the input's device and dtype before calling the kernel.

The vectorized RMSNorm kernel achieves a 2.67x average speedup over the PyTorch baseline on H100, reaching 38% of theoretical memory bandwidth. End-to-end video generation speedup is about 6% because RMSNorm is only around 5% of total compute.

Yes, dedicated optimization guides cover A100 (sm_80) and T4 (sm_75) alongside H100 (sm_90). Note that T4 lacks BF16 support and requires FP16, and A100 has lower memory bandwidth and no TMA or FP8 features.