add-cuda-kernel

Adding a CUDA kernel to FlashInfer involves an end-to-end workflow: implementing the kernel, creating a host launcher, setting up TVM-FFI bindings, generating JIT modules, exposing a Python API, and registering for ahead-of-time compilation.

TVM-FFI binding exposes CUDA kernels to Python by creating a host launcher in the csrc directory that bridges the C++ implementation with the TVM runtime, enabling direct invocation and testing from Python APIs.

Generating JIT modules in FlashInfer involves building runtime compilation artifacts after defining the CUDA kernel and TVM-FFI bindings, allowing dynamic compilation and seamless Python API exposure for usage.

Yes, you need to write tests and benchmarks when integrating a CUDA kernel to validate correctness and measure performance across various CUDA architectures before achieving production readiness.

Registering FlashInfer modules for ahead-of-time compilation improves startup latency and promotes module reuse, ensuring the integrated CUDA kernel is production-ready and avoids runtime compilation overhead.