add-reference-tests

Generate and validate pytest tests for flashinfer_trace reference implementations against FlashInfer or SGLang ground truth.

264|47|Updated Jun 30, 2025
One-click install
npx skills add https://github.com/flashinfer-ai/flashinfer-bench --skill add-reference-tests
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: add-reference-tests
Source: https://github.com/flashinfer-ai/flashinfer-bench/tree/main/.claude/skills/add-reference-tests
Command: npx skills add https://github.com/flashinfer-ai/flashinfer-bench --skill add-reference-tests

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates the creation and validation of pytest-based tests that compare flashinfer_trace reference implementations against FlashInfer or SGLang ground truth, helping maintain kernel correctness as new op_types are added or existing ones are updated.

Core Features & Use Cases

  • Automated test scaffolding: generates tests under flashinfer_trace/tests/references/ for new definitions and op_types.
  • Ground-truth validation: ensures reference implementations match the chosen ground truth (FlashInfer by default, with SGLang as fallback when needed).
  • Reproducible testing: provides deterministic inputs and standardized test templates to support consistent results across environments.

Quick Start

/add-reference-tests --op-type mla_paged /add-reference-tests --op-type moe /add-reference-tests --op-type gqa_paged /add-reference-tests --op-type rmsnorm

Frequently Asked Questions about add-reference-tests

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate pytest tests to validate FlashInfer kernel reference implementations?▼

To validate FlashInfer kernel reference implementations, you can generate pytest tests automatically under flashinfer_trace/tests/references/ by specifying the target op_type. The generated tests compare your reference implementations against FlashInfer ground truth.

How does ground truth selection work when validating flashinfer_trace reference implementations?▼

Ground truth validation for flashinfer_trace reference implementations follows a strict priority: FlashInfer is used as the primary ground truth, with SGLang as a fallback when FlashInfer definitions are unavailable.

Can I use this automated test generation for SGLang op_types like mla_paged and moe?▼

Yes, you can validate SGLang op_types like mla_paged, moe, gqa_paged, and rmsnorm. The test scaffolding generates deterministic inputs and standardized templates to ensure reproducible results across environments.

What is the best way to automate reference test scaffolding for new op_types in kernel validation?▼

Automating reference test scaffolding for new op_types involves running a command with the specific op_type argument. This generates standardized pytest files that verify reference definitions against the chosen framework ground truth.

Do I need existing test scaffolding in the repository to generate reference tests for kernel validation?▼

Yes, generating reference tests for kernel validation requires access to the repository's existing test scaffolding. The automated generation relies on this structure to place tests correctly under flashinfer_trace/tests/references/.