conformance-loop

Compares Fallow analysis output against competing tools and verified source truth to improve accuracy.

4.4k|154|Updated Mar 17, 2026
One-click install
npx skills add https://github.com/fallow-rs/fallow --skill conformance-loop
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: conformance-loop
Source: https://github.com/fallow-rs/fallow/tree/main/.agents/skills/conformance-loop
Command: npx skills add https://github.com/fallow-rs/fallow --skill conformance-loop

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Static analysis tools can drift from ground truth, producing false positives and false negatives that erode trust. This Skill provides a disciplined loop for measuring and improving Fallow's analysis accuracy against competing tools and manually verified source code.

Core Features & Use Cases

  • Comparative Benchmarking: Run Fallow and competing analyzers with documented equivalent settings across a stable real-world project corpus.
  • Disagreement Classification: Manually verify divergent findings against source and classify each as true positive, false positive, false negative, or model difference.
  • Regression-Guarded Fixes: Implement one general correction at a time with a regression fixture, re-run the full corpus, and retain only net improvements.
  • Use Case: When Fallow reports unused exports differently than another dead-code tool on a monorepo, use this loop to verify which tool is correct, fix the underlying analysis gap, and lock in the improvement with a fixture.

Quick Start

Run the conformance loop to compare Fallow's dead-code findings against competing tools on a stable corpus and classify every disagreement.

Frequently Asked Questions about conformance-loop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I verify static analysis findings against ground truth?▼

Run the analyzer and competing tools with documented equivalent settings on a stable corpus, then manually inspect each disagreement against the source code. Classify every divergence as a true positive, false positive, false negative, or model difference before changing any logic.

How to compare dead-code detection tools on a real codebase?▼

Select a stable set of real-world projects, run each tool with equivalent configurations, and diff the reported findings. Treat competitor output as a lead rather than ground truth, and confirm disputed findings by reading the actual source.

Can competitor tool output be trusted as ground truth?▼

No. Competitor output is a lead, not ground truth. Each disagreement must be manually verified against the source code, since differences may reflect model differences rather than actual errors in either tool.

When should a fix for a false positive be kept or reverted?▼

Implement one general correction with a regression fixture, then re-run the full corpus. Retain the change only if it produces a net improvement across the corpus; revert fixes that trade one false positive for new false negatives.

What are the limitations of corpus-based conformance testing?▼

Results depend on corpus representativeness and equivalent tool configuration, which can be hard to guarantee. Manual verification also scales poorly, so the loop works best on stable, well-understood project sets.