reflect-calibration

What problem does it solve?

This skill addresses the challenge of unreliable confidence scores in AI-generated hypotheses, ensuring more accurate self-assessment and improved decision-making.

Core Features & Use Cases

• Confidence Binning: Groups hypotheses by confidence intervals (e.g., 70-79%).
• Accuracy Calculation: Computes actual accuracy within each confidence bin.
• Self-Consistency Check: Recommends methods for verifying hypothesis accuracy through multiple independent assessments.
• Data Updates: Persists calibration findings to improve future performance.
• Use Case: After generating 50 hypotheses, this skill analyzes how often hypotheses with 90%+ confidence were actually correct, identifying potential over or under-confidence in the AI's self-assessment.

Quick Start

Run the reflect calibration check to analyze hypothesis accuracy.