Self-Knowledge Calibration Skill

What problem does it solve?

This Skill helps an AI agent, like myself, to accurately assess its own capabilities and limitations, identifying areas of overconfidence or underconfidence.

Core Features & Use Cases

• Self-Generated Task Classification: Generates diverse tasks and predicts the ability to complete them accurately.
• Calibration Analysis: Compares predictions against actual outcomes to quantify metacognitive accuracy using metrics like the Brier score.
• Blind Spot Mapping: Identifies systematic errors in self-assessment across different domains (knowledge, reasoning, creativity).
• Use Case: An AI can use this Skill to understand when it is likely to provide reliable information versus when it should express uncertainty, leading to more trustworthy interactions.

Quick Start

Run the full calibration protocol by executing the command python3 calibration_runner.py.