benchmark-framework

What problem does it solve?

This skill provides a rigorous, data-driven framework for empirically evaluating and comparing cognitive reasoning patterns, enabling objective selection based on measured trade-offs.

Core Features & Use Cases

• Empirical pattern evaluation: Quantitatively compare ToT, BoT, SRC, HE, AR, DR, AT, RTR, and NDF across defined tasks.
• Experimental design support: Facilitates reproducible, controlled experiments with problem sets, randomization, and baseline controls.
• Results governance: Centralized collection and storage of metrics (quality, cost, time, confidence) for cross-domain analysis and trend tracking.

Quick Start

• Prepare an experiment_config.yaml describing problem domains, patterns, and evaluation criteria, then run the benchmark runner to start the evaluation.
• After completion, review the generated reports and analyze results stored in the Chromadb-backed store for trend insights.