sc-evaluate

What problem does it solve?

This Skill automates the process of evaluating the quality of LLM-generated outputs against a set of predefined standards, identifying weaknesses and suggesting improvements.

Core Features & Use Cases

• Automated Evaluation: Runs LLM outputs against gold standard datasets and scores them using an LLM-as-judge approach.
• Performance Analysis: Identifies specific steps or cases where the LLM performs poorly.
• Actionable Recommendations: Provides concrete suggestions for improving prompts based on evaluation results.
• Use Case: A team developing a customer service chatbot can use this Skill to rigorously test new prompt variations, ensuring consistent quality and identifying areas for refinement before deployment.

Quick Start

Run a full evaluation of all test cases and pipeline steps using the default judge model.