eval-engine

What problem does it solve?

This Skill automates the process of building and running evaluation pipelines for Large Language Models (LLMs), enabling systematic quality assessment and comparison.

Core Features & Use Cases

• Dataset Management: Load and manage datasets for evaluation.
• LLM-as-Judge Evaluation: Utilize LLMs to evaluate responses based on various criteria.
• Customizable Metrics: Supports accuracy, relevance, hallucination, harmfulness, and Korean-specific quality.
• A/B Testing: Facilitates comparison between different model runs or configurations.
• Use Case: Evaluate a new chatbot's responses against a benchmark dataset using multiple metrics, identifying areas for improvement.

Quick Start

Use the eval-engine skill to run an evaluation on the dataset named 'dataset_abc' using accuracy and hallucination evaluators with the Codex-haiku judge model.