sv-eval

What problem does it solve?

This Skill streamlines the process of evaluating AI models on security-related tasks, providing clear metrics and analysis for performance comparison.

Core Features & Use Cases

• Run Evaluations: Execute reproducible tests for network log anomaly detection (E1) and configuration verification (E2).
• Generate Reports: Create detailed metrics reports to understand model performance and identify areas for improvement.
• Analyze Results: Understand key metrics like Accuracy, ECE, FN%, FP% for E1, and MeanReward, FormatSuccess%, AvgTools for E2.
• Use Case: You want to compare how well GPT-5-mini and Llama-3.1-8b perform on the network log anomaly detection task. You would use this Skill to run evaluations for both models and then generate a comparison report.

Quick Start

Use the sv-eval skill to run an evaluation for the network logs environment using the gpt-5-mini model with 100 examples.