What problem does it solve? Testing LLM agents is hard because model output is non-deterministic, real API calls are slow and expensive, and tuning prompts or sampling parameters by hand leads to overfitting and wasted tokens. This Skill defines an objective, gate-based testing and optimization strategy for Agno v2 agents so quality becomes measurable and improvable. ## Core Features & Use Cases - Three-layer test strategy: deterministic transport e2e with fake LLM adapters for PR gates, behavior evals with real models in nightly pipelines, and live smoke tests against deployed API Gateway/Lambda targets. - Eval harness with golden sets: versioned JSONL datasets, deterministic scorers plus LLM-as-judge signals, accuracy and out-of-scope recall thresholds, and structured-log assertions for p95 latency, token budgets, and fallback rates. - Systematic optimization: Optuna (TPE, pruning, Pareto multi-objective) for sampling/RAG/orchestration parameters, DSPy for prompt and few-shot compilation, promptfoo for model comparison matrices, and Ragas for RAG faithfulness, all guarded by train/validation/holdout splits and trial budgets. - Use Case: Before changing a support agent's model or prompt, run the golden set through the eval harness, optimize temperature and RAG top_k with Optuna against the train split, then validate once on the holdout before promoting the change. ## Quick Start Ask the agent to design a pytest test suite with fake LLM adapters, a golden-set eval, and an Optuna optimization loop for your Agno agent project.
