testing-llm

What problem does it solve?

This Skill addresses the challenge of reliably testing AI and LLM-generated content, ensuring quality, accuracy, and deterministic behavior in your applications.

Core Features & Use Cases

• LLM Mocking: Create deterministic unit tests by mocking LLM API responses.
• Quality Evaluation: Validate LLM outputs using frameworks like DeepEval and RAGAS for metrics like relevancy, faithfulness, and hallucination detection.
• Structured Output Validation: Ensure LLM responses adhere to predefined schemas using Pydantic.
• Agentic Test Workflows: Implement advanced testing patterns with planner, generator, and healer agents.
• Use Case: When developing a chatbot that relies on an LLM for responses, use this Skill to write tests that verify the chatbot's answers are relevant, factually correct based on provided context, and adhere to a specific JSON structure.

Quick Start

Use the testing-llm skill to validate the quality of an LLM response against a set of DeepEval metrics.