LangSmith Evaluators

What problem does it solve?

This Skill streamlines the process of building, defining, and running evaluations for your AI applications, ensuring quality and performance.

Core Features & Use Cases

• Create Evaluators: Define LLM-as-Judge or custom code evaluators to assess AI outputs.
• Define Run Functions: Capture agent outputs and trajectories for detailed analysis.
• Run Evaluations: Execute evaluations locally or automatically via LangSmith.
• Use Case: You've built a customer support chatbot and want to ensure its responses are accurate and helpful. Use this Skill to define an LLM-as-Judge evaluator that scores responses against expected answers and a custom code evaluator that checks for adherence to specific response formats.

Quick Start

Use the LangSmith Evaluators skill to upload your Python evaluator script 'my_evaluators.py' to the dataset 'My Dataset'.