langsmith-evaluator

What problem does it solve?

This Skill streamlines the process of evaluating AI agent performance by providing tools and guidance for creating, defining, and running evaluations within the LangSmith platform.

Core Features & Use Cases

• Create Evaluators: Supports both LLM-as-Judge and custom code evaluators for diverse evaluation needs.
• Define Run Functions: Guides on capturing agent outputs and trajectories for accurate evaluation.
• Run Evaluations: Facilitates local execution with evaluate() and automatic execution via LangSmith uploads.
• Use Case: You've built a customer support chatbot and want to ensure its responses are accurate and helpful. Use this Skill to set up an LLM-as-Judge evaluator that grades responses against expected outcomes and a custom code evaluator to check for adherence to specific response formats.

Quick Start

Use the langsmith-evaluator skill to upload a Python evaluator script named 'evaluators.py' with the function 'accuracy_evaluator' to your LangSmith project named 'My Project'.