skills.rest
r-senchuk
r-senchukCOMMUNITY
Explore all skills by this author →
GitHub

agent-testing

Builds tiered test suites for AI agents with mocked unit tests, integration tests, and eval frameworks.

Updated Mar 29, 2026
Search Tags:#pytest#agent testing#promptfoo#deepeval#tool call verification#llm mocking#eval framework
One-click install
npx skills add https://github.com/r-senchuk/agentskills --skill agent-testing-r-senchuk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-testing
Source: https://github.com/r-senchuk/agentskills/tree/main/.agents/skills/agent-testing
Command: npx skills add https://github.com/r-senchuk/agentskills --skill agent-testing-r-senchuk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pytest, pytest-asyncio, pytest-recording, syrupy, deepeval, and includes references (resource) components.

What problem does it solve? AI agents are hard to test because LLM outputs are non-deterministic and real API calls are expensive. This Skill provides a structured procedure for building a complete agent test suite — from free mocked unit tests through cheap integration tests to nightly quality evals — so regressions in tool calls, routing, and output quality are caught before production. ## Core Features & Use Cases - Three-Tier Test Pyramid: Designs unit (fully mocked, free), integration (real cheap model), and eval (quality metrics) tiers with cost-aware CI separation. - Tool Call Verification: Tests tool schemas, tool selection, and argument correctness without calling the LLM, using mock factories for OpenAI, Anthropic, and Mistral SDKs. - Eval Framework Setup: Configures promptfoo, DeepEval, Braintrust, or LangSmith with working examples for agentic metrics like ToolCorrectness and TaskCompletion. - Use Case: After refactoring an agent's prompt, run pytest -m unit to verify tool selection logic for free, then trigger the nightly eval suite to detect quality regressions against snapshot baselines. ## Quick Start Ask the agent to set up a test suite for your agent project, specifying your framework, LLM provider, and whether you need unit, integration, or eval tiers.

Frequently Asked Questions about agent-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test an AI agent without calling the LLM API?▼

Mock the SDK client with unittest.mock AsyncMock factories that return canned responses, or use a FakeModel class that queues deterministic outputs popped FIFO on each call. This gives fully deterministic unit tests with zero LLM traffic and zero cost.

promptfoo vs DeepEval for agent evaluation?▼

promptfoo is YAML-first, CI-friendly, and supports multi-provider comparison with tool-call assertions out of the box. DeepEval is Python-native with pytest integration and richer agentic metrics like ToolCorrectness, TaskCompletion, and RAG faithfulness.

How do I verify an agent calls the right tool with correct arguments?▼

Return a mocked tool_call response from the mock client, run the agent step, then assert on the tool name and parse the JSON arguments. Also validate tool schemas in isolation with jsonschema and test tool execution directly without the LLM.

Does this testing approach work with LangChain and CrewAI agents?▼

Yes, the procedure is framework-agnostic and works with OpenAI Agents SDK, LangChain, CrewAI, Mistral SDK, or custom loops. Mock patterns are provided for OpenAI, Anthropic, and Mistral SDKs, plus HTTP-level interception for frameworks that do not support client injection.

How do I keep LLM testing costs low in CI?▼

Separate tests into markers: run fully mocked unit tests on every commit for free, integration tests with a cheap model like gpt-4.1-mini only on pull requests, and expensive eval suites nightly. VCR.py cassettes record real API responses once and replay them for free.

When should I not use this agent testing approach?▼

Do not use it for testing non-agent applications like standard web apps or CLIs, where normal pytest patterns suffice. It also does not cover production monitoring, observability, or load testing of LLM APIs, which are separate domains.

Related Skills

Explore More Software Engineering Skills→
yaalalabsyaalalabs
113

ak-build

Add tools, agents, and handoffs to Agent Kernel projects.

Official
Intermediate
chiruu12chiruu12
1

canary

Monitor production apps for errors and regressions after deployment.

Community
Advanced
JetBrainsJetBrains
302

observability-integrator

Instrument Kotlin and Spring services with metrics, logs, and traces.

Official
Advanced
tankpkgtankpkg
1

@tank/llm-observability

Diagnose LLM observability and evaluation gaps across production tracing and prompt workflows.

Official
Advanced