benchmark-agents

Runs interactive Claude Code eval sessions to verify Vercel plugin skill injection and hook behavior.

Updated Aug 21, 2025
One-click install
npx skills add https://github.com/Adithiya-S/AI-Study-Companion --skill benchmark-agents-adithiya-s
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark-agents
Source: https://github.com/Adithiya-S/AI-Study-Companion/tree/main/.agents/skills/benchmark-agents
Command: npx skills add https://github.com/Adithiya-S/AI-Study-Companion --skill benchmark-agents-adithiya-s

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Testing a Claude Code plugin's skill injection and hook behavior requires real interactive sessions, since print-mode runs never fire hooks or create files. This Skill provides the exact commands, monitoring queries, and verification checks to run reliable end-to-end evals of the Vercel plugin. ## Core Features & Use Cases - Interactive Eval Launch: Spawns WezTerm panes running real Claude Code sessions with the plugin installed via npx add-plugin, using timestamped project directories. - Injection Monitoring: Checks skill claim directories and debug logs to count injected skills, hook firings, and PostToolUse validation catches. - Code Verification: Greps generated projects for correct patterns such as withWorkflow config, AI Gateway usage, Gemini image models, and AI Elements components. - Use Case: After updating a skill's prompt signals, launch three parallel eval sessions with natural-language prompts, confirm the expected skills were injected, inspect the generated Next.js code, then write a coverage report before releasing. ## Quick Start Launch a benchmark eval by creating a timestamped directory under ~/dev/vercel-plugin-testing, installing the Vercel plugin with npx add-plugin, and spawning a WezTerm pane running an interactive Claude session with a natural-language product prompt.

Frequently Asked Questions about benchmark-agents

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test Claude Code plugin skill injection?▼

Install the plugin with npx add-plugin in a test directory, then spawn an interactive Claude session via wezterm cli spawn with VERCEL_PLUGIN_LOG_LEVEL=debug. Check the session's claim directory in the temp folder to see which skills were injected.

Why doesn't claude --print work for plugin evals?▼

Print mode generates text without executing tools, so PreToolUse, PostToolUse, and UserPromptSubmit hooks never fire. No files are created, no dependencies install, and no session_id exists for claim files or debug logs.

How do I check which skills were injected in a session?▼

List the contents of the claim directory at $TMPDIR/vercel-plugin-<session-id>-seen-skills.d. Each file represents one injected skill, and you can count them or grep for specific skills like workflow.

What prompts trigger the workflow skill in evals?▼

Use natural product language like multi-step pipeline, streams progress, or durable pipeline rather than naming technologies. The plugin's promptSignals match these phrases to inject the workflow skill.

Why must eval directories avoid uppercase letters?▼

npm rejects uppercase letters in package names, so a capital letter in a timestamped directory name breaks create-next-app scaffolding. Use lowercase slugs with date-based suffixes like my-app-20260309-1227.