skills.rest
sixteen-dev
sixteen-devOFFICIAL
Explore all skills by this author →
Website𝕏@sixteen_devGitHub

obai-e2e-regression

Runs cost-aware black-box regression tests against the OBaI multi-agent platform.

11|6|Updated Mar 18, 2026
Search Tags:#evaluation#multi-agent#regression testing#e2e testing#mcp servers#html report#opik tracing
One-click install
npx skills add https://github.com/sixteen-dev/obai --skill obai-e2e-regression-sixteen-dev
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: obai-e2e-regression
Source: https://github.com/sixteen-dev/obai/tree/main/.agents/skills/obai-e2e-regression
Command: npx skills add https://github.com/sixteen-dev/obai --skill obai-e2e-regression-sixteen-dev

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, and includes scripts (resource) components.

What problem does it solve? Validating changes to prompts, routing, or specialist agents in a multi-agent system is expensive and error-prone; this Skill gates those changes behind a deduplicated, cost-controlled end-to-end regression suite so regressions are caught before merge without wasting billable model requests. ## Core Features & Use Cases - Cost-aware regression gating: Runs a deduplicated core tier of test cases by default, reserving the live canary and broader evaluation corpus for explicit requests because they consume additional billable model requests. - Pre-flight readiness checks: Verifies OPENAI_API_KEY is set, the local Opik tracing server is reachable, and all 9 OBaI MCP servers report healthy before any paid run starts. - Trace inspection and HTML reporting: Fetches curated Opik trace views (skill loads, specialist calls, verdicts) and renders results.json plus report.md into a structured HTML report with verdict distribution and per-case detail. - Use Case: After modifying the crypto specialist's routing prompt, ask for an OBaI E2E regression run to execute the canonical case set (product resolution, quotes, order books, OHLCV, backtests, artifact export, and fail-closed boundary checks) and review the rendered HTML report. ## Quick Start Ask the assistant to run the OBaI E2E regression core tier to validate your recent prompt or routing changes before merging.

Frequently Asked Questions about obai-e2e-regression

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run an end-to-end regression test for a multi-agent AI system?▼

Request an OBaI E2E regression run explicitly; the skill delegates to the canonical suite runner which executes the deduplicated core tier of YAML-defined cases against the live hub. Preflight checks confirm the API key, Opik server, and MCP servers are ready before any billable requests are made.

How do I inspect Opik traces from an agent evaluation run?▼

Use the inspect_trace.py script with a trace ID to print a curated view of the user query, load_skill calls, specialist analysis calls with operators and verdicts, and the hub's final output. It defaults to the obai-eval project at http://localhost:5173.

When should I run the full regression suite versus the core tier?▼

Default to the deduplicated core tier for routine merge validation. Run the live canary or broader evaluation corpus only when explicitly needed, because they consume additional billable model requests beyond the core set.

Why does the regression preflight check fail?▼

Preflight fails when OPENAI_API_KEY is unset, the local Opik server is unreachable at http://localhost:5173, or obai status reports any of the 9 MCP servers unhealthy. Fix the reported check and re-run preflight before executing paid cases.

Can I run individual regression cases directly with run_one.py?▼

No, the legacy direct case runner is intentionally disabled. Paid cases must go through the canonical suite runner so they stay bound to the run manifest, preflight, attempt ledger, and between-case request limits.

Related Skills

Explore More Software Engineering Skills→
growthxaigrowthxai
430

output-services-check

Verify Docker containers, API server, and Temporal UI health during local development.

Official
Intermediate
mattbutlerengineeringmattbutlerengineering
1

rialto-visual-diff-review

Classify Playwright visual diff failures as intentional, regression, or stale.

Community
Intermediate
lespacemanlespaceman
4

evaluate-task

Evaluate completed WebBench tasks as PASS, PARTIAL, or FAIL using LLM-based scoring on traces and extracted data.

Community
Advanced
sethjuarezsethjuarez

castia-lifecycle

Build, evaluate, and optimize Foundry hosted agents using the castia Python SDK.

Community
Advanced