chip-correctness-testing

Validates that agent chips return correct values from stores across heuristic and LLM routing modes.

1|Updated Jun 25, 2026
One-click install
npx skills add https://github.com/curtismu7/AI-DEMO2 --skill chip-correctness-testing-curtismu7
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: chip-correctness-testing
Source: https://github.com/curtismu7/AI-DEMO2/tree/main/.claude/skills/chip-correctness-testing
Command: npx skills add https://github.com/curtismu7/AI-DEMO2 --skill chip-correctness-testing-curtismu7

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Existing tests only prove the agent pipeline runs (status codes, executed flags) but never prove the answer is right, so a hallucinated balance passes every test. This Skill closes that gap by asserting actual values against ground truth. ## Core Features & Use Cases - Value-level assertions: Assert routing params, tool payloads, and rendered narration against the seed/store or that turn's /api/mcp/tool payload instead of status codes. - Trap detection: Avoids false-green tests caused by the heuristic floor answering chips before the LLM, dual result shapes (banking vs vertical), and per-vertical money key differences like amountDue in healthcare. - Ground truth mapping: Locates each vertical's real data source (seed.json files, GET /api/accounts/my) and its own primaryTool contract in useCases.js. - Use Case: Before a demo, verify that the chip "pay my $300 bill" routes with params.amount === 300 and that the rendered reply matches the tool payload, in both heuristic and LLM-only modes. ## Quick Start Verify that the healthcare chips return real billing values from the seed data and that the LLM actually ran by checking the llm-proxy logs.

Frequently Asked Questions about chip-correctness-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test that an AI agent returns correct data instead of hallucinating?▼

Assert actual values against ground truth rather than status codes or executed flags. Compare the tool payload from that turn's /api/mcp/tool call against the seed/store, and verify every money figure in the rendered reply appears in that payload.

Why do cross-mode chip tests pass trivially in heuristic and LLM modes?▼

The BFF heuristic floor answers matched chip phrases before calling the LLM in every mode, so both modes run the identical heuristic. To test LLM routing, use a phrase that does not match any heuristic and confirm the source becomes llamacpp_fallback or helix.

How do I prove the LLM actually ran during a test?▼

Check the llm-proxy container logs with docker logs --since 2m ai-demo-llm-proxy and count POST /v1/chat/completions entries. A count of zero means the heuristic floor answered and the LLM was never invoked.

Why does my vertical chip test silently assert null?▼

Banking returns result.banking.action while every other vertical returns result.action directly, so reading only the banking shape yields null for vertical chips. Normalize with r.banking?.action ?? r.action before asserting.

Where is the ground truth for banking account balances?▼

Use GET /api/accounts/my as the signed-in user, not runtimeData.json, which holds stale accounts keyed by local numeric id rather than the PingOne sub. Never hard-code balances since transfers legitimately move money.

Why do Playwright tests against the demo stack time out with no requests?▼

The stack serves a mkcert certificate that Playwright's TLS client rejects, so fetches fail silently without ignoreHTTPSErrors. Create the browser context with ignoreHTTPSErrors: true and set E2E_BASE_URL to avoid hitting the default AWS deployment.