skills.rest
pchemguy
pchemguyCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

codex-readiness-integration-test

Runs an end-to-end Codex CLI agentic loop with build/test execution and scored evaluation.

Updated Feb 4, 2026
Search Tags:#code quality#integration test#llm evaluation#codex cli#agentic loop#deterministic checks#build test scoring
One-click install
npx skills add https://github.com/pchemguy/AISandbox --skill codex-readiness-integration-test-pchemguy
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: codex-readiness-integration-test
Source: https://github.com/pchemguy/AISandbox/tree/main/docs/AgentSkills/openai/skills/skills/.experimental/codex-readiness-integration-test
Command: npx skills add https://github.com/pchemguy/AISandbox --skill codex-readiness-integration-test-pchemguy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve? Validating whether an AI coding agent can actually plan, edit code, and verify its work in a real repository is hard to measure; this Skill runs a repeatable multi-stage integration test that executes a real agentic loop and scores the results. ## Core Features & Use Cases - End-to-End Agentic Loop: Executes an approved change prompt through the Codex CLI against the real repo root, producing real filesystem edits and git diffs. - Deterministic and LLM Evaluation: Combines rule-based checks (planning before edits, verification after edits, repo-root-only changes) with LLM evaluators for agentic loop success and change quality. - Evidence and Scoring Reports: Collects evidence.json, execution summaries, and logs, then writes scored reports and a human-readable summary per run. - Use Case: A platform team wants to benchmark Codex CLI behavior on their repository; they run this test with an auto-generated task, review the scored report, and detect regressions like missing verification steps or sandbox-blocked tool access. ## Quick Start Ask the agent to run the Codex Readiness integration test and choose whether to provide a custom task or auto-generate one.

Frequently Asked Questions about codex-readiness-integration-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run the Codex Readiness integration test?▼

First authenticate the Codex CLI with the repo-local HOME using codex login, then invoke the skill and choose between a custom task or an auto-generated one. The prompt must be approved before the agentic loop executes.

What does the Codex readiness test evaluate?▼

It evaluates agentic loop success, planning signals before code changes, verification commands after changes, repo-root-only diffs, and build/test execution status. LLM evaluators also score change quality, correctness, context usage, and risk.

Why does the integration test require a separate Codex login?▼

The test uses a dedicated repo-local HOME at .codex-home for isolation, so existing CLI authentication is not reused. You must run codex login with HOME pointed at that directory before executing the test.

What happens when the agentic loop is sandbox-blocked?▼

The runner writes requires_escalation: true to agentic_summary.json and exits with code 3. Re-run the integration test with escalated permissions so Codex tool access is no longer blocked.

What are the limitations of this integration test?▼

It uses non-default CLI flags and a dedicated HOME, so interactive guardrails and auth config differ from normal use. Auto-generated one-shot prompts do not simulate interactive guidance, and MCP servers are not exercised unless configured.

Related Skills

Explore More Software Engineering Skills→
smriksmrik
1

codex-cli-bridge

Generate AGENTS.md files and wrap Codex CLI commands for execution.

Community
Advanced
GenEducationGenEducation

health

Run type checking, linting, tests, dead code, and shell hygiene checks into a weighted composite score.

Official
Advanced
gdurandvadasgdurandvadas
2

role-security-auditor

Identify OWASP Top 10 vulnerabilities and report severity with remediation guidance.

Community
Advanced
syl2042syl2042

codex-readiness-integration-test

Automate end-to-end Codex readiness testing for agentic loops.

Community
Advanced