What problem does it solve? Measuring and improving an AI agent's quality in production is hard: teams lack evaluators, continuous monitoring, CI/CD quality gates, and visibility into logs, traces, and costs. This Skill guides developers through setting up evaluation pipelines, observability, and cost optimization for AWS Bedrock AgentCore agents. ## Core Features & Use Cases - Evaluation Setup: Run one-time evals with built-in evaluators (Helpfulness, Correctness, GoalSuccessRate), create custom LLM-as-judge or code-based evaluators, and evaluate local dev traces without deploying. - Continuous Monitoring & CI/CD Gates: Configure online eval configs with sampling rates, pause/resume monitoring, and generate quality gate scripts that fail builds when scores drop below thresholds. - Observability & Cost Optimization: Set up CloudWatch logging, X-Ray tracing, cross-account observability, and reduce costs via model selection, memory strategy tuning, and log retention policies. - Use Case: A developer notices their agent's responses degrading in production. They use this Skill to add an online eval config at 5% sampling, stream eval logs, identify the regression, and add a CI/CD quality gate to prevent future regressions. ## Quick Start Ask the assistant to set up an evaluator and quality monitoring for your AgentCore agent, for example by saying "add a quality gate and online monitoring to my agent".
