skills.rest
agenticgogol
agenticgogolCOMMUNITY
Explore all skills by this author →
GitHub

production-monitor-setup

Generates a production monitoring configuration for deployed agents using tracing tools and calibrated judges.

1|9|Updated Jul 11, 2026
Search Tags:#llm evaluation#observability#tracing#production monitoring#langfuse#langsmith#model drift
One-click install
npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill production-monitor-setup-agenticgogol
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: production-monitor-setup
Source: https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul/tree/main/.claude/skills/production-monitor-setup
Command: npx skills add https://github.com/agenticgogol/Edureka_Coding_Agent_Enabled_Demo_11Jul --skill production-monitor-setup-agenticgogol

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Once an agent is deployed, teams lack a systematic way to trace production traffic, score it with the same judges used in offline evaluation, and catch silent provider-side model drift. This Skill produces a complete eval/monitoring_config.md that closes that gap. ## Core Features & Use Cases - Tracing Tool Selection: Chooses between LangFuse, LangSmith, Arize, or Braintrust based on the agent's stack and scale. - Judge Reuse Policy: Reuses the calibrated judge prompts from eval/judge_prompts/ for production scoring, flagging uncalibrated judges as advisory-only. - Triage and Re-baseline: Defines a human-confirmed triage process feeding new failures into eval/golden_set.jsonl, plus a scheduled re-baseline to detect silent model drift. - Use Case: After deploying a LangGraph-based support agent, run this Skill to generate a monitoring config that samples 10% of production traces, scores them with the calibrated judge, and routes confirmed novel failures back into the golden set. ## Quick Start Set up production monitoring for my deployed agent with tracing, judge-based scoring, and a weekly re-baseline schedule.

Frequently Asked Questions about production-monitor-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor an AI agent in production?▼

Set up tracing with a tool like LangFuse or LangSmith, sample production traffic, and score traces with the same calibrated judge used in offline evaluation. This Skill generates an eval/monitoring_config.md covering tool choice, sampling rate, triage, and re-baselining.

LangFuse vs LangSmith for production tracing?▼

LangFuse is the low-friction default for most stacks, while LangSmith fits agents built on LangChain or LangGraph. Arize or Braintrust suit teams needing enterprise observability or experimentation features beyond basic tracing.

Should I write new judge prompts for production scoring?▼

No. Reuse the same calibrated judge prompts from eval/judge_prompts so production scores stay consistent with offline evaluation. If a judge is not calibrated, its production scores must be flagged as advisory-only.

What sampling rate should I use for production traces?▼

Start with 100% for low-volume agents and 5-10% for high-volume agents, then adjust based on traffic and cost tolerance. The expected volume should be confirmed with the user if unknown.

Why re-baseline an agent after deployment?▼

Providers update models silently server-side even when a model string is pinned, causing silent drift. A recurring re-baseline against production-sampled inputs detects this drift by diffing against the stored baseline.

Can production failures be added to the golden set automatically?▼

No. Sampled low-scoring traces require human review to confirm a real, novel failure before being appended to eval/golden_set.jsonl with category past_failure and source production.

Related Skills

Explore More Software Engineering Skills→
wshobsonwshobson
39.3k

eval-harness-first

Build evaluation harnesses with goldens, graders, and baselines that gate fine-tuning runs.

Community
Advanced
letta-ailetta-ai
3.2k

self-configuration

Inspect and modify Letta Code agent memory, model, permissions, and runtime settings.

Official
Advanced
Nandansai08Nandansai08
1

eval-loop-for-agents

Builds an eval harness that scores agent runs across prompt, model, and tool changes.

Community
Advanced
ThomazRossitoThomazRossito
4

databricks-mlflow-evaluation

Evaluate GenAI agents on Databricks using MLflow 3 scorers, traces, and judge alignment.

Community
Advanced