harness-engineering

Implement defense-in-depth with three layers: input guardrails (length bounds, injection pattern detection), tool guardrails (risk-tier classification with approval gates for destructive operations), and output guardrails (PII detection, schema validation). Each layer runs independently so a failure in one is caught by another.

Combine retry with exponential backoff for transient API errors, a fallback model chain for degraded operation, circuit breakers on tools with external dependencies, and checkpoint-based state persistence so the agent can resume from its last saved step after a crash.

Yes, the harness patterns are framework-agnostic and work with LangChain, LangGraph, CrewAI, AutoGen, Mistral Agents, OpenAI Assistants, or custom Python agents. The guardrails, checkpointing, and retry logic wrap the agent loop rather than depending on framework internals.

Track input and output tokens per call with a pricing table, enforce per-request token and dollar budgets that raise exceptions when exceeded, and optionally route requests to cheaper models based on complexity classification. Log all usage with run IDs for cost analysis.

Escalate when the agent's confidence falls below a defined threshold (e.g., 0.7) or when it attempts destructive actions like deletes or external sends. For production systems, use an async approval queue so non-destructive work continues while waiting for reviewer decisions.

Skip the full harness for single-shot chat completions with no tools, state, or autonomy, and for early prototypes where a max-rounds cap and basic logging suffice. The harness also does not cover prompt engineering, RAG pipeline design, or model fine-tuning.