agency-ai-data-remediation-engineer

What problem does it solve? When data pipelines produce thousands of broken rows, fixing them one by one is impossible and trusting an LLM to rewrite data directly is dangerous. This Skill compresses massive anomaly sets into a handful of semantic clusters, generates deterministic fix logic with air-gapped local models, and guarantees zero silent data loss. ## Core Features & Use Cases - Semantic Anomaly Compression: Embeds anomalous rows with sentence-transformers and clusters them via ChromaDB or FAISS, turning 50,000 errors into a dozen pattern families. - Air-Gapped Fix Generation: Uses Ollama-hosted models (Phi-3, Llama-3, Mistral) to output strictly validated Python lambdas — no cloud API ever touches PII. - Zero-Data-Loss Reconciliation: Enforces Source_Rows == Success_Rows + Quarantine_Rows on every batch, with full audit logs of old value, new value, lambda, confidence, and model version. - Use Case: A pipeline flags 50,000 rows with inconsistent date formats. The Skill clusters them into 12 patterns, generates 12 validated lambdas locally, applies them vectorized, and routes low-confidence rows to a human quarantine dashboard. ## Quick Start Ask the agent to analyze the anomalous rows tagged NEEDS_AI from your pipeline, cluster them semantically, and generate safe fix lambdas using a local Ollama model.