What problem does it solve? Moving data from APIs, files, and databases into target systems often produces corrupt, duplicated, or silently lost records when pipelines lack validation gates and staging. This Skill provides a structured extract-validate-transform-stage-load workflow that prevents bad data from reaching production systems. ## Core Features & Use Cases - Multi-Source Extraction: Pull records from JSON/YAML/CSV files, paginated REST APIs via httpx, and databases via asyncpg with batched cursors. - Validation & Deduplication: Apply schema and business-rule validation gates, then deduplicate records with configurable merge strategies (keep first, keep latest, merge fields). - Staging & Orchestration: Stage batches with metadata tracking before promotion, and orchestrate the full pipeline through composable extractor, validator, transformer, and loader components. - Use Case: Importing thousands of records from a third-party API into a database—extract paginated results, validate required fields, deduplicate on ID, stage the batch for inspection, then load with row-count verification. ## Quick Start Ask the AI to build a data ingestion pipeline that extracts records from a paginated REST API, validates them against a schema, deduplicates by ID, and stages the batch before loading into a database.
