data-pipeline-operations
OfficialGuide data pipelines Bronze to Gold.
Data & Analytics#duckdb#data-pipeline#gcs#medallion-architecture#bronze-silver-gold#datavalidation#cvr-chr-bfe
AuthorKlimabevaegelsen
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Data engineers often struggle to design, monitor, and debug Python data pipelines and GCS workflows across medallion architecture (Bronze/Silver/Gold). This guide provides a structured approach to building, validating, and optimizing end-to-end data pipelines with consistent data quality checks and identifier handling (CVR/CHR/BFE).
Core Features & Use Cases
- Guidance for implementing Bronze/Silver/Gold layers, including transformations, deduplication, and CRS handling.
- Instructions for GCS uploads/downloads and large-file analytics with DuckDB.
- Use Case: When preparing a dataset of Danish farm data, follow steps to validate CVR/CHR formats and join sources for analysis.
Quick Start
Explain how to run an end-to-end data pipeline across Bronze to Gold with GCS operations and DuckDB-backed validation.
Dependency Matrix
Required Modules
None requiredComponents
Standard package💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: data-pipeline-operations Download link: https://github.com/Klimabevaegelsen/landbruget.dk/archive/main.zip#data-pipeline-operations Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 223,000+ vetted skills library on demand.