data-engineer-2

What problem does it solve?

This Skill provides expert guidance and patterns for building robust, scalable data pipelines, modern data warehouses, and real-time streaming architectures, ensuring data reliability and performance.

Core Features & Use Cases

• Pipeline Design: Architect batch and streaming data pipelines using tools like Spark, Airflow, and Kafka.
• Data Warehousing: Implement modern data warehouses and lakehouses with Snowflake, BigQuery, and Databricks.
• Data Quality: Integrate frameworks like Great Expectations and dbt tests for data validation.
• Use Case: You need to design a data pipeline to ingest customer transaction data from Kafka, process it using Spark, store it in a Snowflake data warehouse, and ensure data quality throughout the process.

Quick Start

Use the data-engineer skill to design a scalable data pipeline for processing streaming data from Kafka to BigQuery.