bigquery-etl-core

What problem does it solve?

Navigating a large BigQuery ETL repository and understanding its intricate structure, naming conventions, data privacy rules, and best practices can be overwhelming. This core skill provides foundational knowledge, making it easier to work within the bigquery-etl ecosystem.

Core Features & Use Cases

• Project Structure & Conventions: Provides comprehensive guidance on directory structure, table/dataset naming, and versioning patterns.
• Schema & Description Discovery: Directs users to the most reliable sources for schema and field descriptions (local files, Glean Dictionary, ProbeInfo API, DataHub) with token-efficient strategies.
• Data Privacy & Governance: Outlines Mozilla's strict privacy guidelines, PII considerations, data retention policies, and best practices for data handling.
• BigQuery & Mozilla Specifics: Explains partitioning, clustering, and common Mozilla User-Defined Functions (UDFs) like mozfun.
• Use Case: A new data engineer joins the team and needs to quickly understand the bigquery-etl repository. They use this skill to get a foundational overview of project standards, data sources, and privacy requirements, accelerating their onboarding.

Quick Start

Explain the directory structure and naming conventions for tables and datasets in the bigquery-etl repository.