metadata-manager

What problem does it solve?

Managing BigQuery table metadata, schemas, and Airflow DAG configurations manually is complex, time-consuming, and prone to inconsistencies. This skill streamlines the entire process, ensuring data governance and operational efficiency.

Core Features & Use Cases

• Intelligent Schema & Metadata Generation: Automatically creates schema.yaml and metadata.yaml files with intelligent descriptions, leveraging various data sources (local files, Glean Dictionary, DataHub).
• DAG Configuration Management: Helps find existing Airflow DAGs or creates new ones, inferring optimal scheduling, ownership, and impact tiers.
• Test Coordination: Coordinates updates to unit tests (invoking sql-test-generator) when BigQuery queries are modified, ensuring tests remain in sync with query logic.
• Description Improvement: Proactively detects missing or vague descriptions in source tables and offers to improve them, enhancing metadata completeness across the repository.
• Use Case: A data team needs to create a new derived BigQuery table. Instead of manually drafting YAML files for schema, metadata, and Airflow DAGs, they use this skill to generate all configurations, auto-populating descriptions and ensuring compliance with bigquery-etl standards.

Quick Start

Generate schema.yaml and metadata.yaml for the new BigQuery table my-project.my-dataset.new_table_v1. Infer the best Airflow DAG based on existing patterns.