gcp-managed-airflow-recommendations

Run the provided Python scripts with your project ID, location, environment name, and query duration in hours. They fetch CPU, memory, disk, restart, and DAG parsing metrics from Cloud Monitoring via PromQL, giving you data to identify saturation or instability.

Use the workload_restarts.py script to get restart counts per component, then check workload_memory_usage.py for memory saturation indicating OOM kills. High scheduler restarts often point to memory limits or unoptimized DAGs blocking the event loop.

Cloud Composer publishes metrics under composer.googleapis.com including environment health, database health, webserver health, workload CPU/memory/disk usage, restart counts, and DAG processing stats like total parse time and parse error counts.

Yes, the scripts query the cloud_composer_environment and cloud_composer_workload monitored resources used by managed Airflow environments. Recommendations should distinguish between generations since their architectures and scaling options differ.

Slow parsing usually comes from heavy top-level code in DAG files, such as database connections or API calls outside task definitions. Run dag_parsing_stats.py to measure total parse time and processor timeouts, then move heavy logic into operators.

The scripts authenticate via gcloud application-default credentials and call the Cloud Monitoring PromQL query API. Your account needs permission to read monitoring time series data in the target GCP project.