senior-data-engineer

Build scalable data pipelines by orchestrating ETL/ELT workflows with Python, Spark, and Airflow DAGs to achieve production-grade, fault-tolerant batch processing and streaming ingestion across modern data architectures.

Orchestrate Airflow DAGs to automate streaming ingestion, batch processing, and data quality checks, applying DataOps best practices for pipeline observability, maintainability, and automated data governance.

Apply data modeling and governance workflows within your data platform by integrating DataOps best practices, ensuring data quality checks, security, and compliance across analytics and ML pipelines.

Yes, you can use Spark and Airflow to build real-time processing and ML pipelines, supporting scalable architectures that handle streaming ingestion, fault tolerance, monitoring, and cost optimization.

Batch processing in ETL workflows can introduce latency compared to streaming ingestion; however, combining it with Airflow orchestration and data quality checks ensures fault tolerance and scalable architectures.