Airflow Insights

What is Airflow?

Apache Airflow is a platform to programmatically author, schedule and monitor workflows. Airflow is commonly used to create and manage complex data pipelines, automating tasks like data extraction, transformation, and loading (ETL).

Find organizations using Airflow on Sumble →

What other technologies are related to Airflow?

Airflow Competitor Technologies

Luigi

Luigi is a Python-based workflow management tool that competes with Airflow in scheduling and executing complex pipelines.

mentioned alongside Airflow in 96% (4.6k) of relevant job posts

Dagster

Dagster is a data orchestration platform that competes with Airflow in defining and executing data workflows.

mentioned alongside Airflow in 74% (4.1k) of relevant job posts

Prefect

Prefect is a workflow orchestration platform that competes with Airflow in managing and monitoring data pipelines.

mentioned alongside Airflow in 73% (4.1k) of relevant job posts

Azkaban

Azkaban is a batch workflow job scheduler created at LinkedIn, it is a competitor of Airflow for the execution of ETL and data pipelines.

mentioned alongside Airflow in 89% (1.6k) of relevant job posts

GCP Cloud Composer

GCP Cloud Composer is a managed Airflow service that can be considered both a complement (as it's Airflow) and a competitor (as an alternative deployment strategy).

mentioned alongside Airflow in 41% (2.5k) of relevant job posts

Airflow Complementary Technologies

Spark

Spark is a distributed processing engine often used for data transformation tasks within Airflow DAGs.

mentioned alongside Airflow in 16% (51.3k) of relevant job posts

DBT

DBT is a transformation tool that can be orchestrated by Airflow for managing data pipelines in the data warehouse.

mentioned alongside Airflow in 35% (22.4k) of relevant job posts

Snowflake

Snowflake is a cloud-based data warehouse that can be a target for Airflow-orchestrated data pipelines.

mentioned alongside Airflow in 14% (31.5k) of relevant job posts

Which job functions mention Airflow?

Job function

Jobs mentioning Airflow

Orgs mentioning Airflow

Engineering & R&D

Engineer

Software Engineer

Data Engineer

18% (50.8k) job posts

Analytics Engineer

See more or filter by date, location, industry, etc →

6% of Data Infrastructure Migration projects involve Airflow

Microsoft

Principal Software Engineer

To develop a scalable cloud data warehouse that powers SQL analytics in Azure compute, utilizing technologies such as Azure Data Lake and Microsoft Fabric.

Read more ↗

Siemens

IT Solution Expert for SAP HCM Integration (m/f/d)

To integrate Workday with multiple SAP HCM systems using integration platforms like ShapeIn to streamline HR processes.

Read more ↗

Amazon

Applied Scientist, Q Developer Code Agent Science

Developing services that leverage generative AI and foundation models to improve the experience of developers using AWS tools and services.

Read more ↗

Walmart

(USA) Senior Manager, Automation Engineering - Data Engineer

To develop scalable data architectures and real-time data pipelines using technologies like Apache Airflow, Google Cloud Platform, and BigQuery to support AI/ML analytics.

Read more ↗