Showing Apache Software Foundation services in Data Pipeline from 8,000+ services. Search within this scoped route, or loosen the category and company filters below.
Apache Airflow is an open-source platform to programmatically author, schedule, and monitor data pipelines as directed acyclic graphs (DAGs). Originally created at Airbnb in 2014 and donated to the ASF, it is the de facto standard for batch workflow orchestration with a rich ecosystem of providers for cloud services, databases, and SaaS systems.
Apache Spark is an open-source unified analytics engine for large-scale data processing with built-in modules for SQL, streaming, machine learning, and graph processing. Created at UC Berkeley in 2009, Spark replaced Hadoop MapReduce as the dominant batch processing engine and underpins commercial platforms including Databricks, AWS EMR, and Google Dataproc.