Skip to content

Entry

Bruin

Appears in 6 awesome lists

Run and schedule dbt-style SQL transformations without Airflow. Adds data ingestion (50+ sources) and built-in data quality to the transformation layer. Open-source CLI or managed Bruin Cloud for teams who want dbt Cloud-like experience with ingestion included.

Open github.combruin-data/bruin

Found in these lists

Awesome Apache Airflow

Section: Libraries, Hooks, Utilities · Data ingestion and transformation layer with SQL and Python support. Ingests from 50+ sources (like Airbyte/Fivetran) and transforms (like dbt). Can be triggered from Airflow DAGs or run standalone. Includes built-in data quality checks.

FreshScore 81

Awesome Big Data

Section: Data Ingestion · end-to-end data pipeline tool combining ingestion, transformations, and data quality checks.

FreshScore 90

Awesome Data Engineering

Section: Workflow · End-to-end data pipeline tool that combines ingestion, transformation (SQL + Python), and data quality in a single CLI. Connects to BigQuery, Snowflake, PostgreSQL, Redshift, and more. Includes VS Code extension with live previews.

FreshScore 93

Awesome dbt

Section: Orchestration & Ingestion · Run and schedule dbt-style SQL transformations without Airflow. Adds data ingestion (50+ sources) and built-in data quality to the transformation layer. Open-source CLI or managed Bruin Cloud for teams who want dbt Cloud-like experience with ingestion included.

FreshScore 83

Awesome DuckDB

Section: Integrations · Data pipeline CLI that runs SQL and Python transformations with built-in quality checks, using DuckDB as one of its supported platforms.

FreshScore 91

Awesome Pipeline

Section: Extract, transform, load (ETL) · Data pipeline framework supporting SQL and Python in the same DAG. Built-in data quality assertions, cross-database lineage, and incremental processing. Targets data warehouses (BigQuery, Snowflake, Postgres, etc.).

FreshScore 82

Luigi

Python module for building complex pipelines of batch jobs. Handles dependency resolution, workflow management, visualization, and Hadoop integration. Built at Spotify and battle-tested in production. Apache 2.0 licensed.

In 15 listsDetails

Apache Airflow

"Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it…

In 13 listsDetails

Dagster

Cloud-native orchestration platform for developing and maintaining data assets including ML models. Declarative programming model with integrated lineage and observability. Apache 2.0 licensed.

In 12 listsDetails

Hamilton

| Python | - A scalable general purpose micro-framework for defining dataflows. You can use it to build dataframes, numpy matrices, python objects, ML models, etc. Embed Hamilton anywhere python runs, e.g. spark, airflow, jupyter, fastapi, python scripts, etc.

In 9 listsDetails

Kestra

Event-driven orchestration and scheduling platform for mission-critical workflows. Infrastructure-as-Code approach with declarative YAML, Git version control integration, and hundreds of plugins for data pipelines and ML workflows. Apache 2.0 licensed.

In 8 listsDetails

Logstash

is a tool for managing events and logs. When used generically, the term encompasses a larger system of log collection, processing, storage and searching activities.

In 7 listsDetails

Kafka

🛠 - 🐙 - Distributed, fault tolerant, high throughput pub-sub messaging system.

In 6 listsDetails

ingestr

CLI tool to copy data from any source to any destination with a single command. Load data into your warehouse before dbt transforms it. Supports 50+ sources including PostgreSQL, MongoDB, Salesforce, Shopify.

In 6 listsDetails