Awesome Big Data
Section: Data Quality and Observability · open-source data quality tool that profiles tables, generates tests from the profiling results, and monitors freshness, volume, and schema, with a web UI.
Entry
Appears in 4 awesome lists
Open-source data quality tool that profiles tables and generates and runs data quality tests; connects to DuckDB databases and uses DuckDB to profile and test CSV, Parquet, and JSON files locally, in Amazon S3, or in Azure Blob Storage. Apache 2.0.
Section: Data Quality and Observability · open-source data quality tool that profiles tables, generates tests from the profiling results, and monitors freshness, volume, and schema, with a web UI.
Section: Testing · Open-source data quality tool that profiles tables, generates validation tests from the profiling results, and monitors freshness, volume, and schema from a web UI.
Section: Data Quality & Governance · Open-source data quality tool that profiles tables and generates and runs data quality tests; connects to DuckDB databases and uses DuckDB to profile and test CSV, Parquet, and JSON files locally, in Amazon S3, or in Azure Blob Storage. Apache 2.0.
Section: Data Validation · Profiles tables, generates data quality tests, and monitors freshness, volume, and schema.
Gemini CLI extension with three Agent Skills for analytics engineering: explore a warehouse or DuckDB file read-only (profiles, PII flags, verified joins), author dbt models and semantic layers as reviewable diffs, and detect schema drift. Install with gemini extensions install…
Open Source data validation framework to manage data quality. Users can define and document “expectations” rules about how data should look and behave.
A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production.
Command-line tool to test data against data contracts in the Open Data Contract Standard (ODCS). Converts from and to formats such as SQL DDL, dbt, Avro, and JSON Schema. Uses DuckDB to run schema and quality checks on sources such as CSV, Parquet and JSON files (local, S3, GCS, Azure), DuckDB…
Zero-config data quality CLI. Profiles every table on first run, then auto-detects anomalies (volume drops, schema drift, freshness misses, distribution shifts) on subsequent runs. No YAML, no rules to write. Works with Postgres, BigQuery, Snowflake, and dbt.
Interview practice with SQL query execution, Python, and data modeling exercises.
Open-source tool that scans SQL databases for sensitive columns, masks them with deterministic fakes, and validates the masked copy row by row against the original.
A lightweight Snowflake emulator built with Go and DuckDB for local development and testing.