How do I detect a broken load on Databricks?
Write a custom check comparing today's Databricks row count to the recent baseline. A large drop fails the run so you catch broken loads early. Bruin does this in one platform: ingestion, SQL and Python pipelines, quality checks, lineage, and an AI data analyst that answers in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email and the browser.
Command
bruin runDefined in
YAML + SQL
Works with
Databricks + Bruin CLI
What you get
Anomaly detectionin-pipelineblocking
How it works in code
custom_checks:
- name: fresh
query: SELECT count(*) FROM t WHERE loaded_at < now() - interval '1 day' -- databricksRun bruin run and the Databricks run stops the moment the rule is violated.
Related use cases
Quality checks by warehouse
Enforce freshness on Snowflake source tables
How do I fail a run when a Snowflake source is stale?
Quality checks by warehouseCatch row-count anomalies on Snowflake
How do I detect a broken load on Snowflake?
Quality checks by warehouseEnforce a data contract on Snowflake
How do I enforce a schema contract on a Snowflake table?
Catch bad data before it ships
Bruin CLI and ingestr are on GitHub.
A demo walks through your own data.