How do I run SQL and Python transformations together on ClickHouse?
Define each step as a Bruin asset with YAML metadata on top of a .sql or .py file. Bruin resolves dependencies and runs them in order against ClickHouse, mixing SQL and Python in one pipeline.
Command
bruin runDefined in
SQL + Python + YAML
Works with
ClickHouse + Bruin CLI
What you get
SQL and Python assetsdependency resolutionone runtime
How it works in code
-- orders.sql
/* @bruin
name: mart.orders
materialization:
type: table
depends: [raw.orders]
@bruin */
SELECT id, amount, created_at FROM raw.ordersRun bruin run and Bruin runs each asset against ClickHouse in dependency order.
Related use cases
SQL and Python pipelines
Build a SQL + Python pipeline on Snowflake
How do I run SQL and Python transformations together on Snowflake?
SQL and Python pipelinesMaterialize a table incrementally on Snowflake
How do I run incremental models on Snowflake without hand-writing merge logic?
SQL and Python pipelinesBuild a SQL + Python pipeline on BigQuery
How do I run SQL and Python transformations together on BigQuery?
SQL and Python, one pipeline
Bruin CLI and ingestr are on GitHub.
A demo walks through your own data.