Build a Self-Healing Pipeline Agent
What the agent reads
In Bruin, every table or view is built by an asset: one file holding the SQL or Python plus the metadata that describes what it produces. There is no separate catalog to keep in sync, which is the reason any of this is workable.
You are not documenting everything
A pipeline can easily have two hundred assets, and nobody is documenting two hundred assets by hand. You do not have to.
When something breaks, the agent does not read the project. It:
- Starts at the failed asset.
- Walks the lineage.
- Opens the few files on that path.
For a typical incident that is five to fifteen assets, no matter how big the project is.
So the target is not "every asset is beautifully documented". The target is that whichever path the agent walks, the files it opens tell it something useful.
Build context from the top down
Broad context first. It is cheap to write, and it makes everything below it easier to review.
- Project and pipeline level. An
AGENTS.mdat the repo root for how to work here and for company-wide context, plus aREADME.mdin each pipeline folder for why that pipeline exists and why it is shaped the way it is. A coding agent reads these before it opens any SQL. For most projects this is an afternoon of work, and it improves every answer that follows. - Asset and column descriptions. Write a plain sentence per asset, then run
bruin ai enhanceacross the pipeline folder to fill in column-level detail and suggest checks. Step 3 covers running that at scale and reviewing what it produces. - Quality checks and unit tests. These come last and carry the most weight, because they record answers rather than descriptions.
Checks and tests answer the questions agents ask
Three different jobs:
- A description says what a column is.
- A check says what is true about it.
- A unit test says what the logic does with it.
The third kind is what agents keep needing:
| Question an agent has | What settles it |
|---|---|
| What is the grain of this table? | A unique check on order_id states one row per order in a way that cannot be misread. |
| Does this revenue figure include unrecognized revenue? | A custom check on the filter. |
| Are deactivated or unsubscribed users counted in this report? | A unit test with a deactivated user in the input rows and an expected output that excludes them. It settles the question permanently. |
An agent reading a description has to take it on trust. An agent reading a check knows the rule is enforced on every run, and can tell you the exact run where it stopped holding. That is why business logic belongs in checks and tests rather than only in prose.
The rest of this step covers what a complete asset file looks like, which parts you write versus generate, and the habits that keep all of it current.