Amazon S3 integration
Amazon S3 data, in your warehouse.
A built-in ingestr connector: add credentials, pick tables, schedule it.
name: raw.table
type: ingestr
parameters:
source_connection: amazon_s3
source_table: '<table>'
destination: snowflake
incremental_strategy: merge$ bruin run assets/raw/amazon_s3.asset.yml
- extract · Amazon S3 dataincremental
- load · snowflake raw.datamerged
- checks · not_null, unique
Loaded and checked. Downstream models can run.
How it connects
Connected in three steps.
Scalable cloud storage service from AWS that allows users to store and retrieve extensive amounts of data with support for various file formats and S3-compatible services.
- 01
Add a Amazon S3 connection with its credentials.
- 02
Pick the tables to load and how: replace, append or merge.
- 03
Bruin runs it on your schedule and checks every load.
Connection parameters
- access_key_id
- AWS access key ID
- secret_access_key
- AWS secret access key
- endpoint_url
- URL of S3-compatible API server (for destinations)
- layout
- Layout template for file organization (for destinations)
Step-by-step
Pick a destination.
Amazon S3 → Snowflake
Read the guide
Amazon S3 → Google BigQuery
Read the guide
Amazon S3 → Databricks
Read the guide
Amazon S3 → Amazon Redshift
Read the guide
Amazon S3 → PostgreSQL
Read the guide
Amazon S3 → ClickHouse
Read the guide
Amazon S3 → DuckDB
Read the guide
Amazon S3 → MotherDuck
Read the guide
The platform
Part of the Bruin platform.
Data in, ready for everything downstream: the models, the checks, the lineage and the AI layer.
Sources
DatabasesWarehousesApps & APIsFiles & storageStreams & webhooksWeb scrapingMove
Data IngestionFrequently asked
Questions about Amazon S3.
Does Bruin have a built-in Amazon S3 integration?
Yes. Built-in ingestr source. Built-in ingestr destination. Runs SQL and Python assets.
Where can Amazon S3 data go?
Snowflake, BigQuery, Databricks, Redshift, ClickHouse, Postgres, DuckDB, MotherDuck, Microsoft Fabric and more, plus files on S3 and GCS.
How fresh is the data?
As fresh as your schedule. Incremental loads append, merge or replace a time window, every few minutes if you like.
Do we need Bruin Cloud?
No. The Bruin CLI and ingestr run locally, in CI or in your own orchestrator. Bruin Cloud adds scheduling, lineage, alerts and the AI data analyst on top.
Ready to connect Amazon S3?
$100 in credits and 50 AI tasks. No credit card.
A demo walks through your own data.