Goldsky

Blockchain data in your own warehouse

Talk to our team

What is blockchain data in your own warehouse?

Blockchain data in your own warehouse means decoded onchain events, such as transfers, trades and mints, loaded into the warehouse your analysts already query, a job data engineers call blockchain ETL.

Finance, data and product teams want it there because their BI tools can then join chain activity with customers, revenue and the rest of the company’s data, under the access rules they already have.

Goldsky backfills your contracts’ history, decodes it and keeps it current, writing Parquet files to your storage bucket or rows straight into tables, while the warehouse, the models and the reports stay yours.

How blockchain data gets into your warehouse

The diagram follows your contracts’ events from the history already on the chain to the BI report that joins them with your own data, and shows who does each step. Your setup may differ.

Choose what to loadThe contracts and events you need

History builds upEvery event since your contracts launched

TurboBackfill the historyHistory loaded first, then kept live

TurboDecode the eventsRaw logs turned into readable columns

Join and reportYour BI tool joins it with your data

TurboLand it in your warehouseFiles in your bucket, or tables directly

Goldsky

How to load blockchain data into your own warehouse

A pipeline reads your contracts’ events from the start of their history, decodes them and keeps writing new ones as blocks arrive, either to a storage bucket as Parquet files or into analytical database tables. Your warehouse loads or queries them from there.

What you’ll need

Goldsky Turbo
The addresses and ABIs of the contracts you want
A warehouse or analytical database for the data
A storage bucket, if your warehouse loads from files
A Kafka cluster, if other systems consume the same events
The BI tool your analysts already use

Choose the contracts and events to load
A warehouse table is only as useful as the events in it, so start from the questions your reports have to answer.
List the contracts, chains and events you need, and get each contract’s ABI (the description of its events) so they can be decoded. An event left out now means a second backfill later.
Backfill the history
Handled by Goldsky Turbo
Reports compare periods, so the warehouse needs every event since your contracts launched, not just new ones.
The load starts at the block where each contract was deployed and moves on to new blocks without a gap or an overlap at the switch. A backfill that skips a range shows up later as totals that don’t match the chain.
A Turbo Pipelines source set to start_at: earliest reads a dataset from the start of its history and then keeps streaming new blocks, so one pipeline covers both the backfill and the live data.
Set where a pipeline starts (opens in a new tab)
Decode the events into columns
Handled by Goldsky Turbo
Raw logs are encoded bytes, and analysts need named columns such as sender, recipient and amount.
Each event is decoded with its contract’s ABI, and token amounts keep their full precision. Large token amounts overflow ordinary number columns, so they need a wide numeric type or a string.
A Turbo Pipelines SQL transform decodes logs as they stream with _gs_log_decode() and your contract’s ABI. For standard token transfers, the curated transfer datasets arrive already decoded.
Decode logs with _gs_log_decode (opens in a new tab)
Land the data in your warehouse
Handled by Goldsky Turbo
Your analysts query the warehouse, so the decoded rows have to arrive in storage you control.
Rows arrive in batches. A crash can deliver a batch twice, and a reorg (the chain replacing its newest blocks) sends corrections for rows already written, so keyed tables absorb both while files and queue consumers need a dedupe step.
Goldsky’s Turbo Pipelines writes Parquet files to an S3 bucket you own, partitioned by date or chain if you choose, and warehouses such as Snowflake, Databricks and BigQuery load them with their own S3 ingestion, since Goldsky has no native sink for them. The same pipeline can also write rows into analytical database tables keyed on a primary_key, or publish them to a Kafka topic for other systems.
Load Parquet files into a warehouse (opens in a new tab)
Join it with your data and report
Chain data answers business questions once it sits next to customers, accounts and revenue.
Your warehouse models join the event tables to your own on shared keys such as wallet address or account ID, and compute totals and period sums there. Without a mapping from address to customer, the reports show wallets, not customers.