Håkon Åmdal

Jeff Ling

Recently, we launched our new Boost product. 

 is a CDN layer that sits in front of a customer’s blockchain node API, and returns data through a specialized cache, saving you expensive API calls. 

This is more than just a typical HTTP cache or load balancer. We actually created a custom data lake format from scratch with its own domain-specific constraints in order to serve this. As a result, we’re able to serve billions of API calls at a fraction of the cost, passing on the savings directly to customers. 

Only if the design goals are clear and you can check correctness automatically. In the past, engineering something like this would be an immense waste of time. The research itself would take months, and just building the test suite to confidently put something to production would take a whole team a quarter to be sure. 

With today’s LLM-assisted coding workflows, a team is able to build such things much faster, as long as the overall design and goals are extremely clear, and there is a strongly constrained way to determine correctness.

We found this to be the case for our blockchain RPC layer, and were able to design and create something that objectively performs better than any existing solution out there, because we overfit to our specific problem. 

The problem: nodes and general-purpose databases don’t fit

Anyone building on a blockchain knows the standard route: firing JSON-RPC calls at a node. Wallets do it for every user action, while indexers and subgraphs query it continuously block after block. It works, but costs quickly pile up. You either end up paying per-call vendor invoices or managing your own archive nodes—neither of which provides an efficient way to query historical data. A common workaround is scraping those APIs into a dedicated database. While faster than querying a node directly, matching node-like latency this way adds operational cost and complexity.

Solana alone has accumulated nearly 600 billion transactions, growing by 3,000-4,000 every second. At publication time, that represents over 600TB of data—far beyond a typical Postgres setup, while analytical engines like ClickHouse lack the sub-10 ms responsiveness required for real-time APIs.

Prism addresses this with a single geo-replicated data lake: keeping recent tip lookups in memory (sub-millisecond) and routing historical point reads through an in-region cache in roughly 9 ms, delivering billions of sub-10 ms lookups without replicating the entire dataset across regions.

Prism is Goldsky's data lake for blockchain data. It stores each chain once and serves all of our products from the same files.

Prism’s architecture reflects its name. Just as a prism splits a single beam of light into multiple rays, we ingest data from each chain once, store it in a single location, and serve those same bytes across 

 that would otherwise query a vendor on every request. This eliminates the redundant data storage common in setups that maintain separate warehouse copies, RPC caches, and streaming topics for the same chain.

A chain writer ingests blocks and stores them in object storage. We batch many blocks into Vortex files. To serve indexed data, we keep an index using SlateDB, right next to the raw data in object storage. 

Data propagation is an issue in this design. Most queries hit the most recent data, so being able to have all of the readers in the CDN serve live data is important. To solve that, we have writers talk to the readers through a persistent connection, giving readers fresh data to serve no matter where they are, but also giving them the ability to stay stateless since even slightly recent data will be available in object storage. 

This wouldn’t be possible without help from the 

 team: Chris and the rest of that team have been amazing to work with, and we’ve been able to contribute back some of the work into the SlateDB codebase. 

 as the backing object storage, after benchmarking every provider out there. Tigris offers highly performant S3-compatible object storage with no egress fees, making this architecture cost effective in our Boost CDN product. While reads from Tigris are already quite fast compared to many object store solutions, we use the Tigris Acceleration Gateway (TAG), which acts as a cache to make a large proportion of reads get served straight from a centralized cache instead of object storage. 

Building Ponder, we encountered a colorful array of problems with correctness and performance that persist to this day across many RPC providers. I'm excited to see that Goldsky is rethinking this problem from first principles for both realtime and historical requests. Prism's architecture just "makes sense" as the solution to this problem.

Most columnar lakes use Parquet, and it’s a great option especially for OLAP use-cases. However, on read you pay a decompression cost over a large slice of the file, and what you get is Arrow in memory, which is blazingly fast and about ten times the size of the file on disk. That is a fine trade if you scan once and throw the buffer away. It is a bad trade if you want the file to 

 the cache. A point lookup still pays for a bulky decompress.

 keeps the same representation on disk and in memory. We can pin a whole file or a segment in RAM, still compressed, and decompress only the columns a lookup needs. You get roughly ten times the cache room in memory. For a lake that is mostly point reads and short ranges, not giant warehouse scans, that is the right bill.

How a request moves through memory, cache, and object storage

When a wallet, a subgraph, or Turbo asks for data, the reader checks the live window first. If the block is there, we never touch object storage. If it is not, the request is 

: not in memory. Cold traffic goes to TAG, then to Tigris if TAG misses. If you work on databases, this is the same shape as a buffer pool in front of disk, except the “disk” is an object store in another building, and the buffer pool is allowed to be a few terabytes of ordinary cloud SSDs.

Here is how requests are handled across the latency hierarchy:

* A cold miss all the way to the bucket is about between 

. However, we automatically fetch all surrounding data so subsequent queries a user would want are typically warm.  

Some numbers from production, serving thousands to hundreds of thousands of requests per second (RPS).

You might notice that most chains are served fully from the Tip. We designed for the typical access patterns we saw from users, so we were able to optimize this path to make it fast for the user and cheap for us to serve. 

The historical lookups vary according to chain. A lot of this is just due to some per-chain configurations we adjust as we get more metrics. These metrics aren't normalized across usage patterns - some of our users on the slower chains are just accessing our services differently. 

During our development, the team was very responsive and fixed many of the bugs we saw within the same week. That collaboration is a large part of why Prism can serve at all.

We open sourced TAG because a cache in front of object storage should be boring shared infrastructure. Goldsky sent us bugs and patches, we shipped the fixes the same week, and Prism is exactly what we hoped people would build with it.

Serving cacheable methods from Prism also turns a variable RPC bill into a fixed one.

Internally, Goldsky’s indexing services have seen up to 99% cache hit rate using Prism, reducing our reliance on nodes as our traffic continues to grow. 

We saw this and created Boost, giving anybody the ability to hit our Prism data lake while using their existing APIs. When we enabled Prism based caching via 

 for one customer, their upstream usage fell from on the order of 

If you use any RPC provider today, give it a try, and you’ll save up to 80% of your bill! 

We’ll be diving deep into how we approach the caching layer for our data lake, and also how we use different technologies to form indexes that allow for extremely fast cold point lookups. 

If you are building on blockchain data, or building on TAG, we would like to talk. 

 -- and if you want the lake behind it, check out 

Ovais Tariq

Co-Founder and CEO

Kevin Koste

Engineering

Prism: A geo-replicated data lake serving billions of <10 ms point lookups

In this episode of CDPod, the Coinbase Developer Podcast, Goldsky co-founder and CTO Jeff Ling joined fellow engineers Yuga Cohler and Erik Reppel to discuss what it takes to make blockchain data usable at production scale, from subgraphs and multichain indexing to reorg handling, streaming pipelines, and the future of onchain applications. We've organized and expanded it for clarity. 

Blockchain indexing is the process of turning raw blockchain data into structured, queryable data.

A blockchain node contains the source data, but not usually in the shape your product wants. It’s optimized for consensus and execution, not for rendering a beautiful app UI.

An indexer sits between the chain and the application. It reads blocks, transactions, logs, traces, or events, applies decoding and business logic, and stores the result somewhere easier to query.

That might be a GraphQL API. It might be Postgres. It might be Kafka, ClickHouse, a webhook, a data warehouse, or some internal backend service.

The point is the same: the chain tells you what happened, but the indexer turns that into product-ready data.

Why blockchain data indexing is harder than it looks

Onchain data is public. That’s one of crypto’s biggest strengths. But public doesn’t mean usable.

Every swap, transfer, mint, loan, vote, bridge transaction, and contract interaction may live onchain, but that doesn’t automatically give you a clean API, a fast dashboard, a reliable activity feed, or a real-time notification system.

That’s where indexing is useful. For simple apps, you might get away with calling an RPC endpoint and reading contract state directly. But once you need historical data, charts, feeds, aggregates, search, alerts, or anything close to a polished user experience, you need something more durable.

You need a way to listen to the chain, extract the data that matters, transform it, handle edge cases, and serve it back in a form your app can actually use.

"When I was the Head of Engineering at Zora, probably at least 30% of our engineering effort went into indexing."

 — Erik Reppel, Head of Engineering @ Coinbase Developer Platform, creator of x402

This is the hidden data engineering behind great onchain products.

Why is Web3 indexing different from Web2 indexing?

Search engines index the web. Data platforms index documents, transactions, logs, and events. Large Web2 companies deal with massive data pipelines all the time.

The difference in Web3 is that the source data has its own weirdness.

The data is public, but the path to getting clean data can be messy. Nodes can lag. RPC providers can behave differently. Caches can return confusing results. Chains can reorganize. Different networks expose different data models. And the “current” chain tip is always moving.

So the hard part isn’t just reading data.

The hard part is reading it reliably, correcting it when needed, normalizing it across chains, and giving developers an interface that doesn’t force them to care about every underlying edge case.

A good indexing system absorbs a lot of that complexity.

At the beginning, a dapp might not need much indexing.

If all you’re doing is submitting a transaction and showing the result, you can often read directly from the chain. A swap modal, for example, can wait for a transaction to confirm and display the output.

But more complex products need to answer bigger questions:

Those questions are hard to answer with direct RPC calls alone. You need a system that has already processed the raw chain data and organized it around the way your application thinks.

That’s why indexing can quietly become a huge chunk of engineering work. It’s not always the most visible part of the product, but it often determines whether the product feels fast, accurate, and reliable.

A subgraph, popularized by The Graph, is one of the most common ways crypto teams start indexing onchain data.

You define the smart contracts and events you care about, write mapping logic, and expose the processed data through an API, often GraphQL. Instead of building a whole indexing stack from scratch, you get a framework for turning contract events into queryable entities.

That’s why subgraphs became such a core part of the industry. They let teams move quickly. They’re also portable, familiar, and widely used across protocols.

For a lot of teams, the progression looks something like this:

 are a great starting point. They’re not always the final form.

Teams usually start to feel subgraph limitations when they need more speed, more scale, or more control.

A subgraph works well when the indexing logic can run in a relatively straightforward sequence: see an event, update the database, move to the next relevant event.

But some workloads don’t fit that pattern well. If you’re indexing a very high-volume contract, doing analytics across an entire chain, handling dense data from a fast network, or trying to power a real-time experience, sequential processing can become a bottleneck.

At that point, the problem starts to look more like traditional data engineering.

You start thinking about parallel workers, queues, Kafka topics, Flink jobs, stream processing, transformations, consumer groups, partitioning, backfills, and downstream databases.

That’s a very different world from “just give me a GraphQL API.”

Chain reorganizations (reorgs) are one of the classic blockchain data problems.

A block can appear to be part of the canonical chain, then later get replaced. The transactions might still land somewhere else, but their ordering, block hash, or surrounding metadata can change.

For a real-time system, that’s painful because you may have already sent the old data downstream.

Maybe it’s already in a user’s database. Maybe it already powered a notification. Maybe it already updated an analytics table.

Now the system has to say, “Actually, that version of history is no longer valid.”

That’s where careful data modeling matters.

Instead of blindly inserting rows, indexers often need to upsert records, track where each record came from, and include enough metadata to invalidate or correct stale data later.

For EVM-style logs, that means thinking carefully about identifiers like chain ID, block hash, transaction hash, and log index. The exact strategy can vary, but the core idea is simple: when the chain corrects itself, your data system needs a clean way to correct itself too.

In normal application development, inserting a row can be enough.

With blockchain data, especially real-time blockchain data, inserts alone can get you into trouble.

If a block gets replaced, or if the same event is seen again through a different path, you don’t want duplicate or stale records. You want deterministic keys and predictable updates.

That’s why upserts come up so naturally in indexing. You’re not just adding data forever. You’re maintaining a view of what the chain currently says is true.

The trick is choosing keys that survive the weirdness of chain data.

The best systems find a practical abstraction: enough information to handle reorgs and corrections, without making every app team write custom reorg logic.

Supporting one chain is already a lot of work.

Supporting hundreds of chains is a different problem.

Even among EVM chains, there can be differences in RPC behavior, node reliability, block times, indexing requirements, and ecosystem tooling. Once you move beyond EVM, the differences get bigger.

Solana, Cosmos chains, Sui, Aptos, Stellar, Arweave, and other ecosystems don’t all expose data in the same way. Some have different transaction models. Some have different finality assumptions. Some have different event structures. Some require custom adapters before the data can even enter the same pipeline.

That’s why multichain indexing isn’t just “add another RPC URL.”

For a familiar chain type, adding support can be mostly configuration. For a new architecture, it may require new adapters, new normalization logic, and new testing.

The goal is to let developers work with a consistent interface without flattening away the parts of each chain that actually matter.

EVM chains have a big advantage: tooling.

There are well-understood concepts like blocks, transactions, logs, traces, receipts, contract events, chain IDs, and RPC methods. There’s also a large ecosystem of libraries, infrastructure providers, and developers who already know how to work with them.

That doesn’t mean EVM indexing is easy. Reorgs, RPC inconsistencies, high-volume contracts, and fast chains can still create plenty of pain.

But compared with a brand-new data model, EVM has a lot of shared vocabulary.

If you’re building an indexer, shared vocabulary matters. It means fewer surprises, more reusable components, and a faster path from raw chain data to something developers can build with.

The best chains don’t treat indexing as an afterthought. They give developers and infra providers clear specs, predictable data structures, stable APIs, useful test cases, and reference transactions before launch.

If a chain has simple structures and clear documentation, indexers can support it quickly. If the node behavior is inconsistent or the data model is unclear, every downstream team pays for that complexity.

Good chain design isn’t just about execution speed or fees. It’s also about how easy it is for the ecosystem to understand what happened.

Because if wallets, explorers, analytics tools, games, DeFi apps, and AI agents can’t easily read the data, the developer experience suffers.

For familiar chain types, adding support can be surprisingly lightweight.

If the indexing system already understands the chain family, a new network may be mostly configuration: RPC details, chain metadata, data source settings, QA checks, and deployment.

For new chain types, there’s more engineering work. You may need to write an adapter, normalize the data, test edge cases, and make sure the new source fits into the existing pipeline.

The interesting part is the abstraction layer.

A good indexing platform needs to support many different chains without making the rest of the system care about every difference. That means turning chain-specific data into normalized streams that can be transformed, filtered, and delivered to customers.

A lot of it is just grinding through edge cases.

That’s the not-so-secret truth of good data engineering: there usually isn’t one magic trick. There are a thousand small decisions that make the system more reliable over time.

A lot of onchain products need data as soon as possible.

Not eventually. Not after a batch job runs. As close to real time as the system can reasonably get.

That’s especially true for trading apps, prediction markets, games, notifications, dashboards, and user-facing activity feeds.

 are built for that world. Instead of waiting for periodic batch jobs, data moves continuously from the chain into downstream systems.

That sounds simple, but under the hood there are a lot of choices to make.

The best user experience hides most of that. Developers should be able to say what data they want, where it should go, and what transformations they need. The platform can handle the messier details behind the scenes.

Batching and streaming are both useful, but they create very different developer experiences.

Batching is efficient for historical data. If you need to process millions or billions of old records, batch jobs can be cheaper and faster.

Unlocking the secrets of onchain data - Coinbase Developer Podcast 

, the data processing runtime built in Rust that powers 

, our data platform. We focused on making sure it starts up fast, has great performance, and keeping it simple to use and extend.

Streamling is built as a runtime which can take WASM or dynamically linked crate plugins. You tie it together with a YAML file to create a pipeline topology, which then runs. You can use it as a pipeline to write from one datastore to another efficiently, or as a processing runtime with your own complex business logic with data guarantees. 

We wrote this to replace our existing pipeline system, which was powered by Apache Flink. Flink was doing the job, but structurally didn’t feel like the framework we could build the ultimate product with. 

Goldsky is a blockchain data and automation company. We have products that provide a safe and operational read-write layer to the world of blockchain. Our customers process $billions in volume every day, and use our services to build APIs and analytics to power their products. 

, which is probably one of the best Flink self-serve platforms out there. It allowed users to create pipelines from raw blockchain data, massaging it to create their own view of the blockchain, making a permanent and always-synced materialized view in their own database. 

Users use it by making a YAML file like this (a format we kept for the rewrite as well)

Configs like this power some of the biggest teams in the blockchain ecosystem. Most people don’t know what’s going on in the backend, they make a YAML file and go. 

Since launch, we’ve had 3000+ pipelines running concurrently. It handled both full historical backfills and real-time in the same pipeline. It felt so much like a database that users would write disgustingly large joins with SQL queries which would just work, as long as they set the XXL size which was 40 Flink task managers running on instance-level SSDs to support them. I was and am still extremely proud and impressed by our team’s achievement, and the traction (and profit) this product brought us. 

It was an extremely difficult decision. It started with a member of our team experimenting with a ‘lighter’ version of our streaming engine, meant to take data from Kafka, filter it, and send it directly to a webhook. We had a use-case to make tens of thousands of these, all with different transformations, filters, Kafka topics, and webhooks. We realized the startup time was incredibly fast and the resource usage was miniscule compared to our usual pipeline. A research spike showed that we could have a clear line of sight to feature parity, and off we went. 

There were many reasons outside of the obvious ‘Rust fast’ narrative. Flink was impossible for the non-streaming-core team to extend. The checkpointing system was working great but was impossible to inspect. And we were paying AWS a 

The one fact that made the decision for us was cold start speed. Our Flink pipelines would take minutes to start up after each topology change or update. Streamling starts sending data in under 5 seconds. Optimizing for fast cold starts means two key things:

I personally have spent a lot of time dissuading the teams I manage and advise from picking Rust for their projects. It’s always tempting as it’s legitimately so cool, but the cons outweigh the pros in many cases from a CTO perspective. 

In the last two years, times have changed. The verbosity that makes it hard to read and write is less of an issue in our agent-driven age. The time-wasting strictness helps our virtual monkey developers ralph-loop their way into correct solutions, making it an ideal language to be lazy in if you’re incredibly thoughtful about the right constraints. 

The RAD (Rust, Arrow, DataFusion) stack is a huge advantage. While Rust does allow us to think about memory reuse quite deliberately, the huge toolset given to us with Apache DataFusion and Arrow allow us to be almost reckless with how we think about data. 

We can store huge batches in memory, allowing stable write throughput even with huge batches and making sure we’re always constrained by external bottlenecks. 

Not only that, our externally acquired data can be stored in the rawest form possible, making QA and data lineage simpler. More work goes into the pipeline instead of being preprocessed for speed. Remapping, filtering, and more takes milliseconds.

Streaming joins are where the complexity lives in any production data system. Matching two streaming data sources involves a lot of coordination, intermediary state, and complexity. The Flink documentation on joining two streams and the different edge cases is a book, and that is not an accident.

One controversial decision we made with Streamling was to not support stateful SQL. All state should be kept in external databases that have their own HA posture. We started embracing completely denormalized data blobs. 

Our team is full of Flink/Java/Scala experts. They’ve run some of the largest streaming production systems out there at Shopify and Activision. We made it work for Mirror, but chose not to do it for Streamling given today’s data landscape. 

Our position is that if your pipeline needs to do a streaming join, the problem is upstream of the pipeline.

Most of the data we process is event data, often nested in other data contexts. That data is naturally narrow and arrives carrying the keys it needs, because the producer knew what it was producing at the moment it produced it. 

To save space, we normalize tables. Instead of one wide row, we make a few tables and have smaller and sometimes fewer rows. We save the data in one table and the context in another table. It saves space and processing takes less compute, unless you suddenly need to join the data back again. 

Instead, with how cheap storage can get, and how fast we can process and unnest data from larger nested objects with Streamling, we now simply store raw data in fast, cheap storage, and process it inside each customer’s pipeline directly. A downside is tons of redundant data flowing through the pipeline. With Streamling, we process it fast and cheaply enough to where the benefits of easier source-level QA and management and transform-level flexibility makes it completely worth it. 

As a consequence of removing the heavy intermediary state, we managed to achieve our goal: extremely fast startups. 

A different approach to high availability

Flink-style high availability is a tax you pay because catching up from a failure is expensive. It can take minutes for a jobmanager/taskmanager combo to spin back up on an image update, or if a task manager goes down. Standby tasks, hot replicas, checkpoints frequently written to S3, and operator state that has to be rehydrated before the pipeline can resume are all design choices shaped by one assumption, which is that a restart is something you must avoid at almost any cost. 

Our thesis is that if a restart takes seconds, you do not need to avoid them.

Once you have thrown out the keyed state from the previous section, the streaming engine becomes effectively stateless, with nothing except Kafka offsets (or other source metadata) to rehydrate when it comes back up. A supervisor (like Kubernetes) can restart the binary, it reads its offset from a checkpoint stored in a database, and it is back to processing in [N] seconds, which is faster than most Flink jobs notice that a node has failed in the first place. 

This makes it trivial for anyone to have a production setup using a single binary. If you put it under a k8s deployment, if the machine goes down, it’ll automatically restart with maybe a few seconds of data lag. It’ll replay the gap automatically. 

You can still run multiple processes. For example, the Kafka source just uses consumer groups, so multiple processes will just rebalance as needed. Practically, rebalancing takes seconds anyways, so while multiple processes can increase the throughput linearly, it doesn’t necessarily save time. 

Plugins changed how our engineering team works. Here is a whole (contrived and useless) plugin:

Streamling Rust plugin

Every stream processor on the out there advertises itself as extensible, but in practice, extending requires adding to your own implementation of the streaming processor framework. We wanted to take a Runtime approach instead of a Framework approach, meaning the only Streamling code you need to understand is the plugin interface. 

A Streamling plugin, by contrast, is a Rust crate containing a single trait to implement, which will give you the ability to implement row-based or even vectorized transforms. Running it is as simple as compiling the crate and giving a path to the Streamling execution. 

Plugins pass data without copying. Normally, to pass data from one node to another in a pipeline, you have to read, deserialize, transform, serialize, and write. With Streamling, often plugins can directly write on the RecordBatch using Arrow/DataFusion, and manage any copying and transformations deliberately. 

This is what makes Streamling plugins the right interface for AI agents to write code against. We have been internally generating Streamling plugins with Claude for months now. The output is under a hundred lines of Rust, it compiles on the first try most of the time, often running the plugin against the engine multiple times against real data to tune, and it runs as fast as if you wrote it right into the engine itself. 

With agents, everything can be easy, but it’s dangerously easy. Being able to isolate generated code as individual plugins that we can include at runtime allows us to have a strong level of safety and confidence. The isolated nature of each plugin allows us to ship fast and partner with our customers directly without crashing everyone else. In fact, our sales, product, and support teams have now all contributed working plugins to our codebase, often immediately in reaction to a customer ask.

Yaroslav Tkachenko

Xiao Meng

Rafael Aguiar

Avishek Neupane

Open-sourcing Streamling: the data processing runtime for App teams

 Understanding where your data physically lives and what format it's in gets you more throughput from the same resources. We got 12x faster historical backfills in 

 by reading co-located ClickHouse data as native Arrow instead of consuming Avro through Kafka: ~600,000 rows/s vs ~50,000 rows/s on our densest dataset, with no additional infrastructure cost. All benchmarks are on the smallest pipeline size possible. In production, throughput scales up linearly with different sizes (up to 20x on our self-serve tier).

The hard problem: backfills and real-time in one pipeline

Goldsky processes real-time blockchain data across 

. Our customers generally fall into two camps when it comes to historical data:

Both need historical data as fast as possible and real-time events as new blocks land, from a single pipeline.

These are fundamentally different access patterns. Real-time streaming is a firehose of individual events, and Kafka (we use WarpStream) excels at this. Historically, we used Kafka for all backfills too. Kafka is well-suited for bulk sequential reads and it kept load off our data lake. But it meant every message had to be individually deserialized from Avro, and scaling Kafka consumption for large backfills required expensive autoscaling infrastructure.

 architecture in Turbo to address this for the "give me a slice" customers first. The idea: read historical data directly from our ClickHouse data lake via fast-scan, then flip to WarpStream once the pipeline catches up to the chain tip. For selective backfills with filters, this was already working well.

Existing fast-scan: already fast for selective backfills

The hybrid source's fast-scan was optimized for the 

 customers: filtering on a contract address or event type. For these sparse, selective backfills, fast-scan can rip through an entire chain in under 20 seconds.

 customers, we were still using Kafka. This scaled very well since you can have as many parallel workers as you have Kafka partitions, allowing us to have up to 2 million records per second. However, you would be just throwing compute at the problem.

, one of our densest datasets (wide rows, massive volume). All tests used:

Results: Kafka vs bloom-filter vs partition-based pagination

Scenario 1: Kafka consumption baseline, ~50K rows/s

Scenario 2: Arrow with bloom-filter pagination, ~350K rows/s

Scenario 3: Arrow with partition-based pagination, ~600K rows/s

The Arrow format accounts for the bulk of the improvement (7x) and the pagination strategy adds another 1.7x on top.

. Each pipeline is capped at 1 vcpu with 1 parallelism.

With Kafka-based ingest, throughput wasn't necessarily the bottleneck. Scaling Kafka consumption required expensive infrastructure and a pretty fancy autoscaling setup. With the Arrow-from-ClickHouse path, throughput is so high on a single small worker that many full-chain backfills complete without needing to scale at all.

Multiple pipeline workers scale throughput linearly as well. The partition-based pagination scheme is naturally parallelizable, with different workers fetching non-overlapping ranges concurrently. We can scale cheaply, but now we mostly don't have to.

The Arrow-from-ClickHouse path is absurdly efficient. When Turbo consumes from Kafka, every message is Avro-encoded and requires per-message deserialization. For a wide dataset like Base traces multiplied by billions of rows, that CPU cost adds up.

The ClickHouse path eliminates this entirely. Data arrives as native Apache Arrow record batches, the same columnar format Turbo's execution engine (DataFusion) operates on internally. There's no deserialization step. Data flows straight from source into transform and sink stages.

Columnar compression makes the wire transfer compact too. Similar values in the same column cluster together and compress well. A billion-row result set as Arrow is surprisingly small on the wire compared to equivalent Avro messages.

Bloom-filter vs partition-based pagination

. Intuitively, bloom filters are much faster. ClickHouse can use them to skip granules that definitely don't contain matching rows, which is why selective queries (filtering on a contract address) can blast through an entire chain in seconds. For the "give me a slice" customer, this is the right approach.

But when a chain is huge and the data is wide and dense, the physical mechanics change. Bloom filters don't help when nearly every granule contains matching data. On dense datasets, bloom filter queries were causing excessive CPU usage on ClickHouse and frequent timeouts from the database. Page sizes had to stay small to avoid those timeouts, which meant lots of round trips for bulk dumps. This is why we hadn't used the bloom-filter strategy for dense backfills before.

Jeff Ling

Jeff's articles

Solutions

Products

Chains

Build guides

Resources

Company