Prism: A geo-replicated data lake serving billions of <10 ms point lookups

Key takeaways

• Under 1ms hot lookups and <100ms cold reads globally. About 90% of requests hit the chain tip, so Prism keeps recent blocks in memory in every region and serves history from a single copy in object storage through an in-region cache.
• Pick a file format that stays compressed in memory. Vortex gives roughly 10x the cache room of Parquet decompressed into Arrow.
• LLM-assisted development makes a domain-specific data lake practical when the design goals are clear and there's a strict way to check correctness. Otherwise, stick with Iceberg, Parquet, ClickHouse, and other industry leading tools.

Recently, we launched our new Boost product. Boost is a CDN layer that sits in front of a customer’s blockchain node API, and returns data through a specialized cache, saving you expensive API calls. 

This is more than just a typical HTTP cache or load balancer. We actually created a custom data lake format from scratch with its own domain-specific constraints in order to serve this. As a result, we’re able to serve billions of API calls at a fraction of the cost, passing on the savings directly to customers. 

Should you build your own data lake? 

Only if the design goals are clear and you can check correctness automatically. In the past, engineering something like this would be an immense waste of time. The research itself would take months, and just building the test suite to confidently put something to production would take a whole team a quarter to be sure. 

With today’s LLM-assisted coding workflows, a team is able to build such things much faster, as long as the overall design and goals are extremely clear, and there is a strongly constrained way to determine correctness.

We found this to be the case for our blockchain RPC layer, and were able to design and create something that objectively performs better than any existing solution out there, because we overfit to our specific problem. 

The problem: nodes and general-purpose databases don’t fit

Anyone building on a blockchain knows the standard route: firing JSON-RPC calls at a node. Wallets do it for every user action, while indexers and subgraphs query it continuously block after block. It works, but costs quickly pile up. You either end up paying per-call vendor invoices or managing your own archive nodes—neither of which provides an efficient way to query historical data. A common workaround is scraping those APIs into a dedicated database. While faster than querying a node directly, matching node-like latency this way adds operational cost and complexity.

Solana alone has accumulated nearly 600 billion transactions, growing by 3,000-4,000 every second. At publication time, that represents over 600TB of data—far beyond a typical Postgres setup, while analytical engines like ClickHouse lack the sub-10 ms responsiveness required for real-time APIs.

Prism addresses this with a single geo-replicated data lake: keeping recent tip lookups in memory (sub-millisecond) and routing historical point reads through an in-region cache in roughly 9 ms, delivering billions of sub-10 ms lookups without replicating the entire dataset across regions.

Write once, refract many

Prism is Goldsky's data lake for blockchain data. It stores each chain once and serves all of our products from the same files.

Prism’s architecture reflects its name. Just as a prism splits a single beam of light into multiple rays, we ingest data from each chain once, store it in a single location, and serve those same bytes across Turbo scans, Edge RPC and Boost point lookups, and subgraphs that would otherwise query a vendor on every request. This eliminates the redundant data storage common in setups that maintain separate warehouse copies, RPC caches, and streaming topics for the same chain.

A chain writer ingests blocks and stores them in object storage. We batch many blocks into Vortex files. To serve indexed data, we keep an index using SlateDB, right next to the raw data in object storage.

Data propagation is an issue in this design. Most queries hit the most recent data, so being able to have all of the readers in the CDN serve live data is important. To solve that, we have writers talk to the readers through a persistent connection, giving readers fresh data to serve no matter where they are, but also giving them the ability to stay stateless since even slightly recent data will be available in object storage. 

This wouldn’t be possible without help from the SlateDB (opens in a new tab) team: Chris and the rest of that team have been amazing to work with, and we’ve been able to contribute back some of the work into the SlateDB codebase. 

We selected Tigris (opens in a new tab) as the backing object storage, after benchmarking every provider out there. Tigris offers highly performant S3-compatible object storage with no egress fees, making this architecture cost effective in our Boost CDN product. While reads from Tigris are already quite fast compared to many object store solutions, we use the Tigris Acceleration Gateway (TAG), which acts as a cache to make a large proportion of reads get served straight from a centralized cache instead of object storage. 

Building Ponder, we encountered a colorful array of problems with correctness and performance that persist to this day across many RPC providers. I'm excited to see that Goldsky is rethinking this problem from first principles for both realtime and historical requests. Prism's architecture just "makes sense" as the solution to this problem.

- Kevin Koste, Creator of Ponder (opens in a new tab) and Co-author of MIP-16 (opens in a new tab)

Why we use Vortex instead of Parquet

Most columnar lakes use Parquet, and it’s a great option especially for OLAP use-cases. However, on read you pay a decompression cost over a large slice of the file, and what you get is Arrow in memory, which is blazingly fast and about ten times the size of the file on disk. That is a fine trade if you scan once and throw the buffer away. It is a bad trade if you want the file to be the cache. A point lookup still pays for a bulky decompress.

Vortex (opens in a new tab) keeps the same representation on disk and in memory. We can pin a whole file or a segment in RAM, still compressed, and decompress only the columns a lookup needs. You get roughly ten times the cache room in memory. For a lake that is mostly point reads and short ranges, not giant warehouse scans, that is the right bill.

How a request moves through memory, cache, and object storage

When a wallet, a subgraph, or Turbo asks for data, the reader checks the live window first. If the block is there, we never touch object storage. If it is not, the request is cold: not in memory. Cold traffic goes to TAG, then to Tigris if TAG misses. If you work on databases, this is the same shape as a buffer pool in front of disk, except the “disk” is an object store in another building, and the buffer pool is allowed to be a few terabytes of ordinary cloud SSDs.

Here is how requests are handled across the latency hierarchy:

* A cold miss all the way to the bucket is about between 10 ms and 800 ms. However, we automatically fetch all surrounding data so subsequent queries a user would want are typically warm.  

Latency by chain in production

Some numbers from production, serving thousands to hundreds of thousands of requests per second (RPS).

You might notice that most chains are served fully from the Tip. We designed for the typical access patterns we saw from users, so we were able to optimize this path to make it fast for the user and cheap for us to serve. 

The historical lookups vary according to chain. A lot of this is just due to some per-chain configurations we adjust as we get more metrics. These metrics aren't normalized across usage patterns - some of our users on the slower chains are just accessing our services differently. 

Why we chose Tigris and TAG

We chose Tigris for two main reasons:

1. This is an extremely high egress workload, and Tigris offers free egress, along with a few other providers. 
2. Of all the object storage providers we tried, Tigris offered the most stable performance for our workload 

During our development, the team was very responsive and fixed many of the bugs we saw within the same week. That collaboration is a large part of why Prism can serve at all.

We open sourced TAG because a cache in front of object storage should be boring shared infrastructure. Goldsky sent us bugs and patches, we shipped the fixes the same week, and Prism is exactly what we hoped people would build with it.

- Ovais Tariq, Founder of Tigris Data

Prism saves users money

Serving cacheable methods from Prism also turns a variable RPC bill into a fixed one.

Internally, Goldsky’s indexing services have seen up to 99% cache hit rate using Prism, reducing our reliance on nodes as our traffic continues to grow. 

We saw this and created Boost, giving anybody the ability to hit our Prism data lake while using their existing APIs. When we enabled Prism based caching via Boost for one customer, their upstream usage fell from on the order of 200M requests/day to about 80M, saving them 5 figures a month. 

If you use any RPC provider today, give it a try, and you’ll save up to 80% of your bill! 

More details

We’ll be diving deep into how we approach the caching layer for our data lake, and also how we use different technologies to form indexes that allow for extremely fast cold point lookups. 

If you are building on blockchain data, or building on TAG, we would like to talk. TAG is open source (opens in a new tab) -- and if you want the lake behind it, check out Turbo, Edge RPC and Boost.