Jeff Ling

, the data processing runtime built in Rust that powers 

, our data platform. We focused on making sure it starts up fast, has great performance, and keeping it simple to use and extend.

Streamling is built as a runtime which can take WASM or dynamically linked crate plugins. You tie it together with a YAML file to create a pipeline topology, which then runs. You can use it as a pipeline to write from one datastore to another efficiently, or as a processing runtime with your own complex business logic with data guarantees. 

We wrote this to replace our existing pipeline system, which was powered by Apache Flink. Flink was doing the job, but structurally didn’t feel like the framework we could build the ultimate product with. 

Goldsky is a blockchain data and automation company. We have products that provide a safe and operational read-write layer to the world of blockchain. Our customers process $billions in volume every day, and use our services to build APIs and analytics to power their products. 

, which is probably one of the best Flink self-serve platforms out there. It allowed users to create pipelines from raw blockchain data, massaging it to create their own view of the blockchain, making a permanent and always-synced materialized view in their own database. 

Users use it by making a YAML file like this (a format we kept for the rewrite as well)

Configs like this power some of the biggest teams in the blockchain ecosystem. Most people don’t know what’s going on in the backend, they make a YAML file and go. 

Since launch, we’ve had 3000+ pipelines running concurrently. It handled both full historical backfills and real-time in the same pipeline. It felt so much like a database that users would write disgustingly large joins with SQL queries which would just work, as long as they set the XXL size which was 40 Flink task managers running on instance-level SSDs to support them. I was and am still extremely proud and impressed by our team’s achievement, and the traction (and profit) this product brought us. 

It was an extremely difficult decision. It started with a member of our team experimenting with a ‘lighter’ version of our streaming engine, meant to take data from Kafka, filter it, and send it directly to a webhook. We had a use-case to make tens of thousands of these, all with different transformations, filters, Kafka topics, and webhooks. We realized the startup time was incredibly fast and the resource usage was miniscule compared to our usual pipeline. A research spike showed that we could have a clear line of sight to feature parity, and off we went. 

There were many reasons outside of the obvious ‘Rust fast’ narrative. Flink was impossible for the non-streaming-core team to extend. The checkpointing system was working great but was impossible to inspect. And we were paying AWS a 

The one fact that made the decision for us was cold start speed. Our Flink pipelines would take minutes to start up after each topology change or update. Streamling starts sending data in under 5 seconds. Optimizing for fast cold starts means two key things:

I personally have spent a lot of time dissuading the teams I manage and advise from picking Rust for their projects. It’s always tempting as it’s legitimately so cool, but the cons outweigh the pros in many cases from a CTO perspective. 

In the last two years, times have changed. The verbosity that makes it hard to read and write is less of an issue in our agent-driven age. The time-wasting strictness helps our virtual monkey developers ralph-loop their way into correct solutions, making it an ideal language to be lazy in if you’re incredibly thoughtful about the right constraints. 

The RAD (Rust, Arrow, DataFusion) stack is a huge advantage. While Rust does allow us to think about memory reuse quite deliberately, the huge toolset given to us with Apache DataFusion and Arrow allow us to be almost reckless with how we think about data. 

We can store huge batches in memory, allowing stable write throughput even with huge batches and making sure we’re always constrained by external bottlenecks. 

Not only that, our externally acquired data can be stored in the rawest form possible, making QA and data lineage simpler. More work goes into the pipeline instead of being preprocessed for speed. Remapping, filtering, and more takes milliseconds.

Streaming joins are where the complexity lives in any production data system. Matching two streaming data sources involves a lot of coordination, intermediary state, and complexity. The Flink documentation on joining two streams and the different edge cases is a book, and that is not an accident.

One controversial decision we made with Streamling was to not support stateful SQL. All state should be kept in external databases that have their own HA posture. We started embracing completely denormalized data blobs. 

Our team is full of Flink/Java/Scala experts. They’ve run some of the largest streaming production systems out there at Shopify and Activision. We made it work for Mirror, but chose not to do it for Streamling given today’s data landscape. 

Our position is that if your pipeline needs to do a streaming join, the problem is upstream of the pipeline.

Most of the data we process is event data, often nested in other data contexts. That data is naturally narrow and arrives carrying the keys it needs, because the producer knew what it was producing at the moment it produced it. 

To save space, we normalize tables. Instead of one wide row, we make a few tables and have smaller and sometimes fewer rows. We save the data in one table and the context in another table. It saves space and processing takes less compute, unless you suddenly need to join the data back again. 

Instead, with how cheap storage can get, and how fast we can process and unnest data from larger nested objects with Streamling, we now simply store raw data in fast, cheap storage, and process it inside each customer’s pipeline directly. A downside is tons of redundant data flowing through the pipeline. With Streamling, we process it fast and cheaply enough to where the benefits of easier source-level QA and management and transform-level flexibility makes it completely worth it. 

As a consequence of removing the heavy intermediary state, we managed to achieve our goal: extremely fast startups. 

A different approach to high availability

Flink-style high availability is a tax you pay because catching up from a failure is expensive. It can take minutes for a jobmanager/taskmanager combo to spin back up on an image update, or if a task manager goes down. Standby tasks, hot replicas, checkpoints frequently written to S3, and operator state that has to be rehydrated before the pipeline can resume are all design choices shaped by one assumption, which is that a restart is something you must avoid at almost any cost. 

Our thesis is that if a restart takes seconds, you do not need to avoid them.

Once you have thrown out the keyed state from the previous section, the streaming engine becomes effectively stateless, with nothing except Kafka offsets (or other source metadata) to rehydrate when it comes back up. A supervisor (like Kubernetes) can restart the binary, it reads its offset from a checkpoint stored in a database, and it is back to processing in [N] seconds, which is faster than most Flink jobs notice that a node has failed in the first place. 

This makes it trivial for anyone to have a production setup using a single binary. If you put it under a k8s deployment, if the machine goes down, it’ll automatically restart with maybe a few seconds of data lag. It’ll replay the gap automatically. 

You can still run multiple processes. For example, the Kafka source just uses consumer groups, so multiple processes will just rebalance as needed. Practically, rebalancing takes seconds anyways, so while multiple processes can increase the throughput linearly, it doesn’t necessarily save time. 

Plugins changed how our engineering team works. Here is a whole (contrived and useless) plugin:

Streamling Rust plugin

Every stream processor on the out there advertises itself as extensible, but in practice, extending requires adding to your own implementation of the streaming processor framework. We wanted to take a Runtime approach instead of a Framework approach, meaning the only Streamling code you need to understand is the plugin interface. 

A Streamling plugin, by contrast, is a Rust crate containing a single trait to implement, which will give you the ability to implement row-based or even vectorized transforms. Running it is as simple as compiling the crate and giving a path to the Streamling execution. 

Plugins pass data without copying. Normally, to pass data from one node to another in a pipeline, you have to read, deserialize, transform, serialize, and write. With Streamling, often plugins can directly write on the RecordBatch using Arrow/DataFusion, and manage any copying and transformations deliberately. 

This is what makes Streamling plugins the right interface for AI agents to write code against. We have been internally generating Streamling plugins with Claude for months now. The output is under a hundred lines of Rust, it compiles on the first try most of the time, often running the plugin against the engine multiple times against real data to tune, and it runs as fast as if you wrote it right into the engine itself. 

With agents, everything can be easy, but it’s dangerously easy. Being able to isolate generated code as individual plugins that we can include at runtime allows us to have a strong level of safety and confidence. The isolated nature of each plugin allows us to ship fast and partner with our customers directly without crashing everyone else. In fact, our sales, product, and support teams have now all contributed working plugins to our codebase, often immediately in reaction to a customer ask.

Yaroslav Tkachenko

Xiao Meng

Rafael Aguiar

Avishek Neupane

Open-sourcing Streamling: the data processing runtime for App teams

Mirror pipelines are a new way to think about blockchain data. Teams get data 

 into their infrastructure, as opposed to traditional API polling architectures. Goldsky developed a unique, zero-copy, realtime streaming system that can work with a variety of databases, used by companies like 

 last year, we immediately knew that we wanted to use data streaming technologies like 

 as a foundation. Theoretically, the blockchain could be modelled very well as a traditional Kafka changelog. However, we didn’t realize how 

 they actually work for the blockchain data.

In this post, we’ll illustrate how several concepts from the world of data streaming and stream processing can be applied to Web3 data.

Decentralized Ledger 🤝 Distributed Log

Modern distributed log technologies like Apache Kafka have several key qualities:

Sounds familiar? Because it is — blockchains have the same qualities! Blocks are immutable. They are also strictly ordered (hence the “chain”) and have a monotonically increasing 

💡 Of course, there is an exception to this rule: reorgs. We’ll cover it in a bit!

So, it’s only natural to take blockchain data and put it into a distributed log like Apache Kafka, for example, with a topic per dataset (blocks, transactions, logs, receipts, etc.). Raw blockchain data has a fairly straightforward schema, and it can benefit from a strict format like 

 (backed by a schema registry, of course).

And that’s precisely how Goldsky’s data platform is designed!

When all your data is available as topics in Apache Kafka, it’s very easy to apply stream processing with a tool like Apache Flink to project, filter, or aggregate data in any way.

 data using a certain contract address. Or build a Top 10 

 of the wallet balances. Or even perform complex 

It’s also very easy to start from the genesis block by consuming data in a topic from the 

 offset. Or you can decide to process data from the current point in time by using the 

 is a powerful SQL dialect that supports most of Flink’s functionality.

However, things can get very complicated very quickly. Typically, mature streaming deployment would require a dedicated team to manage the resources, data freshness, QA checks, and downstream consumers. We built Mirror as a serverless solution that allows users to just specify a YAML file to get started.

Earlier, we said that the blockchains are immutable. It’s almost true, but sometimes (depending on the blockchain), a reorg can happen, which, in practice, means that the current data for a certain block should be retracted and corrected.

And again, there is a very similar concept of retractions in the data streaming world! For example, when a windowed aggregation is performed, it’s typical to configure a watermark which can control how late an event can arrive. So we could model a reorg as a 

 event, and Flink will perform retractions and update results automatically! Since the finality time is known for each chain, we don’t need to guess when choosing the watermark.

A different, more generic approach is to model every data stream as a changelog stream. In this case, whenever a reorg happens, we just need to re-emit all data for a block that’s no longer valid with a DELETE marker. It’ll be propagated to all defined transformations and sinks. That’s why when using Goldsky and a database like 

, you don’t need to think about reorgs, they’re handled for you.

 are a big source of semantic data in the blockchain world, and contains a lot of valuable abstractions, often written by protocol authors themselves. 

We wanted to allow our users to take advantage of Mirror without having to rewrite everything with SQL transforms, using subgraphs as a data source.

We realized subgraphs could be deconstructed into a changelog stream using Change Data Capture techniques, which allows the modelling of all changes as INSERTs, UPDATEs or DELETEs.

And since all records have primary keys, it’s easy to leverage upserts to maintain the latest version of each entity in the destination database. For example, when you mirror subgraph data to 

 with Goldsky, you automatically get the latest version of each entity. So subgraphs with many updates don’t take up a lot of storage.

The synergy between data streaming and decentralized data is clear, and Mirror is flexible enough to support a wide range of data sources and sinks.

Loading off-chain data, getting a price feed, or loading NFT metadata are all examples of data enrichment, a very well-known problem in the data streaming space.

Since we already did the upfront work of modelling the blockchain and snapshotting data into streams, an incredible ecosystem of 

 in the community makes it easy to deliver raw or transformed data to any datastore with low latency.

The possibilities are endless, and we’re very excited about what Goldsky’s architecture can provide for builders in the future. 🚀

 mission is to unleash the full potential of Web3 data.

We’re solving complex data engineering challenges so that builders can focus on what matters most — building transformational applications and delivering value to users.

 for full access to Goldsky’s realtime data streaming platform today, and be sure to give us a follow on 

Using Changelogs and Streams to Solve Blockchain Data Challenges

 is a robust data serialization framework that’s frequently used with data streaming systems like Apache Kafka and Redpanda. We use it in Goldsky as a storage format for all our topics.

Avro schemas are stored in a Schema Registry and, most of the time, don’t require any extra attention: a message producer publishes a message and, optionally, registers a schema (if it’s new). A message consumer reads the message and, optionally, fetches the schema from the Schema Registry.

But what if you forget to upload the right schema? Different client libraries provide different guarantees, and we learned the hard way that it can be very easy to accidentally publish a message with an incorrect schema. However, this is a story with a happy end, so we were able to turn the bug into a feature by tweaking the way Avro decoders work!

Mirror is our flagship product, which we use to stream onchain data from various blockchains directly into customers’ existing database infrastructure. The idea behind Mirror is to make it much easier for developers to work with event logs, transactions, and traces emitted from the blockchain, streamlining builds for complex (and essential) data pipelines.

One sunny day in June, some of our Flink applications started to fail with various deserialization exceptions like:

We immediately started debugging this issue and were able to reproduce it locally. The discovery we made was surprising at the beginning: no matter what recent message we started consuming from, the 

 message would always be decoded successfully, but the 

 would always fail. We tried different topics, partitions and offsets. Sometimes the exceptions changed, but all came from the Avro decoder.

 was especially puzzling: all records in affected topics were fairly small. After some debugging (by the way, do you know you can catch OutOfMemoryErrors?), we realized that the Avro decoder tried to read the string and allocate a few gigabytes of memory for that! 🤯

Another equally puzzling thing: Redpanda Console, which we use for browsing topics, didn’t experience any issues.

The Avro Spec and Why Schemas Are So Important

Apache Avro supports two types of encodings: binary and JSON. In a production environment, binary encoding is the most popular one - it’s very efficient. Here’s the quote from the 

Binary encoding does not include field names, self-contained information about the types of individual bytes, nor field or record separators. Therefore readers are wholly reliant on the schema used when the data was encoded.

So this means that the Avro payload just contains a bunch of byte values that represent different values together, and you need a schema to be used as a map to navigate between them. Now let’s see how strings specifically are encoded:

 is encoded as a long followed by that many bytes of UTF-8 encoded character data. For example, the three-character string “foo” would be encoded as the long value 3 (encoded as hex 06) followed by the UTF-8 encoding of ‘f’, ‘o’, and ‘o’ (the hex bytes 66 6f 6f)

It seems that, somehow, when reading the payload, the Avro decoder used an incorrect value for the string length, which caused OutOfMemoryError.. but why? And why doesn’t it affect Redpanda Console?

At this time, we also found the producer application emitting the data, which was a Golang application. A suspicious commit introduced a new payload field, but after some investigation, we realized that the updated schema was never uploaded to the Schema Registry 😱. So all recent payloads had an 

 representing the value of the new field.

 from the Java client became our main suspect. Here’s what the documentation says:

This class may read-ahead and buffer bytes from the source beyond what is required to serve its read methods.

) that can be reused for decoding different payloads.

So, to summarize, this is what the decoder expects based on the schema:

Each field consists of a certain number of bytes dictated by the schema. But instead, this is what happens:

It assumes it’s reading the second payload, but it’s actually still reading the first!

Is there anything we can do to fix decoding without reprocessing a lot of data? And why didn’t it affect Redpanda Console, which uses a decoding library written in Golang (

It turns out that using a shared buffer is not really necessary. It may result in some performance improvements, but it looks like it’s not how the 

 is implemented, for example. And even Avro implementation in Java comes with a different decoder called 

, we no longer saw any deserialization issues! 🎉 The payloads are still, technically, incorrect (they don’t match the schema), but we’re able to read them just fine. It was only possible because we added a new field 

, so it doesn’t affect other fields and can be safely ignored. If we were to put the new field in the middle, it’d result in the same issue since the order of fields would be violated.

To start building onchain data pipelines of your own, visit 

 to create an account. We handle the heavy engineering lift, making it easy to build enterprise-grade data workflows to build powerful applications faster. With full-featured indexing services, subgraph hosting, and streaming capabilities, you can rest easy knowing your data stack is in good hands.

To keep up with the latest Goldsky news and updates, be sure to give us a follow on 

Real-time Data

RPC Infrastructure

Automation

Open Source

Follow

Yaroslav Tkachenko

Yaroslav's articles