Skip to main content

Pace Turbo pipelines with the throttle transform

Follow @goldskyio on Twitter to stay updated on everything we ship.

The new throttle transform caps how fast data moves through a Turbo pipeline. Use it to stay under a downstream API's rate limit or smooth out a bursty source.

Throttle groups records into batches and emits one batch per interval. It doesn't change your data, only when it arrives.

transforms:
  my_throttle:
    type: throttle
    from: my_source
    max_batch_size: 100
    min_batch_interval: 10s

Bounded backfills on Edge EVM datasets

Edge EVM datasets now support end_at. Set it alongside start_at to backfill a fixed block range.

  • Added chain support: Gensyn Mainnet (Subgraphs)
  • Turbo sources, transforms, and sinks now accept batch_size and batch_interval. S3 sinks benefit most
  • ClickHouse sinks now surface table creation errors, like a nullable column in the primary key, so you can fix the config right away
  • The Active tab on the Pipelines page now includes Turbo pipelines
  • New Compose example apps: a prediction market and a proof of reserves app
  • Compose app templates now ship with AGENT.md and SKILLS.md files for AI coding assistants
  • Compose now redeploys the Turbo pipeline behind an onchain trigger only when the trigger itself changes
  • Fixed duplicate Compose failure emails
  • Fixed charts not resizing when you collapse the details pane

© Endless Sky Inc. All rights reserved.

PrivacyTermsSecurity

System status