Navigating the Transition from Rockset: Exploring Alternatives for Dynamo DB Users

July 12, 2024
Table of Contents

The recent acquisition of Rockset and its impending service sunset in September has prompted a search for alternative solutions. Rockset has been pivotal for many businesses, particularly in facilitating real-time analytics by integrating seamlessly with NoSQL databases like DynamoDB. As we approach the discontinuation of Rockset's services, it's crucial to explore viable replacements and understand the necessary adjustments in your data stack.

The Two Essential Rockset Components to Replace

Rockset delivered a comprehensive solution consisting of two primary components: real-time data ingestion and an analytical (OLAP) database. These features enabled the rapid processing and querying of data. In its absence, businesses might need to deploy two distinct services to fill the gap, although some providers offer bundled solutions with multiple integrated services.

Component 1: Data Ingestion and Integration

Historically, Rockset connected directly to data sources like DynamoDB, replicating data in near-real-time. With its departure, alternatives for handling data ingestion include:

  • Traditional ETL Tools: Generally suited for batch uploads rather than real-time streaming. Examples of common ETL tools include FiveTran and Airbyte.
  • Streaming ETL Tools: These are designed for real-time data flow, accommodating the immediate data streaming needs that batch ETL tools cannot meet. There are a few new players in this space including Artie and Streamkap who are working on integrations for Dynamo.
  • Destination Specific Ingestion and Integration Products: Tools within the AWS ecosystem, like S3 and Redshift, facilitate data syncing from DynamoDB.
  • In-House Data Pipelines: Building custom pipelines is an option, albeit with considerable resource investment in development and maintenance.

Component 2: OLAP Database

Rockset excelled as an analytics database, optimized for high-speed, complex queries. To replicate this, consider these alternatives:

  • Analytics Databases: Products like ClickHouse and SingleStore offer similar capabilities for fast, efficient query handling.
  • Traditional Data Warehouses: While solutions like AWS Redshift are designed for large datasets, they may not always cater to the specific, quick-response analytics typical of Rockset.
  • Specialized Solutions: Technologies like Pinot and StarTree focus on specific aspects of analytics and data management.

Alternative Solutions to Rockset

Adapting to Rockset’s absence involves selecting an appropriate database and corresponding data ingestion method. Here’s a breakdown of potential alternatives:

ClickHouse Ecosystem

Clickhouse provides a robust analytical database framework, suitable for large datasets and rapid queries. To integrate data from DynamoDB:

  • Dynamo → (Kinesis / ClickPipe) → ClickHouse: This pathway leverages AWS Kinesis  and ClickPipe, ensuring data streams through with minimal delay (approximately 5+ seconds).
  • Dynamo → S3 → ClickHouse: Exporting data from Dynamo to S3 is also an option as ClickHouse can ingest data directly from S3 while AWS has built-in solutions for exporting to S3. This method does require batch exports though so will not be streaming or realtime unlike the solution above.

SingleStore

For integrating with SingleStore, consider:

  • Dynamo → (Traditional ETL) → SingleStore: Utilize ETL tools like Fivetran or Airbyte for data transfer, though this method involves batch processing rather than real-time updates.

Redshift

AWS's Redshift offers a structured, albeit less immediate, alternative for handling large datasets from DynamoDB:

  • AWS Dynamo to Redshift Connector : This beta feature simplifies the integration process, though it’s tailored more towards batch processing.

Next Steps

As Rockset phases out, a variety of alternative solutions emerge, each capable of integrating with DynamoDB data for analytics purposes. The choice of technology will depend on specific business needs, particularly regarding real-time processing and query complexity.

For further assistance or to discuss these alternatives in detail, feel free to reach out to the Explo team. We're here to help you navigate this transition smoothly and ensure your analytics capabilities remain robust and effective.

Andrew Chen

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

Heading 1

Heading 2

Heading 3

Heading 4

Heading 5
Heading 6

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.

Block quote

Ordered list

  1. Item 1
  2. Item 2
  3. Item 3

Unordered list

  • Item A
  • Item B
  • Item C

Text link

Bold text

Emphasis

Superscript

Subscript

ABOUT EXPLO

Explo, the publishers of Graphs & Trends, is an embedded analytics company. With Explo’s Dashboard and Report Builder product, you can a premium analytics experience for your users with minimal engineering bandwidth.
Learn more about Explo →