Reinventing Kafka on object storage (Interview)
Ryan Worl, Co-founder and CTO at WarpStream, joins us to talk about the world of Kafka and data streaming and how WarpStream redesigned the idea of Kafka to run in modern cloud environments directly on top of object storage. Last year they posted a blog titled, “Kafka is dead, long live Kafka” that hit the top of Hacker News to put WarpStream on the map. We get the backstory on Kafka and why it’s so widely used, who created it and for what purpose, and the behind the scenes on all things WarpStream.
Join the discussion
Changelog++ members save 14 minutes on this episode because they made the ads disappear. Join today!
Sponsors:
Speakeasy – Production-ready, enterprise-resilient, best-in-class SDKs crafted in minutes. Speakeasy takes care of the entire SDK workflow to save you significant time, delivering SDKs to your customers in minutes with just a few clicks! Create your first SDK for free!
Supabase – Supabase just finished their 12th launch week! Check it out. Or get a month of Supabase Pro (FREE) by going to supabase.com/changelogpod
Paragon – Ship native integrations to production in days with more than 130 pre-built connectors, or configure your own custom integrations. Built for product and engineering. Learn more at useparagon.com/changelog
Unblocked – Other developer tools can’t tell you how your codebase works and why. Unblocked can. We augment your code with context from Slack, Confluence, Jira, and more, so you get accurate answers without having to search for them. Sign up for free at getunblocked.com
Featuring:
Ryan Worl – Website, GitHub, LinkedIn, X
Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, X
Jerod Santo – Website, GitHub, LinkedIn, Mastodon, X
Show Notes:
Kafka is dead, long live Kafka
Apache Kafka - open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Current 2024
Amazon Managed Streaming for Apache Kafka
Bobby Brown - My Prerogative
Something missing or broken? PRs welcome!
Using FoundationDB As The Bedrock For Your Distributed Systems
Summary
The database market continues to expand, offering systems that are suited to virtually every use case. But what happens if you need something customized to your application? FoundationDB is a distributed key-value store that provides the primitives that you need to build a custom database platform. In this episode Ryan Worl explains how it is architected, how to use it for your applications, and provides examples of system design patterns that can be built on top of it. If you need a foundation for your distributed systems, then FoundationDB is definitely worth a closer look.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With 200Gbit private networking, scalable shared block storage, and a 40Gbit public network, you’ve got everything you need to run a fast, reliable, and bullet-proof data platform. If you need global distribution, they’ve got that covered too with world-wide datacenters including new ones in Toronto and Mumbai. And for your machine learning workloads, they just announced dedicated CPU instances. Go to dataengineeringpodcast.com/linode today to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show!
Alluxio is an open source, distributed data orchestration layer that makes it easier to scale your compute and your storage independently. By transparently pulling data from underlying silos, Alluxio unlocks the value of your data and allows for modern computation-intensive workloads to become truly elastic and flexible for the cloud. With Alluxio, companies like Barclays, JD.com, Tencent, and Two Sigma can manage data efficiently, accelerate business analytics, and ease the adoption of any cloud. Go to dataengineeringpodcast.com/alluxio today to learn more and thank them for their support.
Understanding how your customers are using your product is critical for businesses of any size. To make it easier for startups to focus on delivering useful features Segment offers a flexible and reliable data infrastructure for your customer analytics and custom events. You only need to maintain one integration to instrument your code and get a future-proof way to send data to over 250 services with the flip of a switch. Not only does it free up your engineers’ time, it lets your business users decide what data they want where. Go to dataengineeringpodcast.com/segmentio today to sign up for their startup plan and get $25,000 in Segment credits and $1 million in free software from marketing and analytics companies like AWS, Google, and Intercom. On top of that you’ll get access to Analytics Academy for the educational resources you need to become an expert in data analytics for measuring product-market fit.
You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data management. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, and the Open Data Science Conference. Go to dataengineeringpodcast.com/conferences to learn more and take advantage of our partner discounts when you register.
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the mailing list, read the show notes, and get in touch.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Your host is Tobias Macey and today I’m interviewing Ryan Worl about FoundationDB, a distributed key/value store that gives you the power of ACID transactions in a NoSQL database
Interview
Introduction
How did you get involved in the area of data management?
Can you explain what FoundationDB is and how you got involved with the project?
What are some of the unique use cases that FoundationDB enables?
Can you describe how FoundationDB is architected?
How is the ACID compliance implemented at the cluster level?
What are some of the mechanisms built into FoundationDB that contribute to its fault tolerance?
How are conflicts managed?
FoundationDB has an interesting feature in the form of Layers that provide different semantics on the underlying storage. Can you describe how that is implemented and some of the interesting layers that are available?
Is it possible to apply different layers, such as relational and document, to the same underlying objects in storage?
One of the aspects of FoundationDB that is called out in the documentation and which I have heard about elsewhere is the performance that it provides. Can you describe some of the implementation mechanics of FoundationDB that allow it to provide such high throughput?
For someone who wants to run FoundationDB can you describe a typical deployment topology?
What are the scaling factors for the underlying storage and for the Layers that are operating on the cluster?
Once you have a cluster deployed, what are some of the edge cases that users should watch out for?
How are version upgrades managed in a cluster?
What are some of the ways that FoundationDB impacts the way that an application developer or data engineer would architect their software as compared to working with something like Postgres or MongoDB?
What are some of the more interesting/unusual/unexpected ways that you have seen FoundationDB used?
When is FoundationDB the wrong choice?
What is in store for the future of FoundationDB?
Contact Info
LinkedIn
@ryanworl on Twitter
Website
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
FoundationDB
Jepsen
Andy Pavlo
Archive.org – The Internet Archive
FoundationDB Summit
Flow Language
C++
Actor Model
Erlang
Zookeeper
Podcast Episode
PAXOS consensus algorithm
Multi-Version Concurrency Control (MVCC) AKA Optimistic Locking
ACID
CAP Theorem
Redis
Record Layer
CloudKit
Document Layer
Segment
Podcast Episode
NVMe
SnowflakeDB
FlatBuffers
Protocol Buffers
Ryan Worl FoundationDB Summit Presentation
Google F1
Google Spanner
WaveFront
EtcD
B+ Tree
Michael Stonebraker
Three Vs
Confluent
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast