DuckDB has an answer to “bloated” database messages
DuckDB is rethinking how databases communicate with the launch of Quack 1.0, a production-ready, HTTP-based client/server protocol designed for faster, more efficient analytical workloads. In this episode of The New Stack Podcast, Guillermo Sánchez Dionis, product lead at DuckLabs, explains how Quack addresses the limitations of traditional database protocols, which often introduce unnecessary overhead and inefficient data serialization.
Unlike conventional protocols built around row-based data transmission, Quack leverages DuckDB’s columnar data representation and vectorized execution engine to deliver faster, more efficient query results. Its streaming architecture allows clients to consume results as they are generated, significantly reducing memory usage and eliminating unnecessary processing delays.
Sánchez Dionis also highlights developer-friendly capabilities, including simplified remote connections, command-line server deployment, and improved transaction throughput and concurrency. With minimal server overhead and the flexibility to scale across multiple DuckDB instances, Quack makes distributed analytics more accessible. The conversation explores how DuckLabs is advancing database connectivity while prioritizing performance, simplicity, and server-side stability in future releases.
Learn more from The New Stack around the latest developments around DuckDB:
Why MotherDuck refuses to fork DuckDB
DuckDB: Query Processing Is King
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
DuckLake 1.1 solves the absurd complexity of data lakes at AI scale
DuckLake is simplifying data lakehouse management by replacing complex metadata files with a familiar SQL-based approach. In this episode of The New Stack Podcast, Pedro Holanda, lead DuckLake developer and principal engineer at DuckLake, discusses how the open-source table format reduces complexity, improves performance, and makes large-scale data management more accessible.
With the release of DuckLake 1.1 alongside DuckDB 2.0.0 LTS, Holanda explains how storing metadata in a relational database eliminates the need to navigate countless files to locate and query data. By combining Parquet storage with SQL catalogs, DuckLake streamlines data access while addressing common challenges such as excessive small files, inefficient queries, and costly maintenance.
The conversation also explores DuckLake’s innovative inlining capabilities, which consolidate smaller datasets to improve efficiency, and its ability to support analytics without massive server infrastructure. As AWS integrates DuckDB into its broader analytics strategy, DuckLake represents an important step toward simplifying data portability, reducing operational overhead, and enabling more efficient analytics for developers and AI agents.
Learn more from The New Stack around the latest developments around DuckDB:
Why MotherDuck refuses to fork DuckDB
DuckDB: Query Processing Is King
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
#562: DuckLake: The Lakehouse That's Just SQL and Parquet
How many files does your query read before it reads any data? On some data lakes, you go through JSON and metadata files first, just to learn which Parquet files matter. DuckLake asks one SQL question instead. The metadata lives in a real database. The data stays in plain Parquet. That's the entire format.
Pedro Holanda joined DuckDB in 2018, when it was still a research prototype at CWI. He's the lead DuckLake developer. Guillermo Sanchez Dionis works on DuckLake and the new Quack protocol.
With Quack as the catalog, DuckLake handles 200 transactions a second under heavy contention. No other open table format comes close.
Episode sponsors
Six Feet Up
Talk Python Courses
Links from the show
Guests
Pedro Holanda: pedroholanda.org
Guillermo Sanchez: linkedin.com
PhD on progressive indexes: ir.cwi.nl
SQLite: www.sqlite.org
Litestream: litestream.io
boring hardware: talkpython.fm
DuckDB: duckdb.org
episode 491: talkpython.fm
Iceberg: iceberg.apache.org
manifesto: ducklake.select
DuckLake: ducklake.select
spec: ducklake.select
this diagram: blobs.talkpython.fm
Data inlining: ducklake.select
ducklake-dataframe: github.com
Polars course: training.talkpython.fm
CSV parser: duckdb.org
Zero-copy Arrow: duckdb.org
ART index: duckdb.org
async I/O: duckdb.org
v1.0: ducklake.select
Git-like branching: ducklake.select
Watch this episode on YouTube: youtube.com
Episode #562 deep-dive: talkpython.fm/562
Episode transcripts: talkpython.fm
Theme Song: Developer Rap
🥁 Served in a Flask 🎸: talkpython.fm/flasksong
---== Don't be a stranger ==---
YouTube: youtube.com/@talkpython
Bluesky: @talkpython.fm
Mastodon: @talkpython@fosstodon.org
X.com: @talkpython
Michael on Bluesky: @mkennedy.codes
Michael on Mastodon: @mkennedy@fosstodon.org
Michael on X.com: @mkennedy
Duck Lake: Simplifying the Lakehouse Ecosystem
Summary
In this episode of the Data Engineering Podcast Hannes Mühleisen and Mark Raasveldt, the creators of DuckDB, share their work on Duck Lake, a new entrant in the open lakehouse ecosystem. They discuss how Duck Lake, is focused on simplicity, flexibility, and offers a unified catalog and table format compared to other lakehouse formats like Iceberg and Delta. Hannes and Mark share insights into how Duck Lake revolutionizes data architecture by enabling local-first data processing, simplifying deployment of lakehouse solutions, and offering benefits such as encryption features, data inlining, and integration with existing ecosystems.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data teams everywhere face the same problem: they're forcing ML models, streaming data, and real-time processing through orchestration tools built for simple ETL. The result? Inflexible infrastructure that can't adapt to different workloads. That's why Cash App and Cisco rely on Prefect. Cash App's fraud detection team got what they needed - flexible compute options, isolated environments for custom packages, and seamless data exchange between workflows. Each model runs on the right infrastructure, whether that's high-memory machines or distributed compute. Orchestration is the foundation that determines whether your data team ships or struggles. ETL, ML model training, AI Engineering, Streaming - Prefect runs it all from ingestion to activation in one platform. Whoop and 1Password also trust Prefect for their data operations. If these industry leaders use Prefect for critical workflows, see what it can do for you at dataengineeringpodcast.com/prefect.
Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details.
Your host is Tobias Macey and today I'm interviewing Hannes Mühleisen and Mark Raasveldt about DuckLake, the latest entrant into the open lakehouse ecosystem
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what DuckLake is and the story behind it?What are the particular problems that DuckLake is solving for?
How does this compare to the capabilities of MotherDuck?
Iceberg and Delta already have a well established ecosystem, but so does DuckDB. Who are the primary personas that you are trying to focus on in these early days of DuckLake?
One of the major factors driving the adoption of formats like Iceberg is cost efficiency for large volumes of data. That brings with it challenges of large batch processing of data. How does DuckLake account for these axes of scale?
There is also a substantial investment in the ecosystem of technologies that support Iceberg. The most notable ecosystem challenge for DuckDB and DuckLake is in the query layer. How are you thinking about the evolution and growth of that capability beyond DuckDB (e.g. support in Trino/Spark/Flink)?
What are your opinions on the viability of a future where DuckLake and Iceberg become a unified standard and implementation? (why can't Iceberg REST catalog implementations just use DuckLake under the hood?)
Digging into the specifics of the specification and implementation, what are some of the capabilities that it offers above and beyond Iceberg?Is it now possible to enforce PK/FK constraints, indexing on underlying data?
Given that DuckDB has a vector type, how do you think about the support for vector storage/indexing?
How do the capabilities of DuckLake and the integration with DuckDB change the ways that data teams design their data architecture and access patterns?
What are your thoughts on the impact of "data gravity" in today's data ecosystem, with engines like DuckDB, KuzuDB, LanceDB, etc. available for embedded and edge use cases?
What are the most interesting, innovative, or unexpected ways that you have seen DuckLake used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on DuckLake?
When is DuckLake the wrong choice?
What do you have planned for the future of DuckLake?
Contact Info
HannesWebsite
MarkWebsite
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story.
Links
DuckDBPodcast Episode
DuckLake
DuckDB Labs
MySQL
CWI
MonetDB
Iceberg
Iceberg REST Catalog
Delta
Hudi
Lance
DuckDB Iceberg Connector
ACID == Atomicity, Consistency, Isolation, Durability
MotherDuck
MotherDuck Managed DuckLake
Trino
Spark
Presto
Spark DuckLake Demo
Delta Kernel
Arrow
dlt
S3 Tables
Attribute Based Access Control (ABAC)
Parquet
Arrow Flight
Hadoop
HDFS
DuckLake Roadmap
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Move Your Database To The Data And Speed Up Your Analytics With DuckDB
Summary
When you think about selecting a database engine for your project you typically consider options focused on serving multiple concurrent users. Sometimes what you really need is an embedded database that is blazing fast for single user workloads. DuckDB is an in-process database engine optimized for OLAP applications to speed up your analytical queries that meets you where you are, whether that’s Python, R, Java, even the web. In this episode, Hannes Mühleisen, co-creator and CEO of DuckDB Labs, shares the motivations for creating the project, the myriad ways that it can be used to speed up your data projects, and the detailed engineering efforts that go into making it adaptable to any environment. This is a fascinating and humorous exploration of a truly useful piece of technology.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
Your host is Tobias Macey and today I’m interviewing Hannes Mühleisen about DuckDB, an in-process embedded database engine for columnar analytics
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what DuckDB is and the story behind it?
Where did the name come from?
What are some of the use cases that DuckDB is designed to support?
The interface for DuckDB is similar (at least in spirit) to SQLite. What are the deciding factors for when to use one vs. the other?
How might they be used in concert to take advantage of their relative strengths?
What are some of the ways that DuckDB can be used to better effect than options provided by different language ecosystems?
Can you describe how DuckDB is implemented?
How has the design and goals of the project changed or evolved since you began working on it?
What are some of the optimizations that you have had to make in order to support performant access to data that exceeds available memory?
Can you describe a typical workflow of incorporating DuckDB into an analytical project?
What are some of the libraries/tools/systems that DuckDB might replace in the scope of a project or team?
What are some of the overlooked/misunderstood/under-utilized features of DuckDB that you would like to highlight?
What is the governance model and plan long-term sustainability of the project?
What are the most interesting, innovative, or unexpected ways that you have seen DuckDB used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on DuckDB?
When is DuckDB the wrong choice?
What do you have planned for the future of DuckDB?
Contact Info
Hannes Mühleisen
@hfmuehleisen on Twitter
Website
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
DuckDB
CWI
SQLite
OLAP == Online Analytical Processing
Duck Typing
ZODB
Teradata
HTAP == Hybrid Transactional/Analytical Processing
Pandas
Podcast.__init__ Episode
Apache Arrow
Julia Language
Voltron Data
Parquet
Thrift
Protobuf
Vectorized Query Processor
LLVM
DuckDB Labs
DuckDB Foundation
MIT Open Courseware (OCW)
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast