Warehouse Native Incremental Data Processing With Dynamic Tables And Delayed View Semantics
Summary In this episode of the Data Engineering Podcast Dan Sotolongo from Snowflake talks about the complexities of incremental data processing in warehouse environments. Dan discusses the challenges of handling continuously evolving datasets and the importance of incremental data processing for optimized resource use and reduced latency. He explains how delayed view semantics can address these challenges by maintaining up-to-date results with minimal work, leveraging Snowflake's dynamic tables feature. The conversation also explores the broader landscape of data processing, comparing batch and streaming systems, and highlights the trade-offs between them. Dan emphasizes the need for a unified theoretical framework to discuss semantic guarantees in data pipelines and introduces the concept of delayed view semantics, touching on the limitations of current systems and the potential of dynamic tables to simplify complex data workflows. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details. Your host is Tobias Macey and today I'm interviewing Dan Sotolongo about the challenges of incremental data processing in warehouse environments and how delayed view semantics help to address the problem Interview Introduction How did you get involved in the area of data management? Can you start by defining the scope of the term "incremental data processing"?What are some of the common solutions that data engineers build when creating workflows to implement that pattern? What are some common difficulties that they encounter in the pursuit of incremental data? Can you describe what delayed view semantics are and the story behind it?What are the problems that DVS explicitly doesn't address? How does the approach that you have taken in Dynamic View Semantics compare to systems like Materialize, Feldera, etc. Can you describe the technical architecture of the implementation of Dynamic Tables?What are the elements of the problem that are as-yet unsolved? How has the implementation changed/evolved as you learned more about the solution space? What would be involved in implementing the delayed view semantics pattern in other dbms engines? For someone who wants to use DVS/Dyamic Tables for managing their incremental data loads, what does the workflow look like?What are the options for being able to apply tests/validation logic to a dynamic table while it is operating? What are the most interesting, innovative, or unexpected ways that you have seen Dynamic Tables used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Dynamic Tables/Delayed View Semantics? When are Dynamic Tables/DVS the wrong choice? What do you have planned for the future of Dynamic Tables? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. Links Delayed View Semantics: Presentation Slides Snowflake NumPy IPython Jupyter Flink Spark Streaming Kafka Snowflake Dynamic Tables Airflow Dagster Streaming Watermarks Materialize Feldera ACID CAP Theorem) Linearizability Serializable Consistency SIGMOD Materialized Views dbt Data Vault Apache Iceberg Databricks Delta Hudi Dead Letter Queue pg_ivm Property Based Testing Iceberg V3 Row Lineage Prometheus The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA