Reduce Friction In Your Business Analytics Through Entity Centric Data Modeling
Summary
For business analytics the way that you model the data in your warehouse has a lasting impact on what types of questions can be answered quickly and easily. The major strategies in use today were created decades ago when the software and hardware for warehouse databases were far more constrained. In this episode Maxime Beauchemin of Airflow and Superset fame shares his vision for the entity-centric data model and how you can incorporate it into your own warehouse design.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack
Your host is Tobias Macey and today I'm interviewing Max Beauchemin about the concept of entity-centric data modeling for analytical use cases
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what entity-centric modeling (ECM) is and the story behind it?
How does it compare to dimensional modeling strategies?
What are some of the other competing methods
Comparison to activity schema
What impact does this have on ML teams? (e.g. feature engineering)
What role does the tooling of a team have in the ways that they end up thinking about modeling? (e.g. dbt vs. informatica vs. ETL scripts, etc.)
What is the impact on the underlying compute engine on the modeling strategies used?
What are some examples of data sources or problem domains for which this approach is well suited?
What are some cases where entity centric modeling techniques might be counterproductive?
What are the ways that the benefits of ECM manifest in use cases that are down-stream from the warehouse?
What are some concrete tactical steps that teams should be thinking about to implement a workable domain model using entity-centric principles?
How does this work across business domains within a given organization (especially at "enterprise" scale)?
What are the most interesting, innovative, or unexpected ways that you have seen ECM used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on ECM?
When is ECM the wrong choice?
What are your predictions for the future direction/adoption of ECM or other modeling techniques?
Contact Info
mistercrunch on GitHub
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
Entity Centric Modeling Blog Post
Max's Previous Apperances
Defining Data Engineering with Maxime Beauchemin
Self Service Data Exploration And Dashboarding With Superset
Exploring The Evolving Role Of Data Engineers
Alumni Of AirBnB's Early Years Reflect On What They Learned About Building Data Driven Organizations
Apache Airflow
Apache Superset
Preset
Ubisoft
Ralph Kimball
The Rise Of The Data Engineer
The Downfall Of The Data Engineer
The Rise Of The Data Scientist
Dimensional Data Modeling
Star Schema
Database Normalization
Feature Engineering
DRY == Don't Repeat Yourself
Activity Schema
Podcast Episode
Corporate Information Factory (affiliate link)
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
Rudderstack: 
Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)
Support Data Engineering Podcast
Exploring The Evolving Role Of Data Engineers
Summary
Data Engineering is still a relatively new field that is going through a continued evolution as new technologies are introduced and new requirements are understood. In this episode Maxime Beauchemin returns to revisit what it means to be a data engineer and how the role has changed over the past 5 years.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Struggling with broken pipelines? Stale dashboards? Missing data? If this resonates with you, you’re not alone. Data engineers struggling with unreliable data need look no further than Monte Carlo, the world’s first end-to-end, fully automated Data Observability Platform! In the same way that application performance monitoring ensures reliable software and keeps application downtime at bay, Monte Carlo solves the costly problem of broken data pipelines. Monte Carlo monitors and alerts for data issues across your data warehouses, data lakes, ETL, and business intelligence, reducing time to detection and resolution from weeks or days to just minutes. Start trusting your data with Monte Carlo today! Visit dataengineeringpodcast.com/montecarlo to learn more. The first 10 people to request a personalized product tour will receive an exclusive Monte Carlo Swag box.
Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
Your host is Tobias Macey and today I’m interviewing Maxime Beauchemin about the impacts that the evolution of the modern data stack has had on the role and responsibilities of data engineers
Interview
Introduction
How did you get involved in the area of data management?
What is your current working definition of a data engineer?
How has that definition changed since your article on the "rise of the data engineer" and episode 3 of this show about "defining data engineering"?
How has the growing availability of data infrastructure services shifted foundational skills and knowledge that are necessary to be effective?
How should a new/aspiring data engineer focus their time and energy to become effective?
One of the core themes in this current spate of technologies is "democratization of data". In your post on the downfall of the data engineer you called out the pressure on data engineers to maintain control with so many contributors with varying levels of skill and understanding. How well is the "modern data stack" balancing these concerns?
An interesting impact of the growing usage of data is the constrained availability of data engineers. How do you see the effects of the job market on driving evolution of tooling and services?
With the explosion of tools and services for working with data, a new problem has evolved of which ones to use for a given organization. What do you see as an effective and efficient process for enumerating and evaluating the available components for building a stack?
There is also a lot of conversation around the "modern data stack", as well as the need for companies to build a "data platform". What (if any) difference do you see in the implications of those phrases and the skills required to compile a stack vs build a platform?
How do you view the long term viability of templated SQL as a core workflow for transformations?
What is the impact of more acessible and widespread machine learning/deep learning on data engineers/data infrastructure?
How evenly distributed across industries and geographies are the advances in data infrastructure and engineering practices?
What are some of the opportunities that are being missed or squandered during this dramatic shift in the data engineering landscape?
What are the most interesting, innovative, or unexpected ways that you have seen the data ecosytem evolve?
What are the most interesting, unexpected, or challenging lessons that you have learned while contributing to and participating in the data ecosystem?
In episode 3 of this show (almost five years ago) we closed with some predictions for the following years of data engineering, many of which have been proven out. What is your retrospective on those claims, and what are your new predictions for the upcoming years?
Contact Info
LinkedIn
@mistercrunch on Twitter
mistercrunch on GitHub
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
How the Modern Data Stack is Reshaping Data Engineering
The Rise of the Data Engineer
The Downfall of the Data Engineer
Defining Data Engineering – Data Engineering Podcast
Airflow
Superset
Podcast Episode
Preset
Fivetran
Podcast Episode
Meltano
Podcast Episode
Airbyte
Podcast Episode
Ralph Kimball
Bill Inmon
Feature Store
Prophecy.io
Podcast Episode
Ab Initio
Dremio
Podcast Episode
Data Mesh
Podcast Episode
Firebolt
Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
The Grand Vision And Present Reality of DataOps
Summary
The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. More than just a collection of tools, there are a number of organizational and conceptual changes that a proper DataOps approach depends on. In this episode Kevin Stumpf, CTO of Tecton, Maxime Beauchemin, CEO of Preset, and Lior Gavish, CTO of Monte Carlo, discuss the grand vision and present realities of DataOps. They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. If you are wondering how to get control of your data platforms and bring all of your stakeholders onto the same page then this conversation is for you.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.
RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today.
Your host is Tobias Macey and today I’m interviewing Max Beauchemin, Lior Gavish, and Kevin Stumpf about the real world challenges of embracing DataOps practices and systems, and how to keep things secure as you scale
Interview
Introduction
How did you get involved in the area of data management?
Before we get started, can you each give your definition of what "DataOps" means to you?
How does this differ from "business as usual" in the data industry?
What are some of the things that DataOps isn’t (despite what marketers might say)?
What are the biggest difficulties that you have faced in going from concept to production with a workflow or system intended to power self-serve access to other members of the organization?
What are the weak points in the current state of the industry, whether technological or social, that contribute to your greatest sense of unease from a security perspective?
As founders of companies that aim to facilitate adoption of various aspects of DataOps, how are you applying the products that you are building to your own internal systems?
How does security factor into the design of robust DataOps systems? What are some of the biggest challenges related to security when it comes to putting these systems into production?
What are the biggest differences between DevOps and DataOps, particularly when it concerns designing distributed systems?
What areas of the DataOps landscape do you think are ripe for innovation?
Nowadays, it seems like new DataOps companies are cropping up every day to try and solve some of these problems. Why do you think DataOps is becoming such an important component of the modern data stack?
There’s been a lot of conversation recently around the "rise of the data engineer" versus other roles in the data ecosystem (i.e. data scientist or data analyst). Why do you think that is?
What are some of the most valuable lessons that you have learned from working with your customers about how to apply DataOps principles?
What are some of the most interesting, unexpected, or challenging lessons that you have learned while building your respective platforms and businesses?
What are the industry trends that you are each keeping an eye on to inform you future product direction?
Contact Info
Kevin
LinkedIn
kevinstumpf on GitHub
@kevinstumpf on Twitter
Maxime
LinkedIn
@mistercrunch on Twitter
mistercrunch on GitHub
Lior
LinkedIn
@lgavish on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Links
Tecton
Monte Carlo
Superset
Preset
Barracuda Networks
Feature Store
DataOps
DevOps
Data Catalog
Amundsen
OpenLineage
The Downfall of the Data Engineer
Hashicorp Vault
Reverse ELT
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
Self Service Data Exploration And Dashboarding With Superset
Summary
The reason for collecting, cleaning, and organizing data is to make it usable by the organization. One of the most common and widely used methods of access is through a business intelligence dashboard. Superset is an open source option that has been gaining popularity due to its flexibility and extensible feature set. In this episode Maxime Beauchemin discusses how data engineers can use Superset to provide self service access to data and deliver analytics. He digs into how it integrates with your data stack, how you can extend it to fit your use case, and why open source systems are a good choice for your business intelligence. If you haven’t already tried out Superset then this conversation is well worth your time. Give it a listen and then take it for a test drive today.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.
RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today.
Your host is Tobias Macey and today I’m interviewing Max Beauchemin about Superset, an open source platform for data exploration, dashboards, and business intelligence
Interview
Introduction
How did you get involved in the area of data management?
Can you start by describing what Superset is?
Superset is becoming part of the reference architecture for a modern data stack. What are the factors that have contributed to its popularity over other tools such as Redash, Metabase, Looker, etc.?
Where do dashboarding and exploration tools like Superset fit in the responsibilities and workflow of a data engineer?
What are some of the challenges that Superset faces in being performant when working with large data sources?
Which data sources have you found to be the most challenging to work with?
What are some anti-patterns that users of Superset might run into when building out a dashboard?
What are some of the ways that users can surface data quality indicators (e.g. freshness, lineage, check results, etc.) in a Superset dashboard?
Another trend in analytics and dashboard tools is providing actionable insights. How can Superset support those use cases where a business user or analyst wants to perform an action based on the data that they are being shown?
How can Superset factor into a data governance strategy for the business?
What are some of the most interesting, innovative, or unexpected ways that you have seen Superset used?
dogfooding
What are the most interesting, unexpected, or challenging lessons that you have learned from working on Superset and founding Preset?
When is Superset the wrong choice?
What do you have planned for the future of Superset and Preset?
Contact Info
LinkedIn
@mistercrunch on Twitter
mistercrunch on GitHub
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Links
Superset
Podcast.__init__ Episode
Preset
ASP (Active Server Pages)
VBScript
Data Warehouse Institute
Ralph Kimball
Bill Inmon
Ubisoft
Hadoop
Tableau
Looker
Podcast Episode
The Future of Business Intelligence Is Open Source
Supercharging Apache Superset
Redash
Podcast.__init__ Episode
Metabase
Podcast Episode
The Rise Of The Data Engineer
AirBnB Data University
Python DBAPI
SQLAlchemy
Druid
SQL Common Table Expressions
SQL Window Functions
Data Warehouse Semantic Layer
Amundsen
Podcast Episode
Open Lineage
Datakin
Marquez
Podcast Episode
Apache Arrow
Podcast.__init__ Episode with Wes McKinney
Apache Parquet
DataHub
Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
Feature Stores for Accelerating AI Development - #432
In this special episode of the podcast, we're joined by Kevin Stumpf, Co-Founder and CTO of Tecton, Willem Pienaar, an engineering lead at Gojek and founder of the Feast Project, and Maxime Beauchemin, Founder & CEO of Preset, for a discussion on Feature Stores for Accelerating AI Development.
In this panel discussion, Sam and our guests explored how organizations can increase value and decrease time-to-market for machine learning using feature stores, MLOps, and open source. We also discuss the main data challenges of AI/ML, and the role of the feature store in solving those challenges.
The complete show notes for this episode can be found at twimlai.com/go/432.
Defining Data Engineering with Maxime Beauchemin - Episode 3
Summary
What exactly is data engineering? How has it evolved in recent years and where is it going? How do you get started in the field? In this episode, Maxime Beauchemin joins me to discuss these questions and more.
Transcript provided by CastSource
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers
Your host is Tobias Macey and today I’m interviewing Maxime Beauchemin
Questions
Introduction
How did you get involved in the field of data engineering?
How do you define data engineering and how has that changed in recent years?
Do you think that the DevOps movement over the past few years has had any impact on the discipline of data engineering? If so, what kinds of cross-over have you seen?
For someone who wants to get started in the field of data engineering what are some of the necessary skills?
What do you see as the biggest challenges facing data engineers currently?
At what scale does it become necessary to differentiate between someone who does data engineering vs data infrastructure and what are the differences in terms of skill set and problem domain?
How much analytical knowledge is necessary for a typical data engineer?
What are some of the most important considerations when establishing new data sources to ensure that the resulting information is of sufficient quality?
You have commented on the fact that data engineering borrows a number of elements from software engineering. Where does the concept of unit testing fit in data management and what are some of the most effective patterns for implementing that practice?
How has the work done by data engineers and managers of data infrastructure bled back into mainstream software and systems engineering in terms of tools and best practices?
How do you see the role of data engineers evolving in the next few years?
Keep In Touch
@mistercrunch on Twitter
mistercrunch on GitHub
Medium
Links
Datadog
Airflow
The Rise of the Data Engineer
Druid.io
Luigi
Apache Beam
Samza
Hive
Data Modeling
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
Introducing The Show
Preamble
Hello and welcome to the Data Engineering Podcast, the show about modern data infrastructure
Go to dataengineeringpodcast.com to subscribe to the show, sign up for the newsletter, read the show notes, and get in touch.
You can help support the show by checking out the Patreon page which is linked from the site.
To help other people find the show you can leave a review on iTunes, or Google Play Music, share it on social media, and tell your friends and co-workers.
I’m your host, Tobias Macey, and today I’m speaking with Maxime Beauchemin about what it means to be a data engineer.
Interview
Who am I
Systems administrator and software engineer, now DevOps, focus on automation
Host of Podcast.__init__
How did I get involved in data management
Why am I starting a podcast about Data Engineering
Interesting area with a lot of activity
Not currently any shows focused on data engineering
What kinds of topics do I want to cover
Data stores
Pipelines
Tooling
Automation
Monitoring
Testing
Best practices
Common challenges
Defining the role/job hunting
Relationship with data engineers/data analysts
Get in touch and subscribe
Website
Newsletter
Twitter
Email
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast