#247 Barr Moses: Why Reliable Data is Key to Building Good AI Systems
This episode is sponsored by Netsuite by Oracle, the number one cloud financial system, streamlining accounting, financial management, inventory, HR, and more.
NetSuite is offering a one-of-a-kind flexible financing program. Head to https://netsuite.com/EYEONAI to know more.
In this episode of Eye on AI, Craig Smith sits down with Barr Moses, Co-Founder & CEO of Monte Carlo, the pioneer of data and AI observability. Together, they explore the hidden force behind every great AI system: reliable, trustworthy data.
With AI adoption soaring across industries, companies now face a critical question: Can we trust the data feeding our models? Barr unpacks why data quality is more important than ever, how observability helps detect and resolve data issues, and why clean data—not access to GPT or Claude—is the real competitive moat in AI today.
What You'll Learn in This Episode:
Why access to AI models is no longer a competitive advantage
How Monte Carlo helps teams monitor complex data estates in real-time
The dangers of "data hallucinations" and how to prevent them
Real-world examples of data failures and their impact on AI outputs
The difference between data observability and explainability
Why legacy methods of data review no longer work in an AI-first world
Stay Updated:
Craig Smith on X:https://x.com/craigss
Eye on A.I. on X: https://x.com/EyeOn_AI
(00:00) Intro
(01:08) How Monte Carlo Fixed Broken Data
(03:08) What Is Data & AI Observability?
(05:00) Structured vs Unstructured Data Monitoring
(08:48) How Monte Carlo Integrates Across Data Stacks
(13:35) Why Clean Data Is the New Competitive Advantage
(16:57) How Monte Carlo Uses AI Internally
(19:20) 4 Failure Points: Data, Systems, Code, Models
(23:08) Can Observability Detect Bias in Data?
(26:15) Why Data Quality Needs a Modern Definition
(29:22) Explosion of Data Tools & Monte Carlo's 50+ Integrations
(33:18) Data Observability vs Explainability
(36:18) Human Evaluation vs Automated Monitoring
(39:23) What Monte Carlo Looks Like for Users
(46:03) How Fast Can You Deploy Monte Carlo?
(51:56) Why Manual Data Checks No Longer Work
(53:26) The Future of AI Depends on Trustworthy Data
How Generative AI Is Impacting Data Engineering Teams
Summary
Generative AI has rapidly gained adoption for numerous use cases. To support those applications, organizational data platforms need to add new features and data teams have increased responsibility. In this episode Lior Gavish, co-founder of Monte Carlo, discusses the various ways that data teams are evolving to support AI powered features and how they are incorporating AI into their work.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end-to-end data lakehouse platform built on Trino, the query engine Apache Iceberg was designed for, with complete support for all table formats including Apache Iceberg, Hive, and Delta Lake. Trusted by teams of all sizes, including Comcast and Doordash. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
Your host is Tobias Macey and today I'm interviewing Lior Gavish about the impact of AI on data engineers
Interview
Introduction
How did you get involved in the area of data management?
Can you start by clarifying what we are discussing when we say "AI"?
Previous generations of machine learning (e.g. deep learning, reinforcement learning, etc.) required new features in the data platform. What new demands is the current generation of AI introducing?
Generative AI also has the potential to be incorporated in the creation/execution of data pipelines. What are the risk/reward tradeoffs that you have seen in practice?What are the areas where LLMs have proven useful/effective in data engineering?
Vector embeddings have rapidly become a ubiquitous data format as a result of the growth in retrieval augmented generation (RAG) for AI applications. What are the end-to-end operational requirements to support this use case effectively?As with all data, the reliability and quality of the vectors will impact the viability of the AI application. What are the different failure modes/quality metrics/error conditions that they are subject to?
As much as vectors, vector databases, RAG, etc. seem exotic and new, it is all ultimately shades of the same work that we have been doing for years. What are the areas of overlap in the work required for running the current generation of AI, and what are the areas where it diverges?What new skills do data teams need to acquire to be effective in supporting AI applications?
What are the most interesting, innovative, or unexpected ways that you have seen AI impact data engineering teams?
What are the most interesting, unexpected, or challenging lessons that you have learned while working with the current generation of AI?
When is AI the wrong choice?
What are your predictions for the future impact of AI on data engineering teams?
Contact Info
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your
Links
Monte CarloPodcast Episode
NLP == Natural Language Processing
Large Language Models
Generative AI
MLOps
ML Engineer
Feature Store
Retrieval Augmented Generation (RAG)
Langchain
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Quantifying The Return On Investment For Your Data Team
Summary
As businesses increasingly invest in technology and talent focused on data engineering and analytics, they want to know whether they are benefiting. So how do you calculate the return on investment for data? In this episode Barr Moses and Anna Filippova explore that question and provide useful exercises to start answering that in your company.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack
Your host is Tobias Macey and today I'm interviewing Barr Moses and Anna Filippova about how and whether to measure the ROI of your data team
Interview
Introduction
How did you get involved in the area of data management?
What are the typical motivations for measuring and tracking the ROI for a data team?
Who is responsible for collecting that information?
How is that information used and by whom?
What are some of the downsides/risks of tracking this metric? (law of unintended consequences)
What are the inputs to the number that constitutes the "investment"? infrastructure, payroll of employees on team, time spent working with other teams?
What are the aspects of data work and its impact on the business that complicate a calculation of the "return" that is generated?
How should teams think about measuring data team ROI?
What are some concrete ROI metrics data teams can use?
What level of detail is useful? What dimensions should be used for segmenting the calculations?
How can visibility into this ROI metric be best used to inform the priorities and project scopes of the team?
With so many tools in the modern data stack today, what is the role of technology in helping drive or measure this impact?
How do your respective solutions, Monte Carlo and dbt, help teams measure and scale data value?
With generative AI on the upswing of the hype cycle, what are the impacts that you see it having on data teams?
What are the unrealistic expectations that it will produce?
How can it speed up time to delivery?
What are the most interesting, innovative, or unexpected ways that you have seen data team ROI calculated and/or used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on measuring the ROI of data teams?
When is measuring ROI the wrong choice?
Contact Info
Barr
LinkedIn
Anna
LinkedIn
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
Monte Carlo
Podcast Episode
dbt
Podcast Episode
JetBlue Snowflake Con Presentation
Generative AI
Large Language Models
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Sponsored By:
Rudderstack: 
Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at [dataengineeringpodcast.com/rudderstack](https://www.dataengineeringpodcast.com/rudderstack)
Support Data Engineering Podcast
20 Sales: The Ultimate Guide to Sales Onboarding for New Sales Reps and Sales Leaders, The Biggest Red Flags in the First 30 Days & What Can Be Done to Set Them Up for Success with Leaders from Figma, Dropbox, Miro and more.
Oliver Jay (OJ) is one of the most successful sales leaders of the last decade. Most recently, OJ spent 6 years as CRO @ Asana and grew the team from 20 to over 450, becoming a master in sales onboarding in the process. Before Asana, OJ spent 4 years at Dropbox, where OJ was Head of APAC & LATAM.
Jordan Van Horn is a Revenue Leader @ Monte Carlo. Prior to this role, Jordan spent 4 years at Segment as VP of Sales, leading a sales team of 50+ Account Executives. Before Segment, Jordan was at Dropbox for 4 years leading enterprise sales for Dropbox Business.
Dannie Herzberg is a Partner @ Sequoia Capital and before Sequoia, Dannie spent 4 years at Slack as their Head of Enterprise Sales, growing the business from $100M – $1B in revenue. Before Slack, Dannie spent over 5 years at Hubspot building sales teams.
Zhenya Loginov is the CRO @ Miro. At Miro, Zhenya runs the go-to-market team of 700+ people across. Prior to Miro, Zhenya was the COO @ Segment where he built and ran the global go-to-market team of 200+ people. Pre-Segment, Zhenya led a 100-person team at Dropbox across different functional areas.
Lauren Schwartz is VP of Enterprise Sales at Fivetran, where she has scaled the team to nearly 100, while more than tripling enterprise revenues. Previously, Lauren spent close to 4 years at Segment where she started as the first female AE and ultimately built and led sales teams in enterprise and growth.
Kyle Parrish is VP Sales @ Figma, where he built the sales engine from scratch to today, with over 100 in sales. Before Figma, Kyle spent over 5 years at Dropbox in numerous different roles including Head of Sales, and Global Partnerships lead, responsible for growing Dropbox's partner ecosystem.
In Todays Episode on Sales Onboarding We Discuss:
1.) How should the onboarding process for all new sales reps be structured?
2.) How should the onboarding process for a new sales leader be structured?
3.) What specific things can leaders do to set both reps and leaders best up for success?
4.) What are the single biggest red flags in the first 30 days that a rep is not going to work out?
5.) What tools and software can be used to improve this process of ramping new reps?
A Reflection On Data Observability As It Reaches Broader Adoption
Summary
Data observability is a product category that has seen massive growth and adoption in recent years. Monte Carlo is in the vanguard of companies who have been enabling data teams to observe and understand their complex data systems. In this episode founders Barr Moses and Lior Gavish rejoin the show to reflect on the evolution and adoption of data observability technologies and the capabilities that are being introduced as the broader ecosystem adopts the practices.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their new managed database service you can launch a production ready MySQL, Postgres, or MongoDB cluster in minutes, with automated backups, 40 Gbps connections from your application hosts, and high throughput SSDs. Go to dataengineeringpodcast.com/linode today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don’t forget to thank them for their continued support of this show!
Atlan is the metadata hub for your data ecosystem. Instead of locking your metadata into a new silo, unleash its transformative potential with Atlan’s active metadata capabilities. Push information about data freshness and quality to your business intelligence, automatically scale up and down your warehouse based on usage patterns, and let the bots answer those questions in Slack so that the humans can focus on delivering real value. Go to dataengineeringpodcast.com/atlan today to learn more about how Atlan’s active metadata platform is helping pioneering data teams like Postman, Plaid, WeWork & Unilever achieve extraordinary things with metadata and escape the chaos.
RudderStack helps you build a customer data platform on your warehouse or data lake. Instead of trapping data in a black box, they enable you to easily collect customer data from the entire stack and build an identity graph on your warehouse, giving you full visibility and control. Their SDKs make event streaming from any app or website easy, and their state-of-the-art reverse ETL pipelines enable you to send enriched data to any cloud tool. Sign up free… or just get the free t-shirt for being a listener of the Data Engineering Podcast at dataengineeringpodcast.com/rudder.
The only thing worse than having bad data is not knowing that you have it. With Bigeye’s data observability platform, if there is an issue with your data or data pipelines you’ll know right away and can get it fixed before the business is impacted. Bigeye let’s data teams measure, improve, and communicate the quality of your data to company stakeholders. With complete API access, a user-friendly interface, and automated yet flexible alerting, you’ve got everything you need to establish and maintain trust in your data. Go to dataengineeringpodcast.com/bigeye today to sign up and start trusting your analyses.
Your host is Tobias Macey and today I’m interviewing Barr Moses and Lior Gavish about the state of the market for data observability and their own work at Monte Carlo
Interview
Introduction
How did you get involved in the area of data management?
Can you give the elevator pitch for Monte Carlo?
What are the notable changes in the Monte Carlo product and business since our last conversation in October 2020?
You were one of the early entrants in the market of data quality/data observability products. In your work to gain visibility and traction you invested substantially in content creation (blog posts, presentations, round table conversations, etc.). How would you summarize the focus of your initial efforts?
Why do you think data observability has really taken off? A few years ago, the category barely existed – what’s changed?
There’s a larger debate within the data engineering community regarding whether it makes sense to go deep or go broad when it comes to monitoring your data. In other words, do you start with a few important data sets, or do you attempt to cover the entire ecosystem. What is your take?
For engineers and teams who are just now investigating and investing in observability/quality automation for their data, what are their motivations?
How has the conversation around the value/motivating factors matured or changed over the past couple of years?
In what way have the requirements and capabilities of data observability platforms shifted?
What are the forces in the ecosystem that have driven those changes?
How has the scope and vision for your work at Monte Carlo evolved as the understanding and impact of data quality have become more widespread?
When teams invest in data quality/observability what are some of the ways that the insights gained influence their other priorities and design choices? (e.g. platform design, pipeline design, data usage, etc.)
When it comes to selecting what parts of the data stack to invest in, how do data leaders prioritize? For instance, when does it make sense to build or buy a data catalog? A data observability platform?
The adoption of any tool that adds constraints is a delicate balance. What have you found to be the predominant patterns for teams who are incorporating Monte Carlo? (e.g. maintaining delivery velocity and adding safety/trust)
A corollary to the goal of data engineers for higher reliability and visibility is the need by the business/team leadership to identify "return on investment". How do you and your customers think about the useful metrics and measurement goals to justify the time spent on "non-functional" requirements?
What are the most interesting, innovative, or unexpected ways that you have seen Monte Carlo used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on Monte Carlo?
When is Monte Carlo the wrong choice?
What do you have planned for the future of Monte Carlo?
Contact Info
Barr
LinkedIn
@BM_DataDowntime on Twitter
Lior
LinkedIn
@lgavish on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast helps you go from idea to production with machine learning.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on Apple Podcasts and tell your friends and co-workers
Links
Monte Carlo
Podcast Episode
App Dynamics
Datadog
New Relic
Data Quality Fundamentals book
State Of Data Quality Survey
dbt
Podcast Episode
Airflow
Dagster
Podcast Episode
Episode: Incident Management For Data Teams
Databricks Delta
Patch.tech Snowflake APIs
Hightouch
Podcast Episode
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
20 Sales: Three Reasons Why Sales People Fail | The Two Things That Matter When Hiring Sales Leaders | Why Revenue, Discounting and Price Do Not Matter in the Early Days with Jordan Van Horn, Revenue Leader @ Monte Carlo
Jordan Van Horn is a Revenue Leader @ Monte Carlo, the world's first data observability company. Prior to this role, Jordan spent an incredible 4 years in sales at Segment including as VP of Sales leading a sales team of 50+ Account Executives and leading the first international expansion for the company into Dublin. Before Segment, Jordan was at Dropbox for 4 years leading enterprise sales for Dropbox Business in California.
In Today's Episode with Jordan Van Horn We Discuss:
1.) Entry into the World of Sales:
How did Jordan make his way into the world of sales first with a vineyard?
What are 1-2 of the biggest takeaways for Jordan from seeing the scaling sales teams at both Segment and Dropbox? How did seeing that impact his mindset?
What does Jordan know now that he wishes he had known when he entered sales?
2.) The Sales Playbook:
How does Jordan define "the sales playbook"? What is it not?
What five core things should the sales playbook help you accomplish?
Should the founder be responsible for the sales playbook? Can it be created by a Head of Sales?
How does Jordan advise founders on three signals that now is the right time to bring in a sales hire?
How does Jordan advise founders on whether the first sales hire should be a rep or a leader?
3.) The Secrets to Pricing and Discounting:
Why does Jordan not care what price customers pay in the early days? If it is not about ARR, what should teams be optimizing for?
When does price discipline become important in a company journey? What are the dangers of not having price discipline?
What two tools do sales leaders have to use in order to create urgency in a deal closing process?
How should sales leaders think about building multiple champions within a potential customer? At what price point is it worth it?
4.) The Hiring Process:
How does Jordan structure the hiring process for all new sales hires?
What are the must-ask questions that Jordan asks all new candidates? What does he want to see in those answers?
Who else does he bring into the hiring process? At what stage do they get involved? What are they testing for?
Does Jordan use case studies with candidates? What makes the best? What makes the worst?
5.) The Onboarding:
What is the ideal onboarding process for new sales reps?
What should founders do and prep for when onboarding their first sales hires? What materials and recordings should they have ready?
What are some early signs that a new hire is not working out? How do we measure their impact?
For enterprise sales, it takes a long time to close new deals, how can one determine effectiveness of new reps when the sales cycle is so long?
20VC: The Memo: What is a Sales Playbook? Does the Founder Need to Create It? Should the First Sales Hire Be a Leader or a Rep?
Today we deconstruct the canonical question in early-stage sales. Does the founder need to create the sales playbook? Then secondly, if not, should the first sales hires be reps or a sales leader? Today we are joined by 7 of the best sales leaders to share their thoughts.
Jordan Van Horn is a Revenue Leader @ Montecarlo. Previously Jordan spent 4 years with Segment and before that spent another 4 years at Dropbox.
Oliver Jay (OJ) most recently spent 6 years at Asana where he was hired as the company's first revenue leader. Before Asana, OJ spent 4 years at Dropbox where he scaled the sales team from 0 to 50 while tripling ARR.
Dannie Herzberg is a Partner @ Sequoia Capital and previously spent 4 years at Slack as their Head of Enterprise Sales. Before Slack, Dannie spent 5 years at Hubspot building sales, opening an SF office, and then joining product to launch CRM & platform.
Zhenya Loginov is the CRO @ Miro, where he runs the go-to-market team of 700+ people across 11 global offices. Prior to Miro, Zhenya was the COO @ Segment. Finally, before Segment, Zhenya led a 100-person team at Dropbox across numerous different functional areas.
Kyle Parrish is VP Sales @ Figma, where he has scaled the sales team from 0 to over 100 people in sales. Before Figma, Kyle spent over 5 years at Dropbox in numerous different roles including Head of Sales, where he scaled the Austin, Texas office from 3 to over 80 people.
Sam Taylor is the VP of Sales and Customer Success @ Loom, at Loom Sam leads Revenue Org including: Direct Sales, Customer Success, Self-Serve Revenue Growth/Assist. Prior to Loom, Sam spent over 4 years at Salesforce, following their acquisition of Quip, where he was the first sales leader. Before Salesforce and Quip, Sam spent over 3 years at Dropbox as a mid-market sales leader.
Jeanne DeWitt Grosser is Head of Americas Revenue & Growth @ Stripe. Pre-Stripe, Jeanne was CRO @ Dialpad and also spent many years at Google in numerous different roles including most recently as Director of GSuite SMB & Mid-Market Sales, North America and LATAM.
Mitch Tarica is Head of North America Sales at Zoom Video Communications. Before Zoom, Mitch spent over 5 years at RingCentral and before RingCentral, Mitch was at Oracle for over 7 years in numerous different sales roles.
In Today's Discussion on Sales Playbooks We Learn:
1.) What is the right definition for a "sales playbook"?
2.) When is the right time to change your "sales playbook"?
3.) What are the biggest mistakes or misnomers made around the "sales playbook"?
4.) Should the founder be the one to create the first sales playbook or can it be a sales leader?
5.) When is the right time for founders to hire their first sales leaders?
6.) For the first sales hire, should founders hire sales reps or a sales leader?
7.) When should you hire a rep vs a sales leader? What are the nuances?
Designing And Building Data Platforms As A Product
Summary
The term "data platform" gets thrown around a lot, but have you stopped to think about what it actually means for you and your organization? In this episode Lior Gavish, Lior Solomon, and Atul Gupte share their view of what it means to have a data platform, discuss their experiences building them at various companies, and provide advice on how to treat them like a software product. This is a valuable conversation about how to approach the work of selecting the tools that you use to power your data systems and considerations for how they can be woven together for a unified experience across your various stakeholders.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Are you bored with writing scripts to move data into SaaS tools like Salesforce, Marketo, or Facebook Ads? Hightouch is the easiest way to sync data into the platforms that your business teams rely on. The data you’re looking for is already in your data warehouse and BI tools. Connect your warehouse to Hightouch, paste a SQL query, and use their visual mapper to specify how data should appear in your SaaS systems. No more scripts, just SQL. Supercharge your business teams with customer data using Hightouch for Reverse ETL today. Get started for free at dataengineeringpodcast.com/hightouch.
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
Your host is Tobias Macey and today I’m interviewing Lior Gavish, Lior Solomon, and Atul Gupte about the technical, social, and architectural aspects of building your data platform as a product for your internal customers
Interview
Introduction
How did you get involved in the area of data management? – all
Can we start by establishing a definition of "data platform" for the purpose of this conversation?
Who are the stakeholders in a data platform?
Where does the responsibility lie for creating and maintaining ("owning") the platform?
What are some of the technical and organizational constraints that are likely to factor into the design and execution of the platform?
What are the minimum set of requirements necessary to qualify as a platform? (as opposed to a collection of discrete components)
What are the additional capabilities that should be in place to simplify the use and maintenance of the platform?
How are data platforms managed? Are they managed by technical teams, product managers, etc.? What is the profile for a data product manager? – Atul G.
How do you set SLIs / SLOs with your data platform team when you don’t have clear metrics you’re tracking? – Lior S.
There has been a lot of conversation recently about different interpretations of the "modern data stack". For a team who is just starting to build out their platform, how much credence should they be giving to those debates?
What are the first steps that you recommend for those practitioners?
If an organization already has infrastructure in place for data/analytics, how might they think about building or buying their way toward a well integrated platform?
Once a platform is established, what are some challenges that teams should anticipate in scaling the platform?
Which axes of scale have you found to be most difficult to manage? (scale of infrastructure capacity, scale of organizational/technical complexity, scale of usage, etc.)
Do we think the "data platform" is a skill set? How do we split up the role of the platform? Is there one for real-time? Is there one for ETLs?
How do you handle the quality and reliability of the data powering your solution?
What are helpful techniques that you have used for collecting, prioritizing, and managing feature requests?
How do you justify the budget and resources for your data platform?
How do you measure the success of a data platform?
What is the relationship between a data platform and data products?
Are there any other companies you admire when it comes to building robust, scalable data architecture?
What are the most interesting, innovative, or unexpected ways that you have seen data platforms used?
What are the most interesting, unexpected, or challenging lessons that you have learned while building and operating a data platform?
When is a data platform the wrong choice? (as opposed to buying an integrated solution, etc.)
What are the industry trends that you are monitoring/excited for in the space of data platforms?
Contact Info
Lior Gavish
LinkedIn
@lgavish on Twitter
Lior Solomon
LinkedIn
@liorsolomon on Twitter
Atul Gupte
LinkedIn
@atulgupte on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Monte Carlo
Vimeo
Facebook
Uber
Zynga
Great Expectations
Podcast Episode
Airflow
Podcast.__init__ Episode
Fivetran
Podcast Episode
dbt
Podcast Episode
Snowflake
Podcast Episode
Looker
Podcast Episode
Modern Data Stack Podcast Episode
Stitch
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
499: Data Meshes and Data Reliability
Barr Moses joins us to discuss the importance of data reliability for pipelines and how companies can achieve data mesh.
In this episode you will learn:
Data meshes [4:25]
Self-serve data reliability [15:36]
How Monte Carlo helps data up time [21:13]
How to build an effective data science team [26:50]
LinkedIn Q&A [31:50]
Additional materials: www.superdatascience.com/499
The Grand Vision And Present Reality of DataOps
Summary
The Data industry is changing rapidly, and one of the most active areas of growth is automation of data workflows. Taking cues from the DevOps movement of the past decade data professionals are orienting around the concept of DataOps. More than just a collection of tools, there are a number of organizational and conceptual changes that a proper DataOps approach depends on. In this episode Kevin Stumpf, CTO of Tecton, Maxime Beauchemin, CEO of Preset, and Lior Gavish, CTO of Monte Carlo, discuss the grand vision and present realities of DataOps. They explain how to think about your data systems in a holistic and maintainable fashion, the security challenges that threaten to derail your efforts, and the power of using metadata as the foundation of everything that you do. If you are wondering how to get control of your data platforms and bring all of your stakeholders onto the same page then this conversation is for you.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.
RudderStack’s smart customer data pipeline is warehouse-first. It builds your customer data warehouse and your identity graph on your data warehouse, with support for Snowflake, Google BigQuery, Amazon Redshift, and more. Their SDKs and plugins make event streaming easy, and their integrations with cloud applications like Salesforce and ZenDesk help you go beyond event streaming. With RudderStack you can use all of your customer data to answer more difficult questions and then send those insights to your whole customer data stack. Sign up free at dataengineeringpodcast.com/rudder today.
Your host is Tobias Macey and today I’m interviewing Max Beauchemin, Lior Gavish, and Kevin Stumpf about the real world challenges of embracing DataOps practices and systems, and how to keep things secure as you scale
Interview
Introduction
How did you get involved in the area of data management?
Before we get started, can you each give your definition of what "DataOps" means to you?
How does this differ from "business as usual" in the data industry?
What are some of the things that DataOps isn’t (despite what marketers might say)?
What are the biggest difficulties that you have faced in going from concept to production with a workflow or system intended to power self-serve access to other members of the organization?
What are the weak points in the current state of the industry, whether technological or social, that contribute to your greatest sense of unease from a security perspective?
As founders of companies that aim to facilitate adoption of various aspects of DataOps, how are you applying the products that you are building to your own internal systems?
How does security factor into the design of robust DataOps systems? What are some of the biggest challenges related to security when it comes to putting these systems into production?
What are the biggest differences between DevOps and DataOps, particularly when it concerns designing distributed systems?
What areas of the DataOps landscape do you think are ripe for innovation?
Nowadays, it seems like new DataOps companies are cropping up every day to try and solve some of these problems. Why do you think DataOps is becoming such an important component of the modern data stack?
There’s been a lot of conversation recently around the "rise of the data engineer" versus other roles in the data ecosystem (i.e. data scientist or data analyst). Why do you think that is?
What are some of the most valuable lessons that you have learned from working with your customers about how to apply DataOps principles?
What are some of the most interesting, unexpected, or challenging lessons that you have learned while building your respective platforms and businesses?
What are the industry trends that you are each keeping an eye on to inform you future product direction?
Contact Info
Kevin
LinkedIn
kevinstumpf on GitHub
@kevinstumpf on Twitter
Maxime
LinkedIn
@mistercrunch on Twitter
mistercrunch on GitHub
Lior
LinkedIn
@lgavish on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Links
Tecton
Monte Carlo
Superset
Preset
Barracuda Networks
Feature Store
DataOps
DevOps
Data Catalog
Amundsen
OpenLineage
The Downfall of the Data Engineer
Hashicorp Vault
Reverse ELT
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
Better Data Quality Through Observability With Monte Carlo
Summary
In order for analytics and machine learning projects to be useful, they require a high degree of data quality. To ensure that your pipelines are healthy you need a way to make them observable. In this episode Barr Moses and Lior Gavish, co-founders of Monte Carlo, share the leading causes of what they refer to as data downtime and how it manifests. They also discuss methods for gaining visibility into the flow of data through your infrastructure, how to diagnose and prevent potential problems, and what they are building at Monte Carlo to help you maintain your data’s uptime.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. Go to dataengineeringpodcast.com/97things to add your voice and share your hard-earned expertise.
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $60 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Are you bogged down by having to manually manage data access controls, repeatedly move and copy data, and create audit reports to prove compliance? How much time could you save if those tasks were automated across your cloud platforms? Immuta is an automated data governance solution that enables safe and easy data analytics in the cloud. Our comprehensive data-level security, auditing and de-identification features eliminate the need for time-consuming manual processes and our focus on data and compliance team collaboration empowers you to deliver quick and valuable data analytics on the most sensitive data to unlock the full potential of your cloud data platforms. Learn how we streamline and accelerate manual processes to help you derive real results from your data at dataengineeringpodcast.com/immuta.
Today’s episode of the Data Engineering Podcast is sponsored by Datadog, a SaaS-based monitoring and analytics platform for cloud-scale infrastructure, applications, logs, and more. Datadog uses machine-learning based algorithms to detect errors and anomalies across your entire stack—which reduces the time it takes to detect and address outages and helps promote collaboration between Data Engineering, Operations, and the rest of the company. Go to dataengineeringpodcast.com/datadog today to start your free 14 day trial. If you start a trial and install Datadog’s agent, Datadog will send you a free T-shirt.
You listen to this show to learn and stay up to date with what’s happening in databases, streaming platforms, big data, and everything else you need to know about modern data platforms. For more opportunities to stay up to date, gain new skills, and learn from your peers there are a growing number of virtual events that you can attend from the comfort and safety of your home. Go to dataengineeringpodcast.com/conferences to check out the upcoming events being offered by our partners and get registered today!
Your host is Tobias Macey and today I’m interviewing Barr Moses and Lior Gavish about observability for your data pipelines and how they are addressing it at Monte Carlo.
Interview
Introduction
How did you get involved in the area of data management?
How did you come up with the idea to found Monte Carlo?
What is "data downtime"?
Can you start by giving your definition of observability in the context of data workflows?
What are some of the contributing factors that lead to poor data quality at the different stages of the lifecycle?
Monitoring and observability of infrastructure and software applications is a well understood problem. In what ways does observability of data applications differ from "traditional" software systems?
What are some of the metrics or signals that we should be looking at to identify problems in our data applications?
Why is this the year that so many companies are working to address the issue of data quality and observability?
How are you addressing the challenge of bringing observability to data platforms at Monte Carlo?
What are the areas of integration that you are targeting and how did you identify where to prioritize your efforts?
For someone who is using Monte Carlo, how does the platform help them to identify and resolve issues in their data?
What stage of the data lifecycle have you found to be the biggest contributor to downtime and quality issues?
What are the most challenging systems, platforms, or tool chains to gain visibility into?
What are some of the most interesting, innovative, or unexpected ways that you have seen teams address their observability needs?
What are the most interesting, unexpected, or challenging lessons that you have learned while building the business and technology of Monte Carlo?
What are the alternatives to Monte Carlo?
What do you have planned for the future of the platform?
Contact Info
Visit www.montecarlodata.com?utm_source=rss&utm_medium=rss to lean more about our data reliability platform;
Or reach out directly to barr@montecarlodata.com — happy to chat about all things data!
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Links
Monte Carlo
Monte Carlo Platform
Observability
Gainsight
Barracuda Networks
DevOps
New Relic
Datadog
Netflix RAD Outlier Detection
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast