Focusing on Data Science & Less on Engineering and Dependencies
<p>How do you manage the dependencies of a large-scale data science project? How do you migrate that project from a laptop to cloud infrastructure or utilize GPUs and multiple instances in parallel? This week on the show, Savin Goyal returns to discuss the updates to the open-source framework Metaflow.</p>
<p>Savin briefly describes the Metaflow platform and the goal of simplifying engineering overhead for data scientists and programmers. We discuss how the platform captures snapshots of a project as you work, allowing you to go back in time or share the state of your project with another team member.</p>
<p>We dig into the complicated process of managing dependencies for machine learning and data science projects. Savin describes how the required external libraries can be specified within a flow with the new <code>@pypi</code> or <code>@conda</code> decorators. This allows a project to scale from a local machine to the cloud or multiple instances with all dependencies included.</p>
<p>He talks about starting a new company, Outerbounds, with fellow co-workers from Netflix. Their vision is to continue to build the Metaflow open-source platform and offer customers scalable enterprise-grade infrastructure. </p>
<p>This week’s episode is brought to you by Intel.</p>
<div class="alert alert-primary" role="alert">
<p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/packaging-with-pyproject-toml/">Everyday Project Packaging With <code>pyproject.toml</code></a> </p>
<p>In this Code Conversation video course, you’ll learn how to package your everyday projects with <code>pyproject.toml</code>. Playing on the same team as the import system means you can call your project from anywhere, ensure consistent imports, and have one file that’ll work for many build systems.</p>
</div>
<p>Topics:</p>
<ul>
<li>00:00:00 – Introduction </li>
<li>00:02:25 – Update on Metaflow </li>
<li>00:04:13 – What is Outerbounds? </li>
<li>00:07:26 – An ML platform to serve data scientists needs </li>
<li>00:13:02 – Dependency reproducibility via <code>@conda</code> and <code>@pypi</code> decorators</li>
<li>00:26:18 – Sponsor: Intel </li>
<li>00:27:10 – Storing lock files along with snapshots </li>
<li>00:29:17 – Working alongside code and dependency management systems</li>
<li>00:34:03 – Scaling a project from laptop to the cloud </li>
<li>00:40:13 – Video Course Spotlight </li>
<li>00:41:41 – Getting visibility on processes </li>
<li>00:47:23 – Adjusting your project due to GPU availability </li>
<li>00:52:27 – Example of jumping back into a project one year later </li>
<li>00:55:54 – What are you excited about in the world of Python? </li>
<li>00:57:39 – What do you want to learn next? </li>
<li>00:59:35 – How can people follow your work online? </li>
<li>01:00:19 – Thanks and goodbye </li>
</ul>
<p>Show Links:</p>
<ul>
<li><a href="https://metaflow.org/">Metaflow - a framework for real-life ML, AI, and data science</a></li>
<li><a href="https://outerbounds.com/">Infrastructure for ML, AI, and Data Science - Outerbounds</a></li>
<li><a href="https://www.youtube.com/watch?v=KGpg8jwAda4">Human-Friendly, Production-Ready Data Science with Metaflow- Savin Goyal | SciPy 2022 - YouTube</a></li>
<li><a href="https://realpython.com/podcasts/rpp/61/">Episode #61: Scaling Data Science and Machine Learning Infrastructure Like Netflix – The Real Python Podcast</a></li>
<li><a href="https://outerbounds.com/blog/pypi-announcement/">New in Metaflow: The Long-Awaited <code>@pypi</code> Decorator - Outerbounds</a></li>
<li><a href="https://docs.metaflow.org/scaling/dependencies">Managing Dependencies - Metaflow Docs</a></li>
<li><a href="https://outerbounds.com/blog/secure-ml-secure-software-dependencies/">Secure ML with Secure Software Dependencies - Outerbounds</a></li>
<li><a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed acyclic graph (DAG) - Wikipedia article</a></li>
<li><a href="https://docs.metaflow.org/metaflow/visualizing-results">Visualizing Results - Metaflow Docs</a></li>
<li><a href="https://outerbounds.com/blog/seamless-data-pipelines-airflow-metaflow/">Seamless Data and ML Pipelines with Airflow and Metaflow - Outerbounds</a></li>
<li><a href="https://realpython.com/podcasts/rpp/142/">Episode #142: Orchestrating Large and Small Projects With Apache Airflow – The Real Python Podcast</a></li>
<li><a href="https://twitter.com/SavinGoyal">Savin (@SavinGoyal) - X</a></li>
<li><a href="https://www.linkedin.com/in/savingoyal/">Savin Goyal - LinkedIn</a></li>
<li><a href="https://outerbounds.com/blog/">Building the ML-driven future - Outerbounds Blog</a></li>
</ul>
<p>Level up your Python skills with our expert-led courses:</p>
<ul>
<li><a href="https://realpython.com/courses/packaging-with-pyproject-toml/">Everyday Project Packaging With pyproject.toml</a></li>
<li><a href="https://realpython.com/courses/data-pandas-concat-and-merge/">Combining Data in pandas With concat() and merge()</a></li>
<li><a href="https://realpython.com/courses/python-histograms/">Histogram Plotting in Python: NumPy, Matplotlib, Pandas & Seaborn</a></li>
</ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>
Do You Dare Run Your ML Experiments in Production? with Ville Tuulos - #523
Today we’re joined by a friend of the show and return guest Ville Tuulos, CEO and co-founder of Outerbounds. In our previous conversations with Ville, we explored his experience building and deploying the open-source framework, Metaflow, while working at Netflix. Since our last chat, Ville has embarked on a few new journeys, including writing the upcoming book Effective Data Science Infrastructure, and commercializing Metaflow, both of which we dig into quite a bit in this conversation.
We reintroduce the problem that Metaflow was built to solve and discuss some of the unique use cases that Ville has seen since it's release, the relationship between Metaflow and Kubernetes, and the maturity of services like batch and lambdas allowing a complete production ML system to be delivered. Finally, we discuss the degree to which Ville is catering is Outerbounds’ efforts to building tools for the MLOps community, and what the future looks like for him and Metaflow.
The complete show notes for this episode can be found at twimlai.com/go/523.
From notebooks to Netflix scale with Metaflow
As you start developing an AI/ML based solution, you quickly figure out that you need to run workflows. Not only that, you might need to run those workflows across various kinds of infrastructure (including GPUs) at scale. Ville Tuulos developed Metaflow while working at Netflix to help data scientists scale their work. In this episode, Ville tells us a bit more about Metaflow, his new book on data science infrastructure, and his approach to helping scale ML/AI work.
Sponsors:
RudderStack – Smart customer data pipeline made for developers. RudderStack is the smart customer data pipeline. Connect your whole customer data stack. Warehouse-first, open source Segment alternative.
SignalWire – Build what’s next in communications with video, voice, and messaging APIs powered by elastic cloud infrastructure. Try it today at signalwire.com and use code AI for $25 in developer credit.
Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com
Featuring:
Ville Tuulos – LinkedIn, X
Daniel Whitenack – Website, GitHub, X
Show Notes:
Metaflow
Books
“Effective Data Science Infrastructure” by Ville TuulosUse code podpracticalAI19 for 40% off!
Upcoming Events:
Register for upcoming webinars here!
Scaling Data Science and Machine Learning Infrastructure Like Netflix
<p>Would you move your data science project from a laptop to the cloud? Would you also like to have snapshots of your project saved along the way so that you can go back in time or share the state of your project with another team member? This week on the show, we have Savin Goyal from Netflix. Savin is the technical lead for machine learning infrastructure at Netflix. He joins us to talk about Metaflow, an open-source tool to simplify building, managing, and scaling data science projects.</p>
<p>Metaflow addresses the needs of the numerous data scientists who work at Netflix. Machine learning is key strength for the streaming service. They tried several existing tools to scale their own internal infrastructure and after this experimentation developed Metaflow. </p>
<p>We talk about the history of the project and how someone could get started with the open-source version. Savin also contrasts the cost of infrastructure as compared to data scientists and the cost of their time.</p>
<div class="alert alert-primary" role="alert">
<p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/simplify-gui-dev-pysimplegui/">Simplify Python GUI Development With PySimpleGUI</a></p>
<p>In this step-by-step course, you’ll learn how to create a cross-platform graphical user interface (GUI) using Python and PySimpleGUI. A graphical user interface is an application that has buttons, windows, and lots of other elements that the user can use to interact with your application.</p>
</div>
<p>Topics:</p>
<ul>
<li>00:00:00 – Introduction</li>
<li>00:01:53 – What is Metaflow? </li>
<li>00:04:15 – Savin’s background in data science and infrastructure</li>
<li>00:06:06 – Democratization of infrastructure and iteration of tools</li>
<li>00:10:34 – What information is saved about the infrastructure requirements for a project?</li>
<li>00:17:17 – How are the requirements annotated?</li>
<li>00:18:39 – Sponsor: Digital Ocean’s App Platform</li>
<li>00:19:15 – How do project snapshots work?</li>
<li>00:29:33 – Cost of infrastructure vs data scientists</li>
<li>00:32:28 – Working with data at Netflix scale</li>
<li>00:37:55 – Video Course Spotlight </li>
<li>00:39:06 – Getting an organization to use new tools and then making open-source</li>
<li>00:49:51 – Documentation of Metaflow and getting started on solving infrastructure problems</li>
<li>00:53:57 – What made you interested in working on infrastructure tools?</li>
<li>00:55:13 – What is something you are excited about in the world of Python?</li>
<li>00:56:18 – What do you want to learn next?</li>
<li>00:58:14 – Thanks and goodbye</li>
</ul>
<p>Show Links:</p>
<ul>
<li><a href="https://metaflow.org/">Metaflow: A framework for real-life data science</a></li>
<li><a href="https://docs.metaflow.org/getting-started/tutorials">Metaflow: Tutorials</a></li>
<li><a href="https://www.youtube.com/watch?v=-oMZAS9qfrE">More Data Science, Less Engineering with Netflix’s Metaflow By Savin Goyal - YouTube</a></li>
<li><a href="https://www.r-project.org/">R: The R Project for Statistical Computing</a></li>
<li><a href="https://www.tidyverse.org/">Tidyverse: R packages for data science</a></li>
<li><a href="https://stmorse.github.io/journal/tidyverse-style-pandas.html">Anything you can do, I can do (kinda). Tidyverse pipes in Pandas</a></li>
<li><a href="https://rstudio.github.io/reticulate/">reticulate: R Interface to Python</a></li>
<li><a href="https://airflow.apache.org/">Apache Airflow: Programmatically author, schedule and monitor workflows</a></li>
<li><a href="https://en.wikipedia.org/wiki/Directed_acyclic_graph">Directed acyclic graph (DAG) - Wikipedia article</a></li>
<li><a href="https://realpython.com/courses/pickle-serializing-objects/">Serializing Objects With the Python pickle Module - Real Python Course</a></li>
</ul>
<p>Level up your Python skills with our expert-led courses:</p>
<ul>
<li><a href="https://realpython.com/courses/text-classification-with-keras/">Learn Text Classification With Python and Keras</a></li>
<li><a href="https://realpython.com/courses/using-jupyter-notebooks/">Using Jupyter Notebooks</a></li>
<li><a href="https://realpython.com/courses/simplify-gui-dev-pysimplegui/">Simplify Python GUI Development With PySimpleGUI</a></li>
</ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>
Metaflow, a Human-Centric Framework for Data Science with Ville Tuulos - #326
Today we kick off our re:Invent 2019 series with Ville Tuulos, Machine Learning Infrastructure Manager at Netflix. At re:Invent, Netflix announced the open-sourcing of Metaflow, their “human-centric framework for data science.” In our conversation, we discuss all things Metaflow, including features, user experience, tooling, supported libraries, and much more.
If you’re interested in checking out a Metaflow democast with Villa, reach out at twimlai.com/contact!