Managing Large Python Data Science Projects With Dask
<p>What do you do when your data science project doesn’t fit within your computer’s memory? One solution is to distribute it across multiple worker machines. This week on the show, Guido Imperiale from Coiled talks about Dask and managing large data science projects through distributed computing.</p> <p>We talk about projects where an orchestration system like Dask will help. Dask is designed to take advantage of parallel computing, spreading the work and data across multiple machines. Many familiar techniques for working with pandas and NumPy data are supported with Dask equivalents. </p> <p>We also discuss the differences between managed and unmanaged memory. Guido shares advice on how to tackle memory issues while working with Dask.</p> <p>This week we also talk briefly with Jodie Burchell, who will be a guest host on upcoming episodes. As a data scientist, Jodie will be bringing new topics, projects, and discussions to the show.</p> <div class="alert alert-primary" role="alert"> <p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/exploring-scopes-and-closures-in-python/">Exploring Scopes and Closures in Python</a></p> <p>In this Code Conversation video course, you’ll take a deep dive into how scopes and closures work in Python. To do this, you’ll use a debugger to walk through some sample code, and then you’ll take a peek under the hood to see how Python holds variables internally.</p> </div> <p>Topics:</p> <ul> <li>00:00:00 – Introduction</li> <li>00:01:56 – Guido at PyCon DE 2022</li> <li>00:02:14 – Working on Dask for Coiled</li> <li>00:03:27 – Dask project history</li> <li>00:04:00 – How would someone start to use Dask? </li> <li>00:10:28 – Managing distributed data</li> <li>00:11:18 – Data files CSV vs Parquet</li> <li>00:15:02 – Managed vs unmanaged memory</li> <li>00:22:42 – Video Course Spotlight</li> <li>00:24:01 – Dask active memory manager</li> <li>00:28:36 – Learning best practices and Dask tutorials</li> <li>00:33:06 – Where is Dask being used?</li> <li>00:35:45 – What are you excited about in the world of Python?</li> <li>00:37:55 – What do you want to learn next?</li> <li>00:40:31 – Thanks, Guido</li> <li>00:40:40 – Introduction to Jodie Burchell</li> <li>00:45:28 – Goodbye</li> </ul> <p>Show Links:</p> <ul> <li><a href="https://coiled.io/">Coiled | Python for Data Science on the Cloud with Dask</a></li> <li><a href="https://www.youtube.com/watch?v=pYtjhhVLqFQ">Guido Imperiale: Introducing the Dask Active Memory Manager - PyCon DE 2022 - YouTube</a></li> <li><a href="https://www.youtube.com/watch?v=B0ncePK1uuc">Active Memory Management on Dask.Distributed - Guido Imperiale | Dask Summit 2021 - YouTube</a></li> <li><a href="https://coiled.io/blog/tackling-unmanaged-memory-with-dask/">Tackling unmanaged memory with Dask | Coiled</a></li> <li><a href="https://towardsdatascience.com/the-beginners-guide-to-distributed-computing-6d6833796318">The Beginner’s Guide to Distributed Computing | Richard Pelgrim</a></li> <li><a href="https://coiled.io/blog/common-dask-mistakes/">Common Mistakes to Avoid when Using Dask | Coiled</a></li> <li><a href="https://parquet.apache.org/docs/file-format/">File Format | Apache Parquet</a></li> <li><a href="https://dask.org/">Dask: Scalable analytics in Python</a></li> <li><a href="https://peps.python.org/pep-0554/">PEP 554 – Multiple Interpreters in the Stdlib | peps.python.org</a></li> <li><a href="https://developer.nvidia.com/cuda-python">CUDA Python | NVIDIA Developer</a></li> <li><a href="https://www.rust-lang.org/">Rust Programming Language</a></li> <li><a href="https://coiled.io/product/">Product : Coiled</a></li> <li><a href="https://twitter.com/CoiledHQ">Coiled (@CoiledHQ) / Twitter</a></li> <li><a href="https://twitter.com/t_redactyl">Jodie Burchell (@t_redactyl) | Twitter</a></li> <li><a href="https://www.amazon.com/Learn-Python-through-Nursery-Rhymes/dp/1735907987">Learn Python through Nursery Rhymes and Fairy Tales: Shari Eskenas - Amazon</a></li> </ul> <p>Level up your Python skills with our expert-led courses:</p> <ul> <li><a href="https://realpython.com/courses/data-cleaning-with-pandas-and-numpy/">Data Cleaning With pandas and NumPy</a></li> <li><a href="https://realpython.com/courses/navigate-namespaces-scope/">Navigating Namespaces and Scope in Python</a></li> <li><a href="https://realpython.com/courses/exploring-scopes-and-closures-in-python/">Exploring Scopes and Closures in Python</a></li> </ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>