#425: Memray: The endgame Python memory profiler
See the full show notes for this episode on the website at talkpython.fm/425
Appearances (6)
How the Python team is adapting the language for an AI future
Pablo is a Python core developer, Steering Council member, and release manager of Python 3.10 and 3.11. He’s currently a senior software engineer at Bloomberg. Looking for a comprehensive guide to contributing to Python? The Python Developer’s Guide is the place to start. The Zen of Python is a list of the language’s guiding principles, including, “There should be one—and preferably only one—obvious way to do it.” Find Pablo on LinkedIn, Twitter, and GitHub. Find Kyle, a senior software engineer on Stack Overflow’s public platform, on Linked, Twitter, and GitHub. See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
What it's like to be on the Python Steering Council
Pablo is a Python core developer, Steering Council member, and release manager of Python 3.10 and 3.11. He splits this work 50/50 with his day job as a senior software engineer at Bloomberg. An astrophysicist by training, he did his PhD on rotating black holes. Whether you’re a new contributor or a seasoned veteran, the Python Developer’s Guide is a comprehensive guide to contributing to Python. Pablo is on LinkedIn, Twitter, and GitHub. Kyle is also on Linked, Twitter, and GitHub. Shoutout to Inquisitive Badge winner trever for asking well-received questions on 30 separate days. See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Fostering an Internal Python Community & Managing the 3.11 Release
<p>Does your company have a plan for growing an internal Python community? What are the attributes to look for when bringing someone into your department? This week on the show, Pablo Galindo Salgado returns to talk about building community through the Python Guild at Bloomberg and managing the release of Python 3.11.</p> <p>Pablo describes how the Python Guild started and currently operates inside Bloomberg. We talk about how it fosters community and acts as a way to promote internally developed tools across disparate teams. We also discuss how work groups use it to find new internal candidates for their teams. </p> <p>Pablo talks about his role as release manager for Python 3.10 and 3.11. He shares the intense journey the team has had this year in preparing for the release of 3.11. He details updating testing strategies to work with the new specializing adaptive interpreter.</p> <div class="alert alert-primary" role="alert"> <p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/python-basics-strings-string-methods/">Python Basics: Strings and String Methods</a></p> <p>In Python, collections of text are called strings. In this course, you’ll learn about this fundamental data type and the string methods that you can use to manipulate strings. Along the way, you’ll learn ways to work with strings of numbers, and how to format strings for printing.</p> </div> <p>Topics:</p> <ul> <li>00:00:00 – Introduction</li> <li>00:02:13 – Python Guild inside of Bloomberg</li> <li>00:13:31 – Finding candidates for the guild from other areas</li> <li>00:19:11 – Sponsor: Platform.sh</li> <li>00:19:47 – Considering eagerness to learn and excitement</li> <li>00:29:44 – Structuring the guild into work groups</li> <li>00:33:43 – How are things going as release manager?</li> <li>00:38:25 – Testing for adaptive interpreters</li> <li>00:44:02 – Working toward the feature freeze</li> <li>00:50:39 – Changing the parser went smoothly</li> <li>00:54:34 – Video Course Spotlight</li> <li>00:55:55 – Where do you find the time?</li> <li>00:59:51 – How’s the sweep picking coming along?</li> <li>01:00:33 – What are you excited about in the world of Python?</li> <li>01:01:29 – What do you want to learn next?</li> <li>01:07:18 – How can people follow the work you do?</li> <li>01:08:20 – Thanks and goodbye</li> </ul> <p>Show Links:</p> <ul> <li><a href="https://www.bloomberg.com/company/stories/bloomberg-memray-open-source-profiler-python-code/">Bloomberg publishes Memray, a new open source memory profiler for Python code - Bloomberg LP</a></li> <li><a href="https://github.com//bloomberg/memray">bloomberg/memray: Memray is a memory profiler for Python</a></li> <li><a href="https://www.pluralsight.com/tech-blog/guilds-at-pluralsight/">Pluralsight Tech Blog - Guilds at Pluralsight</a></li> <li><a href="https://www.capitalone.com/tech/open-source/building-python-user-community-among-capital-ones-data-analysts/">Lessons From Building a Community of Python Users Among Capital One’s Analysts - Capital One</a></li> <li><a href="https://peps.python.org/pep-0013/">PEP 13 – Python Language Governance - peps.python.org</a></li> <li><a href="https://pythoninsider.blogspot.com/2022/09/python-3110rc2-is-now-available.html">Python Insider: Python 3.11.0rc2 is now available</a></li> <li><a href="https://docs.python.org/3.11/whatsnew/3.11.html#faster-cpython">What’s New In Python 3.11 — Python 3.11.0rc2 documentation</a></li> <li><a href="https://peps.python.org/pep-0659/">PEP 659 – Specializing Adaptive Interpreter - peps.python.org</a></li> <li><a href="https://en.wikipedia.org/wiki/Andon_(manufacturing)">Andon (manufacturing) - Wikipedia</a></li> <li><a href="https://www.rust-lang.org/learn">Learn Rust - Rust Programming Language</a></li> <li><a href="https://www.apple.com/swift/">Swift - Apple</a></li> <li><a href="https://www.raywenderlich.com/">raywenderlich.com - High quality programming tutorials: iOS, Android, Swift, Kotlin, Flutter, Server Side Swift, Unity, and more!</a></li> <li><a href="https://surveys.jetbrains.com/s3/c2-python-developers-survey-2022">Python Developers Survey 2022</a></li> <li><a href="https://github.com/pablogsal">pablogsal (Pablo Galindo Salgado) - GitHub</a></li> <li><a href="https://twitter.com/pyblogsal">Pablo Galindo Salgado (@pyblogsal) - Twitter</a></li> </ul> <p>Level up your Python skills with our expert-led courses:</p> <ul> <li><a href="https://realpython.com/courses/python-basics-strings-string-methods/">Python Basics: Strings and String Methods</a></li> <li><a href="https://realpython.com/courses/cool-new-features-python-310/">Cool New Features in Python 3.10</a></li> <li><a href="https://realpython.com/courses/python-basics-first-program/">Python Basics: Code Your First Python Program</a></li> </ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>
Using a Memory Profiler in Python & What It Can Teach You
<p>Have you used a memory profiler to gauge the performance of your Python application? Maybe you’re using it to troubleshoot memory issues when loading a large data science project. What could running a profiler show you about a codebase you’re learning? This week on the show, Pablo Galindo Salgado returns to talk about Memray, a powerful tracing memory profiler.</p> <p>Pablo developed Memray while working at Bloomberg to track memory allocations beyond Python code into native extensions and the interpreter itself. It’s a compelling tool that provides fine-grain reports to help you understand where memory is used. </p> <p>Pablo shares the reporting that Memray provides, including live mode, flame graphs, and a <code>pytest</code> plug-in. We also discuss how a tracing memory profiler can help you understand a new codebase. </p> <p>He walks through how he developed the first prototype internally and eventually moved the project into open source. This is the first part of my conversation with Pablo. In a couple of weeks, you’ll get the second part, where we talk about Python guilds inside large companies and his work as the release manager for Python 3.10 and 3.11.</p> <div class="alert alert-primary" role="alert"> <p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/sqlite-sqlalchemy-python/">SQLite and SQLAlchemy in Python: Moving Your Data Beyond Flat Files</a></p> <p>In this video course, you’ll learn how to store and retrieve data using Python, SQLite, and SQLAlchemy as well as with flat files. Using SQLite with Python brings with it the additional benefit of accessing data with SQL. By adding SQLAlchemy, you can work with data in terms of objects and methods.</p> </div> <p>Topics:</p> <ul> <li>00:00:00 – Introduction</li> <li>00:02:48 – When should you use a memory profiler?</li> <li>00:05:13 – Fine-grain reporting </li> <li>00:13:17 – Sampling profiler vs tracing profiler</li> <li>00:19:46 – Sponsor: Deepgram</li> <li>00:20:31 – What is a flame graph?</li> <li>00:30:36 – Using Rich for terminal reporters</li> <li>00:40:08 – Currently only Linux and macOS </li> <li>00:41:13 – <code>pytest</code> plug-in</li> <li>00:42:03 – Showing native allocation details</li> <li>00:44:20 – Video Course Spotlight</li> <li>00:45:52 – Using a profiler to learn a codebase</li> <li>00:54:39 – Moving from internal project to open source</li> <li>01:02:17 – Thanks and goodbye</li> </ul> <p>Show Links:</p> <ul> <li><a href="https://github.com/bloomberg/memray">memray: Memray is a memory profiler for Python - GitHub bloomberg/memray</a></li> <li><a href="https://pypi.org/project/memray/">memray - PyPI</a></li> <li><a href="https://www.bloomberg.com/company/stories/bloomberg-memray-open-source-profiler-python-code/">Bloomberg publishes Memray, a new open source memory profiler for Python code - Bloomberg LP</a></li> <li><a href="https://www.datadoghq.com/knowledge-center/distributed-tracing/flame-graph/">What is a Flame Graph? How it Works & Use Cases - Datadog</a></li> <li><a href="https://en.wikipedia.org/wiki/Gantt_chart">Gantt chart - Wikipedia</a></li> <li><a href="https://pypi.org/project/py-spy/">py-spy - PyPI</a></li> <li><a href="https://pypi.org/project/scalene/">scalene - PyPI</a></li> <li><a href="https://pythonspeed.com/fil/">Reduce your Python program’s memory usage with Fil</a></li> <li><a href="https://github.com/pythonspeed/filprofiler">fil: A Python memory profiler for data processing and scientific computing applications - GitHub pythonspeed/filprofiler</a></li> <li><a href="https://realpython.com/podcasts/rpp/24/">Episode #24: Options for Packaging Your Python Application: Wheels, Docker, and More - The Real Python Podcast</a></li> <li><a href="https://pypi.org/project/rich/">rich - PyPI</a></li> <li><a href="https://www.textualize.io/">Textualize</a></li> <li><a href="https://www.brendangregg.com/overview.html">Brendan Gregg</a></li> <li><a href="https://www.brendangregg.com/perf.html">Linux perf Examples</a></li> <li><a href="https://deepgram.com/realpython/">Deepgram - Speech-to-Text for Developers & Enterprise</a></li> </ul> <p>Level up your Python skills with our expert-led courses:</p> <ul> <li><a href="https://realpython.com/courses/python-debugging-pdb/">Debugging in Python With pdb</a></li> <li><a href="https://realpython.com/courses/testing-your-code-with-pytest/">Testing Your Code With pytest</a></li> </ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>
Creating Better Error Messages for Python 3.10 & 3.11
<p>What goes into creating those enhanced error messages in the latest versions of Python? How does the new PEG parser help to pinpoint where errors have occurred? This week on the show, Pablo Galindo Salgado talks about the work that goes into creating these improvements.</p> <p>Pablo is a core CPython developer and is the release manager for Python versions 3.10 and 3.11. He is also serving his second term on the Python Steering Council. </p> <p>Pablo is pleasantly surprised by the positive feedback for the new error messages in Python 3.10. He shares some of the upcoming enhancements for 3.11. We talk about how the new PEG parser allows for greater context when defining errors and pinpointing where they occur.</p> <p>We talk about how he started contributing to CPython. He also shares some of the programming experiences he had while studying physics at university.</p> <div class="alert alert-primary" role="alert"> <p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/python-linear-regression/">Starting With Linear Regression in Python</a></p> <p>In this video course, you’ll get started with linear regression in Python. Linear regression is one of the fundamental statistical and machine learning techniques, and Python is a popular choice for machine learning.</p> </div> <p>Topics:</p> <ul> <li>00:00:00 – Introduction</li> <li>00:01:56 – Member of the Python Steering Council</li> <li>00:02:40 – Physics background and research use of Python</li> <li>00:08:43 – How did you get involved in core development?</li> <li>00:10:27 – Why did you take on the role of release manager?</li> <li>00:13:38 – What challenges have you found along the way?</li> <li>00:19:08 – Sponsor: LinearB</li> <li>00:19:48 – What motivated you to add enhanced error messages?</li> <li>00:29:33 – How does the PEG parser help in these situations?</li> <li>00:37:04 – PEG parser and infinite lookahead</li> <li>00:40:58 – Identifying where the syntax is wrong</li> <li>00:43:28 – Finding where a comma is missing</li> <li>00:48:49 – Is this a dictionary missing a colon, or is it a set?</li> <li>00:50:19 – Identifying missing portions of a <code>try</code> … <code>except</code> block</li> <li>00:51:44 – Video Course Spotlight</li> <li>00:53:00 – Informing library maintainers and not slowing performance</li> <li>00:56:18 – Enhanced error messages coming in 3.11</li> <li>01:06:38 – Real Python preview of Python 3.11</li> <li>01:07:28 – What are you excited about in the world of Python?</li> <li>01:13:00 – What do you want to learn next?</li> <li>01:15:43 – How to contribute to the project?</li> <li>01:20:24 – Thanks and goodbye</li> </ul> <p>Show Links:</p> <ul> <li><a href="https://peps.python.org/pep-8016/">PEP 8016 – The Steering Council Model | peps.python.org</a></li> <li><a href="https://fortran-lang.org/">Fortran Programming Language</a></li> <li><a href="https://www.wolfram.com/mathematica/">Wolfram Mathematica: Modern Technical Computing</a></li> <li><a href="https://en.wikipedia.org/wiki/C_(programming_language)#External_links">C (programming language) - Wikipedia</a></li> <li><a href="https://www.wired.com/2007/08/mare-nostrum-th/">Mare Nostrum, The Temple Of The Bit | WIRED</a></li> <li><a href="https://pythoninsider.blogspot.com/2022/03/python-3104-and-3912-are-now-available.html">Python Insider: Python 3.10.4 and 3.9.12 are now available out of schedule</a></li> <li><a href="https://peps.python.org/pep-0657/">PEP 657 – Include Fine Grained Error Locations in Tracebacks | peps.python.org</a></li> <li><a href="https://realpython.com/python311-error-messages/">Python 3.11 Preview: Even Better Error Messages – Real Python</a></li> <li><a href="https://github.com/friendly-traceback/friendly-traceback">friendly-traceback: Friendlier Python tracebacks.</a></li> <li><a href="https://ipython.org/">IPython - Interactive Computing</a></li> <li><a href="https://coverage.readthedocs.io/en/6.3.2/">Coverage.py - Documentation</a></li> <li><a href="https://github.com/faster-cpython/ideas/blob/main/FasterCPythonDark.pdf">faster-cpython/ideas - presentation pdf · GitHub</a></li> <li><a href="https://peps.python.org/pep-0659/">PEP 659 – Specializing Adaptive Interpreter | peps.python.org</a></li> <li><a href="https://www.wikihow.com/Sweep-Pick">How to Sweep Pick: 14 Steps (with Pictures) - wikiHow</a></li> <li><a href="https://line6.com/variax-modeling-guitars/shuriken/">Line 6 - Shuriken Variax Guitar</a> </li> <li><a href="https://us.pycon.org/2022/schedule/presentation/69/">Talks: Making Python Better One Error Message at a Time PyCon 2022</a></li> <li><a href="https://devguide.python.org/">Python Developer’s Guide</a></li> <li><a href="https://devguide.python.org/parser/">Guide to CPython’s Parser - Python Developer’s Guide</a></li> <li><a href="https://github.com/pablogsal">pablogsal (Pablo Galindo Salgado) · GitHub</a></li> <li><a href="https://twitter.com/pyblogsal">Pablo Galindo Salgado (@pyblogsal) / Twitter</a></li> </ul> <p>Level up your Python skills with our expert-led courses:</p> <ul> <li><a href="https://realpython.com/courses/cool-new-features-python-310/">Cool New Features in Python 3.10</a></li> <li><a href="https://realpython.com/courses/python-linear-regression/">Starting With Linear Regression in Python</a></li> <li><a href="https://realpython.com/courses/raising-handling-exceptions/">Raising and Handling Python Exceptions</a></li> </ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>