#547: Parallel Python at Anyscale with Ray
When OpenAI trained GPT-3, they didn't roll their own orchestration layer. They used Ray, an open source Python framework born out of the same Berkeley research lab lineage that gave us Apache Spark. And here's the twist: Ray was originally built for reinforcement learning research, then quietly faded as RL hit a wall. Until ChatGPT showed up. Suddenly reinforcement learning was back, as the post-training step that turns a raw language model into something genuinely useful.
Edward Oakes and Richard Liaw, two founding engineers behind Ray and Anyscale, join me on Talk Python to tell that story. We'll trace Ray from its RISE Lab origins at UC Berkeley to powering some of the largest training runs in the world. We'll talk about what Ray actually is, a distributed execution engine for AI workloads, and how a few lines of Python become work running across hundreds of GPUs. We'll cover Ray Data for multimodal pipelines, the dashboard, the VS Code remote debugger, KubRay for Kubernetes, and where Ray fits alongside Dask, multiprocessing, and asyncio.
If you've ever stared at a single-machine Python script and thought, "there has to be a better way to scale this", this one's for you
Episode sponsors
Sentry Error Monitoring, Code talkpython26
AgentField AI
Talk Python Courses
Links from the show
Guests
Richard Liaw: github.com
Edward Oakes: github.com
Ray: www.ray.io
Example code (we used for walk-through): docs.ray.io
Getting Started with Ray: docs.ray.io
Ray Libraries: docs.ray.io
kuberay: github.com
Watch this episode on YouTube: youtube.com
Episode #547 deep-dive: talkpython.fm/547
Episode transcripts: talkpython.fm
Theme Song: Developer Rap
🥁 Served in a Flask 🎸: talkpython.fm/flasksong
---== Don't be a stranger ==---
YouTube: youtube.com/@talkpython
Bluesky: @talkpython.fm
Mastodon: @talkpython@fosstodon.org
X.com: @talkpython
Michael on Bluesky: @mkennedy.codes
Michael on Mastodon: @mkennedy@fosstodon.org
Michael on X.com: @mkennedy
Ray & KubeRay, with Richard Liaw and Kai-Hsun Chen
In this episode, guest host and AI correspondent Mofi Rahman interviews Richard Liaw and Kai-Hsun Chen from Anyscale about Ray and KubeRay. Ray is an open-source unified compute framework that makes it easy to scale AI and Python workloads, while KubeRay integrates Ray's capabilities into Kubernetes clusters.
Do you have something cool to share? Some questions? Let us know:
- web: kubernetespodcast.com
- mail: kubernetespodcast@google.com
- twitter: @kubernetespod
News of the week CNCF Blog - LitmusChaos audit complete!
Kubernetes Podcast from Google episode 234 - LitmusChaos, with Karthik Satchitanand
Google Cloud Blog - Run your AI inference applications on Cloud Run with NVIDIA GPUs
Diginomica article - KubeCon China - at 33-and-a-third, Linux is a long player. So, why does Linus Torvalds hate AI?
CNCF-Hosted Co-Located Event Schedule for KubeCon NA 2024
Google Kubernetes Engine Release Notes - August 20, 2024 (1.31 available in Rapid Channel)
Kubernetes Podcast from Google - Kubernetes v1.31: "Elli", with Angelos Kolaitis
Red Hat Press Release - Red Hat OpenStack Services on OpenShift is Now Generally Available
Red Hat Enables OpenStack to Run Natively on OpenShift Platform
Broadcom Revamps Tanzu to Simplify Cloud-Native App Development and Deployment
Tanzu Platform 10 Offers Cloud Foundry Users Deep Visibility and Productivity Enhancements
VMware Explore Conference Website
CNCF Blog - Announcing 500 Kubestronauts
CNCF - Kubestronaut FAQ
Dapr Day 2024 Virtual Event Website
Links from the interview Kai-Hsun Chen on LinkedIn
Richard Liaw on LinkedIn
Ray from the RISE Lab at UC Berkeley
Ray: A Distributed System for AI by Robert Nishihara and Philipp Moritz - Jan 9, 2018
KubeRay Docs
KubeRay on GitHub
PyTorch
Apache Airflow
Apache Spark
Kubeflow
Apache Submarine (retired)
Jupyter Notebooks
VS Code
Examples of schedulers for Batch/AI workloads in Kubernetes
Kueue
Volcano
Apache Yunikorn
Examples of observability tools for Batch/AI workloads in Kubernetes
Prometheus
Grafana
Fluentbit
Examples of loadbalancers
Nginx
Istio
Ray Data: Scalable Datasets for ML
Dask Python - Parallel Python
Ray Serve: Scalable and Programmable Serving
HPA - Horizontal Pod Autoscaling in Kubernetes
Karpenter - "Just-in-time nodes for any Kubernetes cluster"
Lazy Computation Graphs with the Ray DAG API
Types of hardware accelerators
Google Cloud Tensor Processing Units (TPUs)
AMD Instinct
AMD Radeon
AWS Trainium
AWS Inferentia
Pandas
Numpy
KubeCon EU 2024 - Accelerators(FPGA/GPU) Chaining to Efficiently Handle Large AI/ML Workloads in K8s - Sampath Priyankara, Nippon Telegraph and Telephone Corporation & Masataka Sonoda, Fujitsu Limited
NVidia Megatron
Links from the post-interview chat DRA - Dynamic Resource Allocation in Kubernetes
Different ways of Running RayJob on Kubernetes
Ray framework diagram in the docs