#331 Sergey Levine: The Robot Revolution Nobody Is Talking About
This episode is sponsored by Modulate. Most voice AI focuses on transcription. Velma takes it further by actually understanding conversations, analyzing tone, timing, stress, and intent using its Ensemble Listening Model architecture. Explore the live preview: https://preview.modulate.ai/ What does it actually mean to build a foundation model for robots? In this episode of Eye on AI, Craig Smith sits down with Sergey Levine, co-founder of Physical Intelligence and professor at UC Berkeley, to explore a fundamentally different approach to building robots, one inspired not by programming a single perfect machine, but by training AI on the broadest and most diverse data possible so robots can learn, adapt, and operate in the unpredictable real world.
Sergey explains why the secret to general-purpose robots isn't perfecting one single machine, but training on massive, diverse data from all kinds of robots and even humans. The more variety the model sees, the better it gets. Just like ChatGPT learned from all the text on the internet, robotic foundation models learn from every robot that has ever moved, grabbed, or interacted with the real world.
We also get into the big humanoid robot debate. Are they the future, or is it mostly hype? Sergey gives an honest and technical take on why the form factor conversation is changing now that foundation models exist, and why that actually opens the door for more creativity, not less.
Finally, Sergey shares what he's most excited about next, building a true data flywheel where robots get smarter the more they are deployed, creating a continuous learning cycle that could change everything.
Subscribe for more conversations with the people building the future of AI and emerging technology.
Stay Updated:
Craig Smith on X: https://x.com/craigss
Eye on A.I. on X: https://x.com/EyeOn_AI
(00:00) Introduction: What Are Foundation Models for Robots?
(01:44) Meet Sergey Levine: Physical Intelligence and UC Berkeley
(02:51) Breaking Down Foundation Models for Non-Technical People
(06:46) Why Real World Data Beats Simulation
(15:00) Building a Broad Robotics Foundation From Scratch
(24:00) The Open World Problem in Robotics
(40:00) Generalist vs Specialist Robots: Which Wins?
(47:00) Humanoid Robots: Real Innovation or Just Hype?
(55:10) The Future: Continuous Learning and the Data Flywheel
(56:23) Guilty Pleasure: Sci Fi and Thinking Beyond the Limits
Sergey Levine - Building LLMs for the Physical World - [Invest Like the Best, EP.465]
My guest today is Sergey Levine, a professor at UC Berkeley and co-founder of Physical Intelligence. The company is building robotic foundation models designed to control any embodied system to do any task in any environment.
Sergey argues that solving robotics at full generality is the right path, and that building systems that learn across many robots, environments, and tasks may be the more scalable approach than building narrow specialists. We discuss how these models can perform new tasks without being trained on them directly, and why everyday human actions remain the hardest problems in the field.
He also reflects on how human trust and acceptance may matter as much as technical breakthroughs in determining when robots become part of daily life.
Please enjoy my conversation with Sergey Levine.
For the full show notes, transcript, and links to mentioned content, check out the episode page here.
-----
Become a Colossus member to get our quarterly print magazine and private audio experience, including exclusive profiles and early access to select episodes. Subscribe at colossus.com/subscribe.
-----
Ramp’s mission is to help companies manage their spend in a way that reduces expenses and frees up time for teams to work on more valuable projects. Go to ramp.com/invest to sign up for free and get a $250 welcome bonus.
-----
Trusted by thousands of businesses, Vanta continuously monitors your security posture and streamlines audits so you can win enterprise deals and build customer trust without the traditional overhead. Visit vanta.com/invest.
-----
WorkOS is a developer platform that enables SaaS companies to quickly add enterprise features to their applications. Visit WorkOS.com to transform your application into an enterprise-ready solution in minutes, not months.
-----
Rogo is the AI platform for finance. They're building agents for Wall Street that are trained to understand how bankers and investors actually do work: from diligence and modeling, to turning analysis into deliverables. To learn more, visit rogo.ai/invest.
-----
Ridgeline has built a complete, real-time, modern operating system for investment managers. It handles trading, portfolio management, compliance, customer reporting, and much more through an all-in-one real-time cloud platform. Visit ridgelineapps.com.
-----
Editing and post-production work for this episode was provided by The Podcast Consultant (https://thepodcastconsultant.com).
Timestamps:
(00:00:00) Welcome to Invest Like the Best
(00:02:43) Intro: Sergey Levine
(00:03:29) Why Bet on Generality Over Specialization
(00:07:24) What if PI succeeds?
(00:09:05) Pros and Cons of Humanoid Robotics
(00:11:02) Timeline of Major Milestones in Robotics
(00:15:47) Sergey's Personal Journey
(00:18:22) Making General Intelligence Happen
(00:19:57) Understanding Robot Data Collection
(00:22:12) Most Surprising Discovery at Physical Intelligence
(00:24:48) The Science of Common Sense
(00:25:36) Long-Range Tasks in Robotics
(00:27:24) Why Wouldn’t We Have A Robot in Our Kitchen by 2050
(00:31:21) Other Interesting Approaches
(00:32:38) Cool vs. Useful in Robotics
(00:36:48) Form Factor Innovation
(00:38:22) Physical Intelligence Analogy
(00:39:30) Economic Transformation from Robotics
(00:40:48) Controversies in the Robotics Community
(00:42:16) Arguments Against End-to-End Learning
(00:42:34) Compositional Learning Explained
(00:43:25) Last Tasks Robots will Conquer
(00:44:30) Dark Parts of the Robotics Brain
(00:47:05) What Makes a Great Researcher
(00:50:15) Manufacturing and Scale Challenges
(00:51:17) How Companies Should Prepare for Robotics
(00:53:38) Boston Dynamics' Demos
(00:55:43) Converging Technologies Enabling Robotics
(00:56:47) How to Stay Up To Date in Robotics
(00:59:51) Near Term Objectives
(01:00:49) Confidence Level Among Researchers
(01:03:31) Google's Experimentation Culture
(01:04:24) The Kindest Thing
Fully autonomous robots are much closer than you think – Sergey Levine
Sergey Levine, one of the world’s top robotics researchers and co-founder of Physical Intelligence, thinks we’re on the cusp of a “self-improvement flywheel” for general-purpose robots. His median estimate for when robots will be able to run households entirely autonomously? 2030.
If Sergey’s right, the world 5 years from now will be an insanely different place than it is today. This conversation focuses on understanding how we get there: we dive into foundation models for robotics, and how we scale both the data and the hardware necessary to enable a full-blown robotics explosion.
Watch on YouTube; listen on Apple Podcasts or Spotify.
Sponsors
* Labelbox provides high-quality robotics training data across a wide range of platforms and tasks. From simple object handling to complex workflows, Labelbox can get you the data you need to scale your robotics research. Learn more at labelbox.com/dwarkesh
* Hudson River Trading uses cutting-edge ML and terabytes of historical market data to predict future prices. I got to try my hand at this fascinating prediction problem with help from one of HRT’s senior researchers. If you’re curious about how it all works, go to hudson-trading.com/dwarkesh
* Gemini 2.5 Flash Image (aka nano banana) isn’t just for generating fun images — it’s also a powerful tool for restoring old photos and digitizing documents. Test it yourself in the Gemini App or in Google’s AI Studio: ai.studio/banana
To sponsor a future episode, visit dwarkesh.com/advertise.
Timestamps
(00:00:00) – Timeline to widely deployed autonomous robots
(00:17:25) – Why robotics will scale faster than self-driving cars
(00:27:28) – How vision-language-action models work
(00:45:37) – Changes needed for brainlike efficiency in robots
(00:57:59) – Learning from simulation
(01:09:18) – How much will robots speed up AI buildouts?
(01:18:01) – If hardware’s the bottleneck, does China win by default?
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
π0: A Foundation Model for Robotics with Sergey Levine - #719
Today, we're joined by Sergey Levine, associate professor at UC Berkeley and co-founder of Physical Intelligence, to discuss π0 (pi-zero), a general-purpose robotic foundation model. We dig into the model architecture, which pairs a vision language model (VLM) with a diffusion-based action expert, and the model training "recipe," emphasizing the roles of pre-training and post-training with a diverse mixture of real-world data to ensure robust and intelligent robot learning. We review the data collection approach, which uses human operators and teleoperation rigs, the potential of synthetic data and reinforcement learning in enhancing robotic capabilities, and much more. We also introduce the team’s new FAST tokenizer, which opens the door to a fully Transformer-based model and significant improvements in learning and generalization. Finally, we cover the open-sourcing of π0 and future directions for their research.
The complete show notes for this episode can be found at https://twimlai.com/go/719.
#176 Sergey Levine: Decoding The Evolution of AI in Robotics
Join host Craig Smith on episode #176 of Eye on AI as he dives deep into the realm of robotic artificial intelligence with Sergey Levine, associate professor in the Department of Electrical Engineering and Computer Sciences at UC Berkeley.
In this episode, Sergey unveils the latest advancements in AI control of robots, exploring the implications of reinforcement learning and the concept of embodied AI.
Discover how Sergey's research is pushing the boundaries of AI, enabling robots to learn manipulation skills and generalize across diverse tasks, transforming the potential of home robots and beyond.
Sergey also shares insights into the RTX project, an ambitious collaboration designed to achieve remarkable generalization across different robot morphologies, enhancing robots' ability to perform language-conditioned manipulation tasks.
If you're fascinated by the intersection of AI, robotics, and the quest for creating adaptable, generalizable machines that promise to revolutionize our interaction with technology, this episode is a must-listen.
Remember to rate us on Apple Podcast and Spotify if this episode ignites your interest in the dynamic field of robotic AI and the visionary work of Sergey Levine.
Stay Updated:
Craig Smith Twitter: https://twitter.com/craigss
Eye on A.I. Twitter: https://twitter.com/EyeOn_AI
(00:00) Preview and Introduction to Home Robots and AI in Robotics
(01:43) World Models and Language Models in Robotics
(04:01) The Challenge of Learning-Based Control and Data Utilization
(06:05) RTX Project: Generalizing Controllers Across Different Robots
(10:09) Uniformity in Model Architecture Across Labs
(13:50) Introduction of RT1 and RT2 Models for Robot Control
(16:06) The Future of Robotic Control Research and Architecture
(18:49) The Impact of Hardware Development on Robotics
(22:15) Advances in Controller and AI Model Development
(26:21) Planning and Acting with Vision Language Models
(31:38) The Proprietary vs. Open-Source Debate in Robotics
(36:23) The Future of Commercial and Open Source Robotics Applications
(40:59) The State of Robotics Research in China
AI Trends 2023: Reinforcement Learning - RLHF, Robotic Pre-Training, and Offline RL with Sergey Levine - #612
Today we’re taking a deep dive into the latest and greatest in the world of Reinforcement Learning with our friend Sergey Levine, an associate professor, at UC Berkeley. In our conversation with Sergey, we explore some game-changing developments in the field including the release of ChatGPT and the onset of RLHF. We also explore more broadly the intersection of RL and language models, as well as advancements in offline RL and pre-training for robotics models, inverse RL, Q learning, and a host of papers along the way. Finally, you don’t want to miss Sergey’s predictions for the top developments of the year 2023!
The complete show notes for this episode can be found at twimlai.com/go/612
Sergey Levine explains the challenges of real world robotics
In Episode One of Season Two, Host Pieter Abbeel is joined by guest (and close collaborator) Sergey Levine, professor at UC Berkeley, EECS. Sergey discusses the early years of his career, how Andrew Ng influenced him to become interested in machine learning, his current projects, and his lab's recent accomplishments.
The conversation concludes with Sergey's view on the dangers of machines not being intelligent enough and his advice for students seeking a career in robotic.
| SUBSCRIBE TO THE ROBOT BRAINS PODCAST TODAY | Visit therobotbrains.ai and follow us on YouTube TheRobotBrainsPodcast, Twitter @therobotbrains, and Instagram @therobotbrains.
| Host: Pieter Abbeel | Executive Producers: Alice Patel & Henry Tobias Jones | Audio Production: Kieron Matthew Banerji | Title Music: Alejandro Del Pozo
Hosted on Acast. See acast.com/privacy for more information.
#108 – Sergey Levine: Robotics and Machine Learning
Sergey Levine is a professor at Berkeley and a world-class researcher in deep learning, reinforcement learning, robotics, and computer vision, including the development of algorithms for end-to-end training of neural network policies that combine perception and control, scalable algorithms for inverse reinforcement learning, and deep RL algorithms.
Support this podcast by supporting these sponsors:
– ExpressVPN: https://www.expressvpn.com/lexpod
– Cash App – use code “LexPodcast” and download:
– Cash App (App Store): https://apple.co/2sPrUHe
– Cash App (Google Play): https://bit.ly/2MlvP5w
If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter, LinkedIn, Facebook, Medium, or YouTube where you can watch the video versions of these conversations. If you enjoy the podcast, please rate it 5 stars on Apple Podcasts, follow on Spotify, or support it on Patreon.
Here’s the outline of the episode. On some podcast players you should be able to click the timestamp to jump to that time.
OUTLINE:
00:00 – Introduction
03:05 – State-of-the-art robots vs humans
16:13 – Robotics may help us understand intelligence
22:49 – End-to-end learning in robotics
27:01 – Canonical problem in robotics
31:44 – Commonsense reasoning in robotics
34:41 – Can we solve robotics through learning?
44:55 – What is reinforcement learning?
1:06:36 – Tesla Autopilot
1:08:15 – Simulation in reinforcement learning
1:13:46 – Can we learn gravity from data?
1:16:03 – Self-play
1:17:39 – Reward functions
1:27:01 – Bitter lesson by Rich Sutton
1:32:13 – Advice for students interesting in AI
1:33:55 – Meaning of life
Advancements in Machine Learning with Sergey Levine - #355
Today we're joined by Sergey Levine, an Assistant Professor at UC Berkeley. We last heard from Sergey back in 2017, where we explored Deep Robotic Learning. Sergey and his lab’s recent efforts have been focused on contributing to a future where machines can be “out there in the real world, learning continuously through their own experience.” We caught up with Sergey at NeurIPS 2019, where Sergey and his team presented 12 different papers -- which means a lot of ground to cover!
Episode 14 - Sergey Levine
This week, I talk to Sergey Levine, one of the most prolific researchers in robot learning. We talked about developing a robot's sense of touch and about robot dreams and whether he believes we know what's happening in the field in Russia and China.
Ep. 37: Sergey Levine on How Deep Learning Will Unleash a Robotics Revolution
The robots that have taken on tasks in the real world - which is to say the world where physics apply - are primarily programmed to do a specific job, such as welding a joint in a car or sweeping up cat hair. So what if robots could learn, and take it a step further - what if they could teach themselves, and pass on their knowledge to other robots? Where could that take machines, and the notion of machine intelligence? And how fast could we get there? Those are the questions our guest Sergey Levine, an assistant professor at UC Berkeley's department of Electrical Engineering and Computer Sciences, is finding answers to.
Deep Robotic Learning with Sergey Levine - TWiML Talk #37
This week we continue our Industrial AI series with Sergey Levine, an Assistant Professor at UC Berkeley whose research focus is Deep Robotic Learning. Sergey is part of the same research team as a couple of our previous guests in this series, Chelsea Finn and Pieter Abbeel, and if the response we’ve seen to those shows is any indication, you’re going to love this episode! Sergey’s research interests, and our discussion, focus in on include how robotic learning techniques can be used to allow machines to acquire autonomously acquire complex behavioral skills. We really dig into some of the details of how this is done and I found that our conversation filled in a lot of gaps for me from the interviews with Pieter and Chelsea. By the way, this is definitely a nerd alert episode! Notes for this show can be found at twimlai.com/talk/37