Richard Sutton – Father of RL thinks LLMs are a dead end
Richard Sutton is the father of reinforcement learning, winner of the 2024 Turing Award, and author of The Bitter Lesson. And he thinks LLMs are a dead end.
After interviewing him, my steel man of Richard’s position is this: LLMs aren’t capable of learning on-the-job, so no matter how much we scale, we’ll need some new architecture to enable continual learning.
And once we have it, we won’t need a special training phase — the agent will just learn on-the-fly, like all humans, and indeed, like all animals.
This new paradigm will render our current approach with LLMs obsolete.
In our interview, I did my best to represent the view that LLMs might function as the foundation on which experiential learning can happen… Some sparks flew.
A big thanks to the Alberta Machine Intelligence Institute for inviting me up to Edmonton and for letting me use their studio and equipment.
Enjoy!
Watch on YouTube; listen on Apple Podcasts or Spotify.
Sponsors
* Labelbox makes it possible to train AI agents in hyperrealistic RL environments. With an experienced team of applied researchers and a massive network of subject-matter experts, Labelbox ensures your training reflects important, real-world nuance. Turn your demo projects into working systems at labelbox.com/dwarkesh
* Gemini Deep Research is designed for thorough exploration of hard topics. For this episode, it helped me trace reinforcement learning from early policy gradients up to current-day methods, combining clear explanations with curated examples. Try it out yourself at gemini.google.com
* Hudson River Trading doesn’t silo their teams. Instead, HRT researchers openly trade ideas and share strategy code in a mono-repo. This means you’re able to learn at incredible speed and your contributions have impact across the entire firm. Find open roles at hudsonrivertrading.com/dwarkesh
Timestamps
(00:00:00) – Are LLMs a dead end?
(00:13:04) – Do humans do imitation learning?
(00:23:10) – The Era of Experience
(00:33:39) – Current architectures generalize poorly out of distribution
(00:41:29) – Surprises in the AI field
(00:46:41) – Will The Bitter Lesson still apply post AGI?
(00:53:48) – Succession to AIs
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
#170 Richard Sutton on Pursuing AGI Through Reinforcement Learning
Join host Craig Smith on episode #170 of Eye on AI, for a riveting conversation with Richard Sutton, currently serving as a professor of computing science at the University of Alberta and a research scientist at Keen Technologies.
Sutton is considered one of the founders of modern computational reinforcement learning, having several significant contributions to the field, including temporal difference learning and policy gradient methods.
In this episode, we go through the Alberta Plan for AI development, the transformative potential of reinforcement learning, and the future of AI in augmenting human intelligence.
Richard Sutton shares insights on the importance of computational power, the impact of large language models, and the vision for AI that interacts with the world through goals and learning from its environment.
We also explore the challenges and opportunities in making AI more embodied and goal-oriented, and how this approach could revolutionize our interaction with technology.
A must-listen for anyone interested in the cutting-edge advancements in AI and its societal implications.
Don't forget to rate us on Apple Podcast and Spotify if you enjoyed this episode!
This episode is sponsored by Netsuite by Oracle, the number one cloud financial system, streamlining accounting, financial management, inventory, HR, and more.
Download NetSuite's popular KPI Checklist, designed to give you consistently excellent performance - absolutely free at https://netsuite.com/EYEONAI
Stay Updated:
Craig Smith Twitter: https://twitter.com/craigss
Eye on A.I. Twitter: https://twitter.com/EyeOn_AI
(00:00) Preview and Introduction
(02:15) AI's Evolution: Insights from Richard Sutton
(07:08) Breaking Down AI: From Algorithms to AGI
(10:50) The Alberta Experiment: A New Approach to AI Learning
(18:27) The Horde Architecture Explained
(21:23) Power Collaboration: Carmack, Keen, and the Future of AI
(25:04) Expanding AI's Learning Capabilities
(31:34) Is AI the Future of Technology?
(35:29) The Next Step in AI: Experiential Learning and Embodiment
(40:00) AI's Building Blocks: Algorithms for a Smarter Tomorrow
(45:59) The Strategy of AI: Planning and Representation
(49:27) Learning Methods Face-Off: Reinforcement vs. Supervised
(52:53) The 2030 Vision: Aiming for True AI Intelligence?
.jpg?width=400){kind=small}