Noam Shazeer

This week I welcome on the show two of the most important technologists ever, in any field. Jeff Dean is Google's Chief Scientist, and through 25 years at the company, has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip, to Gemini. Noam Shazeer invented or co-invented all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to Gemini and many other things. We talk about their 25 years at Google, going from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and maybe soon to ASI. My favorite part was Jeff's vision for Pathways, Google’s grand plan for a mutually-reinforcing loop of hardware and algorithmic design and for going past autoregression. That culminates in us imagining *all* of Google-the-company, going through one huge MoE model. And Noam just bites every bullet: 100x world GDP soon; let’s get a million automated researchers running in the Google datacenter; living to see the year 3000.Watch on Youtube; listen on Apple Podcasts or Spotify. Sponsors Scale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale’s Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you’re an AI researcher or engineer, learn about how Scale’s Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh Curious how Jane Street teaches their new traders? They use Figgie, a rapid-fire card game that simulates the most exciting parts of markets and trading. It’s become so popular that Jane Street hosts an inter-office Figgie championship every year. Download from the app store or play on your desktop at figgie.com Meter wants to radically improve the digital world we take for granted. They’re developing a foundation model that automates network management end-to-end. To do this, they just announced a long-term partnership with Microsoft for tens of thousands of GPUs, and they’re recruiting a world class AI research team. To learn more, go to meter.com/dwarkesh To sponsor a future episode, visit dwarkeshpatel.com/p/advertise Timestamps 00:00:00 - Intro 00:02:44 - Joining Google in 1999 00:05:36 - Future of Moore's Law 00:10:21 - Future TPUs 00:13:13 - Jeff’s undergrad thesis: parallel backprop 00:15:10 - LLMs in 2007 00:23:07 - “Holy s**t” moments 00:29:46 - AI fulfills Google’s original mission 00:34:19 - Doing Search in-context 00:38:32 - The internal coding model 00:39:49 - What will 2027 models do? 00:46:00 - A new architecture every day? 00:49:21 - Automated chip design and intelligence explosion 00:57:31 - Future of inference scaling 01:03:56 - Already doing multi-datacenter runs 01:22:33 - Debugging at scale 01:26:05 - Fast takeoff and superalignment 01:34:40 - A million evil Jeff Deans 01:38:16 - Fun times at Google 01:41:50 - World compute demand in 2030 01:48:21 - Getting back to modularity 01:59:13 - Keeping a giga-MoE in-memory 02:04:09 - All of Google in one model 02:12:43 - What’s missing from distillation 02:18:03 - Open research, pros and cons 02:24:54 - Going the distance Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

Jeff Dean

Chief Scientist

Google

Noam Shazeer

Co-lead of Gemini

GoogleCharacter.AI

Noam Shazeer is the co-founder and CEO of Character.AI, a full-stack AI computing platform that gives people access to their own flexible superintelligence. A renowned computer scientist and researcher, Shazeer is one of the foremost experts in artificial intelligence (AI) and natural language processing (NLP). He is a key author for the Transformer, a revolutionary deep learning model enabling language understanding, machine translation, and text generation that has become the foundation of many NLP models. A former member of the Google Brain team, Shazeer led the development of spelling corrector capabilities within Gmail, the algorithm at the heart of AdSense. In Today's Episode with Noam Shazeer We Discuss: 1. Entry into the World of AI and NLP: How did Noam first make his way into the world of AI and come to work on spell corrector with Google? What are 1-2 of his biggest takeaways from spending 20 years at Google? What does Noam know now that he wishes he had known when he started Character? 2. Model Size or Data Size: What is more important, the size of the data or the size of the model? Does Noam agree that "we will not use models in a year that we have today?" What is the lifespan of a model? Does Noam agree that the companies that win are those that are able to switch between models with the most ease? With the majority of data being able to be downloaded from the internet, is there real value in data anymore? 3. The Biggest Barriers: What is the single biggest barrier to Character today? What are the most challenging elements of model training? Why did they need to spend $2M to train an early model? What are the most difficult elements of releasing a horizontal product with so many different use cases? Where does the value accrue in the race for AI dominance; startups or incumbents? 4. AI's Role on Society: Why does Noam believe that AI can create greater not worse human connections? Why is Noam not concerned by the speed of adoption of AI tools? What does Noam know about AI's impact on society that the world does not see?

Noam Shazeer

Co-lead of Gemini

GoogleCharacter.AI

Noam Shazeer played a key role in developing key foundations of modern AI - including co-inventing Transformers at Google, as well as pioneering AI chat pre-chatGPT. These are the foundations supporting today’s AI revolution. On this episode of No Priors, Noam discusses his work as an AI researcher, engineer, inventor, and now CEO.  Noam Shazeer is currently the CEO and Co-founder of Character AI, a service that allows users to design and interact with their own personal bots that take on the personalities of well-known individuals or archetypes. You could have a socratic conversation with Socrates. You could pretend you’re being interviewed by Oprah. Or you could work through a life decision with a therapist bot. Character recently raised $150M from A16Z, Elad Gil, and others. Noam talks about his early AI adventures at Google, why he started Character, and what he sees on the horizon of AI development. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Noam Shazeer - Google Scholar Noam Shazeer - Chief Executive Officer - Character.AI | LinkedIn  Character.AI Sign up for new podcasts every week. Email feedback to show@no-priors.com Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Character_ai Show Notes:  [1:50] - Noam’s early AI projects at Google [7:13] - Noam’s focus on language models and AI applications [11:13] - Character’s co-founder Daniel de Freitas Adiwardana work on Google’s Lambda [13:53] - The origin story of Character.AI  [18:47] - How AI can express emotions [26:51] - What Noam looks for in new hires

Noam Shazeer

Co-lead of Gemini

GoogleCharacter.AI