The Best of 2025 (So Far) with Sarah Guo and Elad Gil
2025 has thus far been a year of great leaps and advances in AI technology. And Sarah and Elad have spoken with some of the most enterprising founders and scientific minds in the field of AI today. So we’re revisiting a few of our favorite conversations on No Priors so far in 2025 – Winston Weinberg (Harvey), Dr. Fei-Fei Li (World Labs), Brendan Foody (Mercor), Dan Hendrycks (Center for AI Safety), Noubar Afeyan (Flagship Pioneering), Brandon McKinzie and Eric Mitchell (OpenAI o3), Isa Fulford (OpenAI), Arvind Jain (Glen), and Dr. Shiv Rao (Abridge).
Sign up for new podcasts every week. Email feedback to show@no-priors.com
Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil
Chapters:
00:00 – Episode Introduction
0:21 – Winston Weinberg on Leaning into New Capabilities
02:01 – Dr. Fei-Fei Li on Spatial Intelligence
04:13 – Brendan Foody on AI Disruption in the Workforce
06:10 – Dan Hendrycks on the Geopolitics of Superintelligence
08:06 – Noubar Afeyan on Entrepreneurship
10:38 – Brandon McKinzie and Eric Mitchell on Reasoning Models
12:41 – Isa Fulford on Training Deep Research
13:49 – Arvind Jain on Innovating Enterprise Search
16:21 – Dr. Shiv Rao on AI’s Human Impact
18:58 – Conclusion
Superintelligence Strategy (Dan Hendrycks)
Deep dive with Dan Hendrycks, a leading AI safety researcher and co-author of the "Superintelligence Strategy" paper with former Google CEO Eric Schmidt and Scale AI CEO Alexandr Wang.
*** SPONSOR MESSAGES
Gemini CLI is an open-source AI agent that brings the power of Gemini directly into your terminal - https://github.com/google-gemini/gemini-cli
Prolific: Quality data. From real people. For faster breakthroughs.
https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=script-gen
***
Hendrycks argues that society is making a fundamental mistake in how it views artificial intelligence. We often compare AI to transformative but ultimately manageable technologies like electricity or the internet. He contends a far better and more realistic analogy is nuclear technology. Like nuclear power, AI has the potential for immense good, but it is also a dual-use technology that carries the risk of unprecedented catastrophe.
The Problem with an AI "Manhattan Project":
A popular idea is for the U.S. to launch a "Manhattan Project" for AI—a secret, all-out government race to build a superintelligence before rivals like China. Hendrycks argues this strategy is deeply flawed and dangerous for several reasons:
- It wouldn’t be secret. You cannot hide a massive, heat-generating data center from satellite surveillance.
- It would be destabilizing. A public race would alarm rivals, causing them to start their own desperate, corner-cutting projects, dramatically increasing global risk.
- It’s vulnerable to sabotage. An AI project can be crippled in many ways, from cyberattacks that poison its training data to physical attacks on its power plants. This is what the paper refers to as a "maiming attack."
This vulnerability leads to the paper's central concept: Mutual Assured AI Malfunction (MAIM). This is the AI-era version of the nuclear-era's Mutual Assured Destruction (MAD). In this dynamic, any nation that makes an aggressive, destabilizing bid for a world-dominating AI must expect its rivals to sabotage the project to ensure their own survival.
This deterrence, Hendrycks argues, is already the default reality we live in.
A Better Strategy: The Three Pillars
Instead of a reckless race, the paper proposes a more stable, three-part strategy modeled on Cold War principles:
- Deterrence: Acknowledge the reality of MAIM. The goal should not be to "win" the race to superintelligence, but to deter anyone from starting such a race in the first place through the credible threat of sabotage.
- Nonproliferation: Just as we work to keep fissile materials for nuclear bombs out of the hands of terrorists and rogue states, we must control the key inputs for catastrophic AI. The most critical input is advanced AI chips (GPUs). Hendrycks makes the powerful claim that building cutting-edge GPUs is now more difficult than enriching uranium, making this strategy viable.
- Competitiveness: The race between nations like the U.S. and China should not be about who builds superintelligence first. Instead, it should be about who can best use existing AI to build a stronger economy, a more effective military, and more resilient supply chains (for example, by manufacturing more chips domestically).
Dan says the stakes are high if we fail to manage this transition:
- Erosion of Control
- Intelligence Recursion
- Worthless Labor
Hendrycks maintains that while the risks are existential, the future is not set.
TOC:
1 Measuring the Beast [00:00:00]
2 Defining the Beast [00:11:34]
3 The Core Strategy [00:38:20]
4 Ideological Battlegrounds [00:53:12]
5 Mechanisms of Control [01:34:45]
TRANSCRIPT:
https://app.rescript.info/public/share/cOKcz4pWRPjh7BTIgybd7PUr_vChUaY6VQW64No8XMs
<truncated, see refs and larger description on YT version>
Three Red Lines We're About to Cross Toward AGI (Daniel Kokotajlo, Gary Marcus, Dan Hendrycks)
What if the most powerful technology in human history is being built by people who openly admit they don't trust each other? In this explosive 2-hour debate, three AI experts pull back the curtain on the shocking psychology driving the race to Artificial General Intelligence—and why the people building it might be the biggest threat of all. Kokotajlo predicts AGI by 2028 based on compute scaling trends. Marcus argues we haven't solved basic cognitive problems from his 2001 research. The stakes? If Kokotajlo is right and Marcus is wrong about safety progress, humanity may have already lost control.
Sponsor messages:
========
Google Gemini: Google Gemini features Veo3, a state-of-the-art AI video generation model in the Gemini app. Sign up at https://gemini.google.com
Tufa AI Labs are hiring for ML Engineers and a Chief Scientist in Zurich/SF. They are top of the ARCv2 leaderboard!
https://tufalabs.ai/
========
Guest Powerhouse
Gary Marcus - Cognitive scientist, author of "Taming Silicon Valley," and AI's most prominent skeptic who's been warning about the same fundamental problems for 25 years (https://garymarcus.substack.com/)
Daniel Kokotajlo - Former OpenAI insider turned whistleblower who reveals the disturbing rationalizations of AI lab leaders in his viral "AI 2027" scenario (https://ai-2027.com/)
Dan Hendrycks - Director of the Center for AI Safety who created the benchmarks used to measure AI progress and argues we have only years, not decades, to prevent catastrophe (https://danhendrycks.com/)
Transcript:
http://app.rescript.info/public/share/tEcx4UkToi-2jwS1cN51CW70A4Eh6QulBRxDILoXOno
TOC:
Introduction: The AI Arms Race
00:00:04 - The Danger of Automated AI R&D
00:00:43 - The Rationalization: "If we don't, someone else will"
00:01:56 - Sponsor Reads (Tufa AI Labs & Google Gemini)
00:02:55 - Guest Introductions
The Philosophical Stakes
00:04:13 - What is the Positive Vision for AGI?
00:07:00 - The Abundance Scenario: Superintelligent Economy
00:09:06 - Differentiating AGI and Superintelligence (ASI)
00:11:41 - Sam Altman: "A Decade in a Month"
00:14:47 - Economic Inequality & The UBI Problem
Policy and Red Lines
00:17:13 - The Pause Letter: Stopping vs. Delaying AI
00:20:03 - Defining Three Concrete Red Lines for AI Development
00:25:24 - Racing Towards Red Lines & The Myth of "Durable Advantage"
00:31:15 - Transparency and Public Perception
00:35:16 - The Rationalization Cascade: Why AI Labs Race to "Win"
Forecasting AGI: Timelines and Methodologies
00:42:29 - The Case for Short Timelines (Median 2028)
00:47:00 - Scaling Limits: Compute, Data, and Money
00:49:36 - Forecasting Models: Bio-Anchors and Agentic Coding
00:53:15 - The 10^45 FLOP Thought Experiment
The Great Debate: Cognitive Gaps vs. Scaling
00:58:41 - Gary Marcus's Counterpoint: The Unsolved Problems of Cognition
01:00:46 - Current AI Can't Play Chess Reliably
01:08:23 - Can Tools and Neurosymbolic AI Fill the Gaps?
01:16:13 - The Multi-Dimensional Nature of Intelligence
01:24:26 - The Benchmark Debate: Data Contamination and Reliability
01:31:15 - The Superhuman Coder Milestone Debate
01:37:45 - The Driverless Car Analogy
The Alignment Problem
01:39:45 - Has Any Progress Been Made on Alignment?
01:42:43 - "Fairly Reasonably Scares the Sh*t Out of Me"
01:46:30 - Distinguishing Model vs. Process Alignment
Scenarios and Conclusions
01:49:26 - Gary's Alternative Scenario: The Neurosymbolic Shift
01:53:35 - Will AI Become Jeff Dean?
01:58:41 - Takeoff Speeds and Exceeding Human Intelligence
02:03:19 - Final Disagreements and Closing Remarks
REFS:
Gary Marcus (2001) - The Algebraic Mind
https://mitpress.mit.edu/9780262632683/the-algebraic-mind/
00:59:00
Gary Marcus & Ernest Davis (2019) - Rebooting AI
https://www.penguinrandomhouse.com/books/566677/rebooting-ai-by-gary-marcus-and-ernest-davis/
01:31:59
Gary Marcus (2024) - Taming SV
https://www.hachettebookgroup.com/titles/gary-marcus/taming-silicon-valley/9781541704091/
00:03:01
AI's Rising Risks: Hacking, Virology, Loss of Control — With Dan Hendrycks
Dan Hendrycks is the Director and co-founder of the Center for AI Safety, and an advisor to Scale AI and xAI. He joins Big Technology Podcast for a discussion of AI's growing risk profile, and what to do about it. Tune in to hear Hendricks explain why virology expertise in AI models is an immediate concern and how these systems might soon enable devastating hacks. We also cover intelligence explosion scenarios, the geopolitical implications of AI development, and why an international AI arms race could lead to faster development than the world can handle. Hit play for an insider's perspective on how governments and AI labs are wrestling with unprecedented technological power that could reshape global security.
Connect with Dan:
https://www.nationalsecurity.ai/
https://x.com/DanHendrycks
Learn more about your ad choices. Visit megaphone.fm/adchoices
National Security Strategy and AI Evals on the Eve of Superintelligence with Dan Hendrycks
This week on No Priors, Sarah is joined by Dan Hendrycks, director of the Center of AI Safety. Dan serves as an advisor to xAI and Scale AI. He is a longtime AI researcher, publisher of interesting AI evals such as "Humanity's Last Exam," and co-author of a new paper on National Security "Superintelligence Strategy" along with Scale founder-CEO Alex Wang and former Google CEO Eric Schmidt. They explore AI safety, geopolitical implications, the potential weaponization of AI, along with policy recommendations.
Sign up for new podcasts every week. Email feedback to show@no-priors.com
Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @DanHendrycks
Show Notes:
0:00 Introduction
0:36 Dan’s path to focusing on AI Safety
1:25 Safety efforts in large labs
3:12 Distinguishing alignment and safety
4:48 AI’s impact on national security
9:59 How might AI be weaponized?
14:43 Immigration policies for AI talent
17:50 Mutually assured AI malfunction
22:54 Policy suggestions for current administration
25:34 Compute security
30:37 Current state of evals
GELU, MMLU, & X-Risk Defense in Depth, with the Great Dan Hendrycks
Join Nathan for an expansive conversation with Dan Hendrycks, Executive Director of the Center for AI Safety and Advisor to Elon Musk's XAI. In this episode of The Cognitive Revolution, we explore Dan's groundbreaking work in AI safety and alignment, from his early contributions to activation functions to his recent projects on AI robustness and governance. Discover insights on representation engineering, circuit breakers, and tamper-resistant training, as well as Dan's perspectives on AI's impact on society and the future of intelligence. Don't miss this in-depth discussion with one of the most influential figures in AI research and safety.
Check out some of Dan's research papers:
MMLU: https://arxiv.org/abs/2009.03300
GELU: https://arxiv.org/abs/1606.08415
Machiavelli Benchmark: https://arxiv.org/abs/2304.03279
Circuit Breakers: https://arxiv.org/abs/2406.04313
Tamper Resistant Safeguards: https://arxiv.org/abs/2408.00761
Statement on AI Risk: https://www.safe.ai/work/statement-on-ai-risk
Apply to join over 400 Founders and Execs in the Turpentine Network: https://www.turpentinenetwork.co/
SPONSORS:
Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive.
LMNT: LMNT is a zero-sugar electrolyte drink mix that's redefining hydration and performance. Ideal for those who fast or anyone looking to optimize their electrolyte intake. Support the show and get a free sample pack with any purchase at https://drinklmnt.com/tcr.
Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution
Oracle: Oracle Cloud Infrastructure (OCI) is a single platform for your infrastructure, database, application development, and AI needs. OCI has four to eight times the bandwidth of other clouds; offers one consistent price, and nobody does data better than Oracle. If you want to do more and spend less, take a free test drive of OCI at https://oracle.com/cognitive
CHAPTERS:
(00:00:00) Teaser
(00:00:48) About the Show
(00:02:17) About the Episode
(00:05:41) Intro
(00:07:19) GELU Activation Function
(00:10:48) Signal Filtering
(00:12:46) Scaling Maximalism
(00:18:35) Sponsors: Shopify | LMNT
(00:22:03) New Architectures
(00:25:41) AI as Complex System
(00:32:35) The Machiavelli Benchmark
(00:34:10) Sponsors: Notion | Oracle
(00:37:20) Understanding MMLU Scores
(00:45:23) Reasoning in Language Models
(00:49:18) Multimodal Reasoning
(00:54:53) World Modeling and Sora
(00:57:07) Arc Benchmark and Hypothesis
(01:01:06) Humanity's Last Exam
(01:08:46) Benchmarks and AI Ethics
(01:13:28) Robustness and Jailbreaking
(01:18:36) Representation Engineering
(01:30:08) Convergence of Approaches
(01:34:18) Circuit Breakers
(01:37:52) Tamper Resistance
(01:49:10) Interpretability vs. Robustness
(01:53:53) Open Source and AI Safety
(01:58:16) Computational Irreducibility
(02:06:28) Neglected Approaches
(02:12:47) Truth Maxing and XAI
(02:19:59) AI-Powered Forecasting
(02:24:53) Chip Bans and Geopolitics
(02:33:30) Working at CAIS
(02:35:03) Extinction Risk Statement
(02:37:24) Outro
