It's Crunch Time: Ajeya Cotra on RSI & AI-Powered AI Safety Work, from the 80,000 Hours Podcast
This cross-post from the 80,000 Hours podcast features Ajeya Cotra in conversation with Rob Wiblin about AI timelines, recursive self-improvement, and the “crunch time” window when AI could rapidly accelerate its own development. Ajeya explains why widespread, compounding automation may face fewer bottlenecks than many expect, and what that could mean for the world by 2050. They also discuss transparency, early warning systems, and the emerging strategy of using each generation of AI to align and control its successors.
LINKS:
Ajeya Cotra podcast episode
Sponsors:
AvePoint:
AvePoint is building the control layer for AI agents so you can securely govern, audit, and recover every action at scale. Design trusted agentic outcomes from day one at https://avpt.co/tcr
VCX:
VCX, by Fundrise, is the public ticker for private tech, giving everyday investors access to high-growth private companies in AI, space, defense tech, and more. Learn how to invest at https://getvcx.com
Claude:
Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr
Tasklet:
Build your own Cognitive Revolution monitoring agent in one click.
Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai
CHAPTERS:
(00:00) About the Episode
(04:17) AI inside safety plans
(06:49) AGI growth expectations (Part 1)
(18:32) Sponsors: AvePoint | VCX
(20:54) AGI growth expectations (Part 2)
(21:56) Disagreement and measurement (Part 1)
(37:28) Sponsors: Claude | Tasklet
(41:19) Disagreement and measurement (Part 2)
(41:19) Transparency before takeoff
(58:00) Redirecting AI labor
(01:09:06) Defense plans and limits
(01:25:26) Pausing versus redirecting
(01:35:22) Open Phil and compute
(01:50:16) Bottlenecks and preparation
(01:57:42) From research to grants
(02:13:03) Burnout and sabbatical
(02:23:22) What EA offered
(02:36:43) EA and religion
(02:47:56) Next career steps
(02:57:35) EA's future niche
(03:08:15) Episode Outro
(03:12:11) Outro
PRODUCED BY:
https://aipodcast.ing
Joe Carlsmith — Preventing an AI takeover
Chatted with Joe Carlsmith about whether we can trust power/techno-capital, how to not end up like Stalin in our urge to control the future, gentleness towards the artificial Other, and much more.
Check out Joe's sequence on Otherness and Control in the Age of AGI here.
Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.
Sponsors:
- Bland.ai is an AI agent that automates phone calls in any language, 24/7. Their technology uses "conversational pathways" for accurate, versatile communication across sales, operations, and customer support. You can try Bland yourself by calling 415-549-9654. Enterprises can get exclusive access to their advanced model at bland.ai/dwarkesh.
- Stripe is financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.
If you’re interested in advertising on the podcast, check out this page.
Timestamps:
(00:00:00) - Understanding the Basic Alignment Story
(00:44:04) - Monkeys Inventing Humans
(00:46:43) - Nietzsche, C.S. Lewis, and AI
(1:22:51) - How should we treat AIs
(1:52:33) - Balancing Being a Humanist and a Scholar
(2:05:02) - Explore exploit tradeoffs and AI
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
AI Ethics at Code 2023
Platformer's Casey Newton moderates a conversation at Code 2023 on ethics in artificial intelligence, with Ajeya Cotra, Senior Program Officer at Open Philanthropy, and Helen Toner, Director of Strategy at Georgetown University’s Center for Security and Emerging Technology. The panel discusses the risks and rewards of the technology, as well as best practices and safety measures.
Recorded on September 27th in Los Angeles.
Learn more about your ad choices. Visit podcastchoices.com/adchoices
The Surgeon General’s Social Media Warning + A.I.’s Existential Risks
The U.S. surgeon general, Dr. Vivek Murthy, says social media poses a “profound risk of harm” to young people. Why do some in the tech industry disagree?
Then, Ajeya Cotra, an A.I. researcher, on how A.I. could lead to a doomsday scenario.
Plus: Pass the hat. Kevin and Casey play a game they call HatGPT.
On today’s episode:
Ajeya Cotra is a senior research analyst at Open Philanthropy
Additional reading:
The surgeon general issued an advisory about the risks of social media for young people.
Ajeya Cotra has researched the existential risks that A.I. poses unless countermeasures are taken.
Binance commingled customer funds and company revenue, former insiders told Reuters.
BuzzFeed announced Botatouille, an A.I.-powered kitchen assistant.
A Twitter bug caused the platform to restore deleted tweets.
Gov. Ron DeSantis of Florida announced his presidential campaign in a Twitter Spaces event rife with glitches.
Two former rivals, Uber and Waymo, are teaming up to bring driverless ride-hailing to Phoenix.
Subscribe today at nytimes.com/podcasts or on Apple Podcasts and Spotify. You can also subscribe via your favorite podcast app here https://www.nytimes.com/activate-access/audio?source=podcatcher. For more podcasts and narrated articles, download The New York Times app at nytimes.com/app.
Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
Holden Karnofsky — History's most important century
Holden Karnofsky is the co-CEO of Open Philanthropy and co-founder of GiveWell. He is also the author of one of the most interesting blogs on the internet, Cold Takes.
We discuss:
* Are we living in the most important century?
* Does he regret OpenPhil’s 30 million dollar grant to OpenAI in 2016?
* How does he think about AI, progress, digital people, & ethics?
Highly recommend!
Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.
Timestamps
(0:00:00) - Intro
(0:00:58) - The Most Important Century
(0:06:44) - The Weirdness of Our Time
(0:21:20) - The Industrial Revolution
(0:35:40) - AI Success Scenario
(0:52:36) - Competition, Innovation , & AGI Bottlenecks
(1:00:14) - Lock-in & Weak Points
(1:06:04) - Predicting the Future
(1:20:40) - Choosing Which Problem To Solve
(1:26:56) - $30M OpenAI Investment
(1:30:22) - Future Proof Ethics
(1:37:28) - Integrity vs Utilitarianism
(1:40:46) - Bayesian Mindset & Governance
(1:46:56) - Career Advice
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
Joseph Carlsmith - Utopia, AI, & Infinite Ethics
Joseph Carlsmith is a senior research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford.
We discuss utopia, artificial intelligence, computational power of the brain, infinite ethics, learning from the fact that you exist, perils of futurism, and blogging.
Watch on YouTube. Listen on Spotify, Apple Podcasts, etc.
Episode website + Transcript here.
Follow Joseph on Twitter. Follow me on Twitter.
Subscribe to find out about future episodes!
Timestamps
(0:00:06) - Introduction
(0:02:53) - How to Define a Better Future?
(0:09:19) - Utopia
(0:25:12) - Robin Hanson’s EMs
(0:27:35) - Human Computational Capacity
(0:34:15) - FLOPS to Emulate Human Cognition?
(0:40:15) - Infinite Ethics
(1:00:51) - SIA vs SSA
(1:17:53) - Futurism & Unreality
(1:23:36) - Blogging & Productivity
(1:28:43) - Book Recommendations
(1:30:04) - Conclusion
Please share if you enjoyed this episode! Helps out a ton!
Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe
