Proactive Agents for the Web with Devi Parikh - #756
Today, we're joined by Devi Parikh, co-founder and co-CEO of Yutori, to discuss browser use models and a future where we interact with the web through proactive, autonomous agents. We explore the technical challenges of creating reliable web agents, the advantages of visually-grounded models that operate on screenshots rather than the browser’s more brittle document object model, or DOM, and why this counterintuitive choice has proven far more robust and generalizable for handling complex web interfaces. Devi also shares insights into Yutori’s training pipeline, which has evolved from supervised fine-tuning to include rejection sampling and reinforcement learning. Finally, we discuss how Yutori’s “Scouts” agents orchestrate multiple tools and sub-agents to handle complex queries, the importance of background, "ambient" operation for these systems, and what the path looks like from simple monitoring to full task automation on the web.
The complete show notes for this episode can be found at https://twimlai.com/go/756.
AI agents for your digital chores
Ryan welcomes Dhruv Batra, co-founder and chief scientist at Yutori, to explore the future of AI agents, how AI usage is changing the way people interact with advertisements and the web as a whole, and the challenges that proactive AI agents may face when being integrated into workflows and personal internet use.
Episode notes:
Yutori is building AI agents that can reliably handle everyday digital tasks on your behalf on the web.
Connect with Dhruv via his website.
Congrats to the winner of today’s Populist badge, user Don Kirkby, who earned it with their answer to Find all references to an object in python.
TRANSCRIPT
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
The Timeline for Realistic 4-D: Devi Parikh from Meta on Research Hurdles for Generative AI in Video and Multimodality
Video dominates modern media consumption, but video creation is still expensive and difficult. AI-generated and edited video is a holy grail of democratized creative expression. This week on No Priors, Sarah Guo and Elad Gil sit down with Devi Parikh. She is a Research Director in Generative AI at Meta and an Associate Professor in the School of Interactive Computing at Georgia Tech. Her work focuses on multimodality and AI for images, audio and video. Recently, she worked on Make a Video 3D, also called MAV3D, which creates animations from text prompts. She is also a talented AI-generated and analog artist herself.
Elad, Sarah and Devi talk about what’s exciting in computer vision, what’s blocking researchers from fully immersive Generative 4-D, and AI controllability.
No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode.
Show Links:
Devi Parikh - Google Scholar
Text-To-4D Dynamic Scene Generation named MAV3D (Make-A-Video3D)
Full Research Paper
Website with examples of image to 4 D generation
Devi’s Substack
Sign up for new podcasts every week. Email feedback to show@no-priors.com
Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @DeviParikh
Show Notes:
(0:00:06) - Democratizing Creative Expression With AI-Generated Video
(0:08:31) - Challenges in Video Generation Research
(0:15:57) - Challenges and Implications of Video Processing
(0:20:43) - Control and Multi-Modal Inputs in Video
(0:25:50) - Audio's Role in Visual Content
(0:39:00) - Don't Self-Select & Devi’s tips for young researchers
Building Maps and Spatial Awareness in Blind AI Agents with Dhruv Batra - #629
Today we continue our coverage of ICLR 2023 joined by Dhruv Batra, an associate professor at Georgia Tech and research director of the Fundamental AI Research (FAIR) team at META. In our conversation, we discuss Dhruv’s work on the paper Emergence of Maps in the Memories of Blind Navigation Agents, which won an Outstanding Paper Award at the event. We explore navigation with multilayer LSTM and the question of whether embodiment is necessary for intelligence. We delve into the Embodiment Hypothesis and the progress being made in language models and caution on the responsible use of these models. We also discuss the history of AI and the importance of using the right data sets in training. The conversation explores the different meanings of "maps" across AI and cognitive science fields, Dhruv’s experience in navigating mapless systems, and the early discovery stages of memory representation and neural mechanisms.
The complete show notes for this episode can be found at https://twimlai.com/go/629
Human-AI Collaboration for Creativity with Devi Parikh - #399
Today we’re joined by Devi Parikh, Associate Professor at the School of Interactive Computing at Georgia Tech, and research scientist at Facebook AI Research (FAIR). In our conversation, we touch on Devi’s definition of creativity, explore multiple ways that AI could impact the creative process for artists, and help humans become more creative. We investigate tools like casual creator for preference prediction, neuro-symbolic generative art, and visual journaling.