Allen Institute for Artificial Intelligence

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)/Jan 23, 2023/1h 46m

AI Trends 2023: Natural Language Proc - ChatGPT, GPT-4 and Cutting Edge Research with Sameer Singh - #613

Today we continue our AI Trends 2023 series joined by Sameer Singh, an associate professor in the department of computer science at UC Irvine and fellow at the Allen Institute for Artificial Intelligence (AI2). In our conversation with Sameer, we focus on the latest and greatest advancements and developments in the field of NLP, starting out with one that took the internet by storm just a few short weeks ago, ChatGPT. We also explore top themes like decomposed reasoning, causal modeling in NLP, and the need for “clean” data. We also discuss projects like HuggingFace’s BLOOM, the debacle that was the Galactica demo, the impending intersection of LLMs and search, use cases like Copilot, and of course, we get Sameer’s predictions for what will happen this year in the field. The complete show notes for this episode can be found at twimlai.com/go/613.

Professor

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)/Nov 7, 2022/53m

The Evolution of the NLP Landscape with Oren Etzioni - #598

Today friend of the show and esteemed guest host John Bohannon is back with another great interview, this time around joined by Oren Etzioni, former CEO of the Allen Institute for AI, where he is currently an advisor. In our conversation with Oren, we discuss his philosophy as a researcher and how that has manifested in his pivot to institution builder. We also explore his thoughts on the current landscape of NLP, including the emergence of LLMs and the hype being built up around AI systems from folks like Elon Musk. Finally, we explore some of the research coming out of AI2, including Semantic Scholar, an AI-powered research tool analogous to arxiv, and the somewhat controversial Delphi project, a research prototype designed to model people’s moral judgments on a variety of everyday situations.

Oren Etzioni

Advisor and Former CEO

Machine Learning Street Talk (MLST)/Apr 7, 2022/56m

#73 - YASAMAN RAZEGHI & Prof. SAMEER SINGH - NLP benchmarks

Patreon: https://www.patreon.com/mlst Discord: https://discord.gg/ESrGqhf5CB YT version: https://youtu.be/RzGaI7vXrkk This week we speak with Yasaman Razeghi and Prof. Sameer Singh from UC Urvine. Yasaman recently published a paper called Impact of Pretraining Term Frequencies on Few-Shot Reasoning where she demonstrated comprehensively that large language models only perform well on reasoning tasks because they memorise the dataset. For the first time she showed the accuracy was linearly correlated to the occurance rate in the training corpus, something which OpenAI should have done in the first place! We also speak with Sameer who has been a pioneering force in the area of machine learning interpretability for many years now, he created LIME with Marco Riberio and also had his hands all over the famous Checklist paper and many others. We also get into the metric obsession in the NLP world and whether metrics are one of the principle reasons why we are failing to make any progress in NLU. [00:00:00] Impact of Pretraining Term Frequencies on Few-Shot Reasoning [00:14:59] Metrics [00:18:55] Definition of reasoning [00:25:12] Metrics (again) [00:28:52] On true believers [00:33:04] Sameers work on model explainability / LIME [00:36:58] Computational irreducability [00:41:07] ML DevOps and Checklist [00:45:58] Future of ML devops [00:49:34] Thinking about future Prof. Sameer Singh https://sameersingh.org/ Yasaman Razeghi https://yasamanrazeghi.com/ References; Impact of Pretraining Term Frequencies on Few-Shot Reasoning [Razeghi et al with Singh] https://arxiv.org/pdf/2202.07206.pdf Beyond Accuracy: Behavioral Testing of NLP Models with CheckList [Riberio et al with Singh] https://arxiv.org/pdf/2005.04118.pdf “Why Should I Trust You?” Explaining the Predictions of Any Classifier (LIME) [Riberio et al with Singh] https://arxiv.org/abs/1602.04938 Tim interviewing LIME Creator Marco Ribeiro in 2019 https://www.youtube.com/watch?v=6aUU-Ob4a8I Tim video on LIME/SHAP on his other channel https://www.youtube.com/watch?v=jhopjN08lTM Our interview with Christoph Molar https://www.youtube.com/watch?v=0LIACHcxpHU Interpretable Machine Learning book @ChristophMolnar https://christophm.github.io/interpretable-ml-book/ Machine Teaching: A New Paradigm for Building Machine Learning Systems [Simard] https://arxiv.org/abs/1707.06742 Whimsical notes on machine teaching https://whimsical.com/machine-teaching-Ntke9EHHSR25yHnsypHnth Gopher paper (Deepmind) https://www.deepmind.com/blog/language-modelling-at-scale-gopher-ethical-considerations-and-retrieval https://arxiv.org/pdf/2112.11446.pdf EleutherAI https://www.eleuther.ai/ https://github.com/kingoflolz/mesh-transformer-jax/ https://pile.eleuther.ai/ A Theory of Universal Artificial Intelligence based on Algorithmic Complexity [Hutter] https://arxiv.org/pdf/cs/0004001.pdf

Yasaman Razeghi

PhD Student

Professor

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)/Jan 7, 2021/1h 22m

Trends in Natural Language Processing with Sameer Singh - #445

Today we continue the 2020 AI Rewind series, joined by friend of the show Sameer Singh, an Assistant Professor in the Department of Computer Science at UC Irvine. We last spoke with Sameer at our Natural Language Processing office hours back at TWIMLfest, and was the perfect person to help us break down 2020 in NLP. Sameer tackles the review in 4 main categories, Massive Language Modeling, Fundamental Problems with Language Models, Practical Vulnerabilities with Language Models, and Evaluation. We also explore the impact of GPT-3 and Transformer models, the intersection of vision and language models, and the injection of causal thinking and modeling into language models, and much more. The complete show notes for this episode can be found at twimlai.com/go/445.

Professor

The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)/Sep 3, 2020/42m

Beyond Accuracy: Behavioral Testing of NLP Models with Sameer Singh - #406

Today we’re joined by Sameer Singh, an assistant professor in the department of computer science at UC Irvine. Sameer’s work centers on large-scale and interpretable machine learning applied to information extraction and natural language processing. We caught up with Sameer right after he was awarded the best paper award at ACL 2020 for his work on Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In our conversation, we explore CheckLists, the task-agnostic methodology for testing NLP models introduced in the paper. We also discuss how well we understand the cause of pitfalls or failure modes in deep learning models, Sameer’s thoughts on embodied AI, and his work on the now famous LIME paper, which he co-authored alongside Carlos Guestrin. The complete show notes for this episode can be found at twimlai.com/go/406.

Professor

Practical AI/Feb 10, 2020/43m

AI-powered scientific exploration and discovery

Daniel and Chris explore Semantic Scholar with Doug Raymond of the Allen Institute for Artificial Intelligence. Semantic Scholar is an AI-backed search engine that uses machine learning, natural language processing, and machine vision to surface relevant information from scientific papers. Featuring: Douglas Raymond – GitHub, LinkedIn, X Chris Benson – Website, GitHub, LinkedIn, X Daniel Whitenack – Website, GitHub, X Show Notes: Semantic Scholar Allen Institute for Artificial Intelligence ELMo SciBERT: A Pretrained Language Model for Scientific Text Semantic Scholar API Upcoming Events: Register for upcoming webinars here!

Douglas Raymond

Engineering Manager

Practical AI/Jul 19, 2019/53m

AI code that facilitates good science

We’re talking with Joel Grus, author of Data Science from Scratch, 2nd Edition, senior research engineer at the Allen Institute for AI (AI2), and maintainer of AllenNLP. We discussed Joel’s book, which has become a personal favorite of the hosts, and why he decided to approach data science and AI “from scratch.” Joel also gives us a glimpse into AI2, an introduction to AllenNLP, and some tips for writing good research code. This episode is packed full of reproducible AI goodness! Sponsors: DigitalOcean – The simplest cloud platform for developers and teams Whether you’re running one virtual machine or ten thousand, makes managing your infrastructure too easy. Get started for free with a $50 credit. Learn more at do.co/changelog. DataEngPodcast – A podcast about data engineering and modern data infrastructure. Brain Science – For the curious! Brain Science is our new podcast exploring the inner-workings of the human brain to understand behavior change, habit formation, mental health, and being human. It’s Brain Science applied — not just how does the brain work, but how do we apply what we know about the brain to transform our lives. Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com. Featuring: Joel Grus – X Chris Benson – Website, GitHub, LinkedIn, X Daniel Whitenack – Website, GitHub, X Show Notes: Allen AI AllenNLP AllenNLP demo website Writing Code for NLP Research I don’t like Notebooks Joel’s website Adversarial learning podcast Books “Data Science from Scratch” by Joel Grus Upcoming Events: Register for upcoming webinars here!

Joel Grus

Senior Research Engineer