AI Trends 2023: Natural Language Proc - ChatGPT, GPT-4 and Cutting Edge Research with Sameer Singh - #613
Today we continue our AI Trends 2023 series joined by Sameer Singh, an associate professor in the department of computer science at UC Irvine and fellow at the Allen Institute for Artificial Intelligence (AI2). In our conversation with Sameer, we focus on the latest and greatest advancements and developments in the field of NLP, starting out with one that took the internet by storm just a few short weeks ago, ChatGPT. We also explore top themes like decomposed reasoning, causal modeling in NLP, and the need for “clean” data. We also discuss projects like HuggingFace’s BLOOM, the debacle that was the Galactica demo, the impending intersection of LLMs and search, use cases like Copilot, and of course, we get Sameer’s predictions for what will happen this year in the field.
The complete show notes for this episode can be found at twimlai.com/go/613.
#302 – Richard Haier: IQ Tests, Human Intelligence, and Group Differences
Richard Haier is a psychologist specializing in the science of human intelligence. Please support this podcast by checking out our sponsors:
– Calm: https://calm.com/lex to get 40% off
– Linode: https://linode.com/lex to get $100 free credit
– BiOptimizers: http://www.magbreakthrough.com/lex to get 10% off
– SimpliSafe: https://simplisafe.com/lex and use code LEX
– MasterClass: https://masterclass.com/lex to get 15% off
EPISODE LINKS:
Richard’s Twitter: https://twitter.com/rjhaier
Richard’s Website: https://richardhaier.com/
Documents & Articles:
1. Child IQ and survival to 79: https://ncbi.nlm.nih.gov/pmc/articles/PMC5491698/
2. Study of Mathematically Precocious Youth: https://my.vanderbilt.edu/smpy/files/2013/02/DoingPsychScience2006.pdf
Books:
1. The Neuroscience of Intelligence: https://amzn.to/3n50DcC
2. The Book of Five Rings: https://amzn.to/3y4Xcc6
3. The Rise and Fall of the Third Reich: https://amzn.to/3zPAW7q
4. Flowers for Algernon: https://amzn.to/3OfRKZS
5. The Bell Curve: https://amzn.to/3Ng4RJe
6. The Mismeasure of Man: https://amzn.to/3N9IkxB
7. Human Diversity: https://amzn.to/3O7Trsc
8. Facing Reality: https://amzn.to/3bfzqkX
PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
YouTube Full Episodes: https://youtube.com/lexfridman
YouTube Clips: https://youtube.com/lexclips
SUPPORT & CONNECT:
– Check out the sponsors above, it’s the best way to support this podcast
– Support on Patreon: https://www.patreon.com/lexfridman
– Twitter: https://twitter.com/lexfridman
– Instagram: https://www.instagram.com/lexfridman
– LinkedIn: https://www.linkedin.com/in/lexfridman
– Facebook: https://www.facebook.com/lexfridman
– Medium: https://medium.com/@lexfridman
OUTLINE:
Here’s the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time.
(00:00) – Introduction
(08:06) – Measuring human intelligence
(22:34) – IQ tests
(45:23) – College entrance exams
(53:59) – Genetics
(59:58) – Enhancing intelligence
(1:07:27) – The Bell Curve
(1:19:58) – Race differences
(1:39:11) – Bell curve criticisms
(1:48:21) – Intelligence and life success
(1:57:57) – Flynn effect
(2:02:49) – Nature vs nuture
(2:29:42) – Testing artificial intelligence
(2:41:46) – Advice
(2:45:53) – Mortality
#293 – Donald Hoffman: Reality is an Illusion – How Evolution Hid the Truth
Donald Hoffman is a cognitive scientist at UC Irvine and author of The Case Against Reality. Please support this podcast by checking out our sponsors:
– Calm: https://calm.com/lex to get 40% off
– LMNT: https://drinkLMNT.com/lex to get free sample pack
– InsideTracker: https://insidetracker.com/lex to get 20% off
– MasterClass: https://masterclass.com/lex to get 15% off
– Indeed: https://indeed.com/lex to get $75 credit
EPISODE LINKS:
Donald’s Twitter: https://twitter.com/donalddhoffman
Donald’s Website: http://cogsci.uci.edu/~ddhoff/
Documents & Articles:
1. Could a Neuroscientist Understand a Microprocessor?: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005268
2. Conscious Agent Networks: https://chrisfieldsresearch.com/CA-circuits-CSR-rev2.pdf
3. The Einstein-Podolsky-Rosen Argument in Quantum Theory: https://plato.stanford.edu/entries/qt-epr/
Books:
1. The Case Against Reality: https://amzn.to/3MhW4Wt
2. Vision: https://amzn.to/3Q4ibTm
PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
YouTube Full Episodes: https://youtube.com/lexfridman
YouTube Clips: https://youtube.com/lexclips
SUPPORT & CONNECT:
– Check out the sponsors above, it’s the best way to support this podcast
– Support on Patreon: https://www.patreon.com/lexfridman
– Twitter: https://twitter.com/lexfridman
– Instagram: https://www.instagram.com/lexfridman
– LinkedIn: https://www.linkedin.com/in/lexfridman
– Facebook: https://www.facebook.com/lexfridman
– Medium: https://medium.com/@lexfridman
OUTLINE:
Here’s the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time.
(00:00) – Introduction
(08:04) – Case against reality
(19:33) – Spacetime
(43:57) – Reductionism
(1:04:23) – Evolutionary game theory
(1:32:46) – Consciousness
(2:28:06) – Visualizing reality
(2:43:23) – Ephemerality of life
(2:51:48) – Simulation theory
(2:57:30) – Difficult ideas
(3:12:32) – Love
(3:16:07) – Advice for young people
(3:18:26) – Meaning of life
#73 - YASAMAN RAZEGHI & Prof. SAMEER SINGH - NLP benchmarks
Patreon: https://www.patreon.com/mlst
Discord: https://discord.gg/ESrGqhf5CB
YT version: https://youtu.be/RzGaI7vXrkk
This week we speak with Yasaman Razeghi and Prof. Sameer Singh from UC Urvine. Yasaman recently published a paper called Impact of Pretraining Term Frequencies on Few-Shot Reasoning where she demonstrated comprehensively that large language models only perform well on reasoning tasks because they memorise the dataset. For the first time she showed the accuracy was linearly correlated to the occurance rate in the training corpus, something which OpenAI should have done in the first place!
We also speak with Sameer who has been a pioneering force in the area of machine learning interpretability for many years now, he created LIME with Marco Riberio and also had his hands all over the famous Checklist paper and many others.
We also get into the metric obsession in the NLP world and whether metrics are one of the principle reasons why we are failing to make any progress in NLU.
[00:00:00] Impact of Pretraining Term Frequencies on Few-Shot Reasoning
[00:14:59] Metrics
[00:18:55] Definition of reasoning
[00:25:12] Metrics (again)
[00:28:52] On true believers
[00:33:04] Sameers work on model explainability / LIME
[00:36:58] Computational irreducability
[00:41:07] ML DevOps and Checklist
[00:45:58] Future of ML devops
[00:49:34] Thinking about future
Prof. Sameer Singh
https://sameersingh.org/
Yasaman Razeghi
https://yasamanrazeghi.com/
References;
Impact of Pretraining Term Frequencies on Few-Shot Reasoning [Razeghi et al with Singh]
https://arxiv.org/pdf/2202.07206.pdf
Beyond Accuracy: Behavioral Testing of NLP Models with CheckList [Riberio et al with Singh]
https://arxiv.org/pdf/2005.04118.pdf
“Why Should I Trust You?” Explaining the Predictions of Any Classifier (LIME) [Riberio et al with Singh]
https://arxiv.org/abs/1602.04938
Tim interviewing LIME Creator Marco Ribeiro in 2019
https://www.youtube.com/watch?v=6aUU-Ob4a8I
Tim video on LIME/SHAP on his other channel
https://www.youtube.com/watch?v=jhopjN08lTM
Our interview with Christoph Molar
https://www.youtube.com/watch?v=0LIACHcxpHU
Interpretable Machine Learning book @ChristophMolnar
https://christophm.github.io/interpretable-ml-book/
Machine Teaching: A New Paradigm for Building Machine Learning Systems [Simard]
https://arxiv.org/abs/1707.06742
Whimsical notes on machine teaching
https://whimsical.com/machine-teaching-Ntke9EHHSR25yHnsypHnth
Gopher paper (Deepmind)
https://www.deepmind.com/blog/language-modelling-at-scale-gopher-ethical-considerations-and-retrieval
https://arxiv.org/pdf/2112.11446.pdf
EleutherAI
https://www.eleuther.ai/
https://github.com/kingoflolz/mesh-transformer-jax/
https://pile.eleuther.ai/
A Theory of Universal Artificial Intelligence based on Algorithmic Complexity [Hutter]
https://arxiv.org/pdf/cs/0004001.pdf
Trends in Natural Language Processing with Sameer Singh - #445
Today we continue the 2020 AI Rewind series, joined by friend of the show Sameer Singh, an Assistant Professor in the Department of Computer Science at UC Irvine.
We last spoke with Sameer at our Natural Language Processing office hours back at TWIMLfest, and was the perfect person to help us break down 2020 in NLP. Sameer tackles the review in 4 main categories, Massive Language Modeling, Fundamental Problems with Language Models, Practical Vulnerabilities with Language Models, and Evaluation.
We also explore the impact of GPT-3 and Transformer models, the intersection of vision and language models, and the injection of causal thinking and modeling into language models, and much more.
The complete show notes for this episode can be found at twimlai.com/go/445.
Beyond Accuracy: Behavioral Testing of NLP Models with Sameer Singh - #406
Today we’re joined by Sameer Singh, an assistant professor in the department of computer science at UC Irvine.
Sameer’s work centers on large-scale and interpretable machine learning applied to information extraction and natural language processing. We caught up with Sameer right after he was awarded the best paper award at ACL 2020 for his work on Beyond Accuracy: Behavioral Testing of NLP Models with CheckList.
In our conversation, we explore CheckLists, the task-agnostic methodology for testing NLP models introduced in the paper. We also discuss how well we understand the cause of pitfalls or failure modes in deep learning models, Sameer’s thoughts on embodied AI, and his work on the now famous LIME paper, which he co-authored alongside Carlos Guestrin.
The complete show notes for this episode can be found at twimlai.com/go/406.