ONTOLOGIK

Support us! https://www.patreon.com/mlst Dr. Walid Saba recently reviewed the book Machines Will Never Rule The World, which argues that strong AI is impossible. He acknowledges the complexity of modeling mental processes and language, as well as interactive dialogues, and questions the authors' use of "never." Despite his skepticism, he is impressed with recent developments in large language models, though he questions the extent of their success. We then discussed the successes of cognitive science. Walid believes that something has been achieved which many cognitive scientists would never accept, namely the ability to learn from data empirically. Keith agrees that this is a huge step, but notes that there is still much work to be done to get to the "other 5%" of accuracy. They both agree that the current models are too brittle and require much more data and parameters to get to the desired level of accuracy. Walid then expresses admiration for deep learning systems' ability to learn non-trivial aspects of language from ingesting text only. He argues that this is an "existential proof" of language competency and that it would be impossible for a group of luminaries such as Montague, Marvin Minsky, John McCarthy, and a thousand other bright engineers to replicate the same level of competency as we have now with LLMs. He then discusses the problem of semantics and pragmatics, as well as symbol grounding, and expresses skepticism about grounded meaning and embodiment. He believes that artificial intelligence should be used to solve real-world problems which require human intelligence but not believe that robots should be built to understand love or other subjective feelings. We discussed the unique properties of natural human language. Walid believes that the core unique property is the ability to do abductive reasoning, which is the process of reasoning to the best explanation or understanding. Keith adds that there are two types of abduction - one for generating hypotheses and one for justifying them. In both cases, abductive reasoning involves choosing from a set of plausible possibilities. Finally, we discussed the book "Machines Will Never Rule The World" and its argument that the current mathematics and technology is not enough to model complex systems. Walid agrees with the book's argument but is still optimistic that a new mathematics can be discovered. Keith suggests the possibility of an AGI discovering the mathematics to create itself. They also discussed how the book could serve as a reminder to temper the hype surrounding AI and to focus on exploration, creativity, and daring ideas. Walid ended by stressing the importance of science, noting that engineers should play within the Venn diagrams drawn by scientists, rather than trying to hack their way through it. Transcript: https://share.descript.com/view/BFQb5iaegJC Discord: https://discord.gg/aNPkGUQtc5 YT: https://youtu.be/IMnWAuoucjo TOC: [00:00:00] Intro [00:06:52] Walid's change of heart on DL/LLMs and on the skeptics like Gary Marcus [00:22:52] Symbol Grounding [00:32:26] On Montague [00:40:41] On Abduction [00:50:54] Language of thought [00:56:08] Why machines will never rule the world book review [01:20:06] Engineers should play in the scientists Venn Diagram! Panel; Dr. Tim Scarfe Dr. Keith Duggar Mark Mcguill

Walid Saba

Senior Scientist

SorceroONONTOLOGIK

Patreon: https://www.patreon.com/mlst Discord: https://discord.gg/HNnAwSduud YT version: https://youtu.be/pMtk-iUaEuQ Dr. Walid Saba is an old-school polymath. He has a background in cognitive psychology, linguistics, philosophy, computer science and logic and he’s is now a Senior Scientist at Sorcero. Walid is perhaps the most outspoken critic of BERTOLOGY, which is to say trying to solve the problem of natural language understanding with application of large statistical language models. Walid thinks this approach is cursed to failure because it’s analogous to memorising infinity with a large hashtable. Walid thinks that the various appeals to infinity by some deep learning researchers are risible. [00:00:00] MLST Housekeeping [00:08:03] Dr. Walid Saba Intro [00:11:56] AI Cannot Ignore Symbolic Logic, and Here’s Why [00:23:39] Main show - Proposition: Statistical learning doesn't work [01:04:44] Discovering a sorting algorithm bottom-up is hard [01:17:36] The axioms of nature (universal cognitive templates) [01:31:06] MLPs are locality sensitive hashing tables References; The Missing Text Phenomenon, Again: the case of Compound Nominals https://ontologik.medium.com/the-missing-text-phenomenon-again-the-case-of-compound-nominals-abb6ece3e205 A Spline Theory of Deep Networks https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf The Defeat of the Winograd Schema Challenge https://arxiv.org/pdf/2201.02387.pdf Impact of Pretraining Term Frequencies on Few-Shot Reasoning https://twitter.com/yasaman_razeghi/status/1495112604854882304?s=21 https://arxiv.org/abs/2202.07206 AI Cannot Ignore Symbolic Logic, and Here’s Why https://medium.com/ontologik/ai-cannot-ignore-symbolic-logic-and-heres-why-1f896713525b Learnability can be undecidable http://gtts.ehu.es/German/Docencia/1819/AC/extras/s42256-018-0002-3.pdf Scaling Language Models: Methods, Analysis & Insights from Training Gopher https://arxiv.org/pdf/2112.11446.pdf DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning https://arxiv.org/abs/2006.08381 On the Measure of Intelligence [Chollet] https://arxiv.org/abs/1911.01547 A Formal Theory of Commonsense Psychology: How People Think People Think https://www.amazon.co.uk/Formal-Theory-Commonsense-Psychology-People/dp/1107151007 Continuum hypothesis https://en.wikipedia.org/wiki/Continuum_hypothesis Gödel numbering + completness theorems https://en.wikipedia.org/wiki/G%C3%B6del_numbering https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems Concepts: Where Cognitive Science Went Wrong [Jerry A. Fodor] https://oxford.universitypressscholarship.com/view/10.1093/0198236360.001.0001/acprof-9780198236368

Walid Saba

Senior Scientist

SorceroONONTOLOGIK

It has been over three decades since the statistical revolution overtook AI by a storm and over two decades since deep learning (DL) helped usher the latest resurgence of artificial intelligence (AI). However, the disappointing progress in conversational agents, NLU, and self-driving cars, has made it clear that progress has not lived up to the promise of these empirical and data-driven methods. DARPA has suggested that it is time for a third wave in AI, one that would be characterized by hybrid models – models that combine knowledge-based approaches with data-driven machine learning techniques. Joining us on this panel discussion is polymath and linguist Walid Saba - Co-founder ONTOLOGIK.AI, Gadi Singer - VP & Director, Cognitive Computing Research, Intel Labs and J. Mark Bishop - Professor of Cognitive Computing (Emeritus), Goldsmiths, University of London and Scientific Adviser to FACT360. Moderated by Dr. Keith Duggar and Dr. Tim Scarfe https://www.linkedin.com/in/gadi-singer/ https://www.linkedin.com/in/walidsaba/ https://www.linkedin.com/in/profjmarkbishop/ #machinelearning #artificialintelligence

Walid Saba

Senior Scientist

SorceroONONTOLOGIK

Gadi Singer

VP & Director

Intel

J. Mark Bishop

Professor

Goldsmiths, University of London

#machinelearning This week Dr. Tim Scarfe, Dr. Keith Duggar and Yannic Kilcher speak with veteran NLU expert Dr. Walid Saba. Walid is an old-school AI expert. He is a polymath, a neuroscientist, psychologist, linguist, philosopher, statistician, and logician. He thinks the missing information problem and lack of a typed ontology is the key issue with NLU, not sample efficiency or generalisation. He is a big critic of the deep learning movement and BERTology. We also cover GPT-3 in some detail in today's session, covering Luciano Floridi's recent article "GPT‑3: Its Nature, Scope, Limits, and Consequences" and a commentary on the incredible power of GPT-3 to perform tasks with just a few examples including the Yann LeCun commentary on Facebook and Hackernews. Time stamps on the YouTube version 0:00:00 Walid intro 00:05:03 Knowledge acquisition bottleneck 00:06:11 Language is ambiguous 00:07:41 Language is not learned 00:08:32 Language is a formal language 00:08:55 Learning from data doesn’t work 00:14:01 Intelligence 00:15:07 Lack of domain knowledge these days 00:16:37 Yannic Kilcher thuglife comment 00:17:57 Deep learning assault 00:20:07 The way we evaluate language models is flawed 00:20:47 Humans do type checking 00:23:02 Ontologic 00:25:48 Comments On GPT3 00:30:54 Yann lecun and reddit 00:33:57 Minds and machines - Luciano 00:35:55 Main show introduction 00:39:02 Walid introduces himself 00:40:20 science advances one funeral at a time 00:44:58 Deep learning obsession syndrome and inception 00:46:14 BERTology / empirical methods are not NLU 00:49:55 Pattern recognition vs domain reasoning, is the knowledge in the data 00:56:04 Natural language understanding is about decoding and not compression, it's not learnable. 01:01:46 Intelligence is about not needing infinite amounts of time 01:04:23 We need an explicit ontological structure to understand anything 01:06:40 Ontological concepts 01:09:38 Word embeddings 01:12:20 There is power in structure 01:15:16 Language models are not trained on pronoun disambiguation and resolving scopes 01:17:33 The information is not in the data 01:19:03 Can we generate these rules on the fly? Rules or data? 01:20:39 The missing data problem is key 01:21:19 Problem with empirical methods and lecunn reference 01:22:45 Comparison with meatspace (brains) 01:28:16 The knowledge graph game, is knowledge constructed or discovered 01:29:41 How small can this ontology of the world be? 01:33:08 Walids taxonomy of understanding 01:38:49 The trend seems to be, less rules is better not the othe way around? 01:40:30 Testing the latest NLP models with entailment 01:42:25 Problems with the way we evaluate NLP 01:44:10 Winograd Schema challenge 01:45:56 All you need to know now is how to build neural networks, lack of rigour in ML research 01:50:47 Is everything learnable 01:53:02 How should we elevate language systems? 01:54:04 10 big problems in language (missing information) 01:55:59 Multiple inheritance is wrong 01:58:19 Language is ambiguous 02:01:14 How big would our world ontology need to be? 02:05:49 How to learn more about NLU 02:09:10 AlphaGo Walid's blog: https://medium.com/@ontologik LinkedIn: https://www.linkedin.com/in/walidsaba/

Walid Saba

Senior Scientist

SorceroONONTOLOGIK