Databricks

We’re excited to have Databricks join us at AIEWF, among hundreds of the top companies in the AI Engineer ecosystem. LS subscribers can use their discount to get past the late bird pricing and access over $50k in sponsor offers! Everyone is still talking about Satya’s Frontier Ecosystems post, but few have actually built a (now $175 billion) frontier ecosystem and cloud like our guests today. From open-sourcing the layer above coding agents to rethinking databases for the agent era, Databricks cofounders Matei Zaharia and Reynold Xin are pushing the company beyond the lakehouse into a full data-and-AI operating system. In this episode, Matei and Reynold join swyx at the 2026 Data + AI Summit to unpack Omnigent, LTAP, Lakebase, agent security, open formats, Mosaic, and why databases may matter more than ever once AI agents start doing real work. We go deep on Omnigent: Databricks’ open-source meta-harness for combining, controlling, and sharing agents across Claude Code, Codex, Cursor, Pi, custom agents, and internal tools. Matei explains why coding agents and enterprise agents run into the same problems: portability, collaboration, session history, security, spend controls, and the need for a common API above every harness. Then Reynold walks through Databricks’ database dream: why CDC is brittle enough to joke that it means “continuous data corruption,” why HTAP has been the holy grail of database engineering, and why Databricks thinks LTAP gets most of the benefits by unifying the storage layer instead of collapsing every query engine. We also cover Databricks’ infrastructure scale, the culture behind rapid prototyping, the difference between tech and enterprise customers, Databricks vs Snowflake, whether vector databases should have ever existed, the Mosaic model strategy, Genie, AI Runtime, RL fine-tuning, and the thesis that traditional software gets rewritten once the data is in the right place and agents sit on top. Databricks began as a company for the big data era. The origination of Spark from the Berkeley AMPLab which eventually turned into the product Lakehouse convinced enterprises that they didn’t need a separate data lake, warehouse, ML platform, and governance layer. They just needed one open foundation where all of their data could live and be reasoned over. Since then a lot has changed, but data has only become more important. Data is no longer something you keep track of and analyze ad hoc, it’s the necessary context agents need in order to act. So the framing has shifted from “where do we put all of our data?” to “how do we expose the right slice of state, history, permissions, and business logic to an AI system at the exact moment it’s doing work?” If frontier model performance becomes commoditized, the durable advantage then becomes the company-specific context around them: proprietary data, governed access, operational state, transaction logs, workflows, and feedback loops. Which makes Databricks positioned perfectly. Now coming fresh off the Data + AI Summit 2026, the company is moving just as fast to keep up, announcing Genie One, Omnigent, LTAP, and many more, indicating a central mission in its newer work: Databricks is trying to become the operating system for enterprise agents. Models are getting good enough, but agents are only useful if they have the right context, permissions, memory, state, cost controls, and access to live business data. Fundamentally it appears that significantly better model performance in production is a systems problem, one that data guys like us are remarkably well prepared to solve! We discuss: * Why Databricks built Omnigent as a meta-harness above existing AI agents * Why coding agents and custom enterprise agents need the same infrastructure * The common API for agent sessions, files, streams, tool calls, and cancellation * Why persistent sessions, cloud sandboxes, sharing, search, and collaboration matter * Why Databricks open-sourced Omnigent instead of keeping it proprietary * Databricks’ internal agent usage, cloud sandboxes, and coding workflows * The scale of Databricks: 50–60 million virtual machines a day and exabytes before breakfast * Why agent security needs contextual and stateful policies * How an agent could read confidential docs, install a compromised npm package, and leak data * Why spend control matters when an agent can burn $500 reading logs * Startup opportunities around coding-agent analytics, quality, skills, and spend * LTAP, Lakebase, and why Databricks wants to rethink the database stack * OLTP vs OLAP, CDC, and why data pipelines break at 3 a.m. * Why HTAP has historically been the holy grail of database engineering * Why Databricks thinks LTAP is “HTAP done right” * How writing transactional data into column-oriented formats changes analytics * Why agents need live operational context from databases, not just telemetry * How Databricks prototypes strategic systems without endless process * Enterprise vs tech customers, governance, procurement, and DIY culture * The “second system syndrome” risk of rewriting a database engine * Building a database engine from a decade of traces and quadrillions of data points * Why vector databases should never have been a separate category * Why open formats and AI changed the race with Snowflake * The Mosaic story, DBRX, Genie, document parsing models, and specialized model training * Why model customization and RL fine-tuning may become mainstream * Why “get the data there, slap some agent on top” may rewrite traditional software Matei Zaharia * LinkedIn: https://www.linkedin.com/in/mateizaharia * X: https://x.com/matei_zaharia Reynold Xin * LinkedIn: https://www.linkedin.com/in/rxin * X: https://x.com/rxin Databricks * Website: https://www.databricks.com * X: https://x.com/databricks Timestamps 00:00:00 Introduction 00:02:22 Omnigent and the Agent Infrastructure Layer 00:08:39 Agent Clouds, Common APIs, and Open Source 00:16:52 Databricks Scale and Internal AI Workflows 00:18:03 Agent Security, Governance, and Spend Controls 00:27:34 LTAP and the Database Dream 00:30:30 CDC, HTAP, and Why Data Pipelines Break 00:34:05 Lakebase, Parquet, and Live Data for Agents 00:36:47 Databricks’ Culture of Fast Prototyping 00:43:40 The Dream Engine and Rewriting the Database Stack 00:51:02 Vector Databases, Query Engines, and LTAP 00:52:36 Databricks vs Snowflake 00:57:48 Mosaic, DBRX, Genie, and Specialized Models 01:03:11 Context, AI Runtime, and RL Fine-Tuning 01:06:15 Why Data + Agents May Rewrite Software 01:07:09 Closing Thoughts Transcript Introduction: Databricks, Data + AI Summit, and Founder Dynamics Swyx [00:00:00]: Matei and Reynold from Databricks, welcome to Latent Space. Reynold Xin [00:00:06]: Hey, thanks for having us. Swyx [00:00:07]: Yeah. Matei Zaharia [00:00:08]: Yeah, thanks so much. Swyx [00:00:09]: thanks for taking time out. You have your Databricks, Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 people Reynold Xin [00:00:17]: Yeah, it was Swyx [00:00:17]: in Berkeley Reynold Xin [00:00:18]: little meetup at Berkeley, I think Matei Zaharia [00:00:19]: Yeah Reynold Xin [00:00:19]: put together Matei Zaharia [00:00:20]: We were doing these tutorials and, yeah, just teach people Spark. Swyx [00:00:23]: Yeah. obviously now it’s like, I think like the headline number’s like 100,000 people around the world, 30,000 in person. Swyx [00:00:30]: it’s a crazy Matei Zaharia [00:00:31]: Amazing Swyx [00:00:31]: community. Well, I just saw the keynote. Swyx [00:00:35]: Ali’s just. Did was it obvious or that back when that Ali would be, like, such a great, like, CEO? Like Reynold Xin [00:00:42]: Oh Swyx [00:00:42]: such a great presenter? Reynold Xin [00:00:43]: What do you think? Matei Zaharia [00:00:44]: I think among our group of founders it was clear that, I think he’d be the best at this. Swyx [00:00:50]: Yeah. Matei Zaharia [00:00:50]: And yeah, it turned out great. And he’s, he’s ramped up on so many topics growing a company. He would just go in and, like, study it and, be talk to all the experts. Like, even if he can’t hire the person, learn enough about, like, finance and sales and whatever it was, and, and go from there. Yeah. Swyx [00:01:09]: Yeah. Reynold Xin [00:01:10]: he’s obviously very high IQ and a very high EQ, but it wasn’t. Like, Ali today is quite different from Ali from, like 10 years ago. I think there’s a lot of work that he put in to, get to this point. Swyx [00:01:20]: Yeah. no, to me the most appealing thing about him is that he’s funny. And like, it, it’s, it’ Matei Zaharia [00:01:26]: It’s true, yeah Swyx [00:01:26]: it’s hard to make jokes about, data warehouses Reynold Xin [00:01:30]: About serious topics Swyx [00:01:31]: security Matei Zaharia [00:01:32]: Yeah Swyx [00:01:32]: what have you. Matei Zaharia [00:01:33]: Oh, yeah. That’s for sure. Swyx [00:01:34]: Yeah. So you guys launched a whole bunch of things. I’ll, I’ll just name check briefly, the stuff because we’re not gonna cover everything. Omnigentt, your baby. LTAP, your baby, your dream engine. Swyx [00:01:47]: we’re also gonna cover Genie, cover CustomerLake, you acquired Panther Matei Zaharia [00:01:52]: Yeah Swyx [00:01:52]: Open Sharing, and there’s Unity AI Gateway. A lot of these, I think, like, are things that you would expect a Databricks to do. It’s, it’s like part of the roadmap. Everyone in your category has similar things. But I think, probably the two of you are leading the two most unique and differentiated initiatives Omnigent and the Agent Infrastructure Layer Swyx [00:02:09]: on, in the landscape. Maybe we’ll start with, Omnigentt we’ll, we’ll, we’ll, we’ll go into it. I do think that a lot of people are exploring this meta harness concept. Matei Zaharia [00:02:21]: Yeah, totally. Swyx [00:02:21]: What led you to it? Matei Zaharia [00:02:22]: Yeah. There were a couple of, like, converging lines, which I think is a good sign that you need something new. So on the one hand, there’s all the coding agent info internally. We have really great, dev infra team. they built something called Isaac, that’s like a wrapper on Claude Code and Codex, and, lets you use them either on the web in, like, sandboxes or, just on your dev machine or on your laptop or whatever. And then, they were adding all kinds of stuff there. And we saw all the more advanced engineers like, were building their own workflows with tons of agents, and they were building their own UIs and stuff on top or even on top of that. And then the other one was, like, us building agents. We ship this, like, data science agent called Genie on the research team, which I lead. We also build a lot of internal ones for various things, and then we have all the customer ones. And all of them running into this thing of like, “Oh, I need to switch model and harness and so on,” every few months. Plus the agent is, like, completely useless if you can’t share sessions with someone and have history and have search and all this, like, layer on top of it for collaboration. I thought a bit about it from both contexts and, at first people thought it was weird. They’re like, “Why are you doing coding agents and custom agents in the same thing?” But I said it’s, it’s the same problems and, you just wanna build the stuff that lets you deliver the agent, maybe control it if you care about security, and, make it portable across things. And then we prototyped some things as experiments. We saw, yeah, we can make it work, and then we built that for real. Swyx [00:04:06]: I’m wondering if this let’s call it architecture Matei Zaharia [00:04:11]: Yeah Swyx [00:04:11]: maps to anything in your careers in the past. like I always think about how a lot of things just tie back to operating systems. Swyx [00:04:18]: A lot of operating Matei Zaharia [00:04:19]: Yeah Swyx [00:04:20]: systems tie back to databases, Matei Zaharia [00:04:21]: So Swyx [00:04:21]: or the other way around Matei Zaharia [00:04:22]: so the thing, I do think it ties a lot to, like, network protocols, internet protocol. we also Swyx [00:04:29]: Communication between entities. Matei Zaharia [00:04:30]: Yeah. We did stuff with, like, data sharing also, which is probably, most viewers probably won’t know unless they’ Swyx [00:04:36]: Yeah, open protocol is the term. Matei Zaharia [00:04:37]: Yeah. Swyx [00:04:38]: Open sharing. Open sharing. Matei Zaharia [00:04:38]: Open sharing. Swyx [00:04:39]: Yes. Matei Zaharia [00:04:39]: Yeah. So it’s like you have a company, you maintain some table, like let’s say like a Walmart or something. They have like the, inventory and what’s been sold in each store. And then you also have suppliers, and they would love to produce more things and ship them, like, exactly the moment you need them. So they would love, like, real-time access to your table. So instead of like sending emails around or Excel sheets or phone calls, why can’t you share like a view of that table in real time with them? Then they query, they, join it with their data, and they decide what to send. So it’s one of these things where you, like you might ask like today since we can vibe code anything so fast, why do we even need to design like protocols or APIs or software? Why can’t you just vibe code things on demand? But for this type of interoperability where multiple parties that are moving at different speeds are building stuff and you still want some layer on top to coordinate, you do wanna design it and build it. So it reminds me of that, like agents talking to each other and, users talking to agents and tools. Agent Clouds, Cloud Sandboxes, and Keeping Sessions Alive Swyx [00:05:42]: Reynold, any other comments alternative viewpoints? Reynold Xin [00:05:46]: I think, by the way, we had a debate on exactly which set of benefits would, matter a lot, and I think around the time we decided to do this thing I was telling Matei, “Hey,” it just happened to be there’s a particular week that I was coding nonstop Swyx [00:06:00]: from the moment I woke up to, like, the moment I went to bed, I was, like, looking at my Claude sessions, my Codex sessions. And one of the things that was particularly annoying was having to keep my laptop open. Swyx [00:06:12]: I was driving to a doctor’s appointment, and I remember because I wanted to make sure the whole thing continues working. Matei Zaharia [00:06:18]: But by the way, it’s so comforting to hear you say that because I’m like, “I don’t know if I’m a clown and I’m doing this or like.” Swyx [00:06:25]: Yeah. Like honestly, I was driving and I was tethering my laptop to my phone. Matei Zaharia [00:06:29]: huh. Swyx [00:06:29]: Keeping it on the side. Whenever I hit a red light, I started looking at what’s going on my laptop. Matei Zaharia [00:06:35]: Yeah. Swyx [00:06:35]: And I just felt that was ridiculous. Matei Zaharia [00:06:37]: Yeah. Swyx [00:06:37]: It felt like we went back to the dark ages Matei Zaharia [00:06:39]: Yeah Swyx [00:06:40]: programming. the productivity you gain from all this coding age is amazing, but, yeah. Matei Zaharia [00:06:45]: Have you heard of cloud? Swyx [00:06:47]: Yeah. Swyx [00:06:48]: It was crazy to me. Matei Zaharia [00:06:49]: Oh, the thing you were working on was the sandboxes or was this before that? Swyx [00:06:52]: It was a sandbox. Matei Zaharia [00:06:53]: Okay. Swyx [00:06:54]: I was work Matei Zaharia [00:06:54]: So you were in Swyx [00:06:55]: So I was approaching from a very different angle. I wanted to, “Hey, we’re gonna have cloud sandboxes that doesn’t shut down. You can get one very quickly,” but not just for running agentic sessions. Matei Zaharia [00:07:06]: Yeah. Swyx [00:07:06]: It’s also for running development. So I was personally building that week, and through building that, I ran into all these issues, and then I wrote Matei Zaharia [00:07:15]: Yeah Swyx [00:07:15]: a document for Matei, it’s like, “Here’s my wish list of what the actual environment should do.” And I think he ended up almost implementing Matei Zaharia [00:07:22]: Yeah Swyx [00:07:22]: every single one of them. Matei Zaharia [00:07:23]: Yeah, I remember Reynolds saying, ‘cause my first prototype of this had just chats with your agent and he said, “I have to be able to open a shell, like my own shell and like list files and like tail them and stuff.” So Swyx [00:07:36]: So SSH into a mainframe. Matei Zaharia [00:07:37]: Yeah. it has that now. Swyx [00:07:39]: Tailing my log. Matei Zaharia [00:07:40]: Yeah. Matei Zaharia [00:07:41]: Yeah. Swyx [00:07:41]: And also another thing I think I asked was, I had. I still use cursor for the sole purpose of rendering markdown files. Matei Zaharia [00:07:48]: huh. Yes. Swyx [00:07:49]: So I said, “If you just give me a way to see my markdown files and render Matei Zaharia [00:07:53]: Yeah Swyx [00:07:53]: them properly, I don’t need a separate tool anymore.” Matei Zaharia [00:07:55]: Yeah. Swyx [00:07:56]: And I think you also built that in. Matei Zaharia [00:07:57]: Yeah, we, yeah, we did that, yeah. Yeah, we had a lot of engineers building, their own vibe coding setup. But then the other thing they all said is like, “Hey, I built something that’s amazing for me, but, like, no one else on the team can use it ‘cause I don’t have a server to collaborate.” And this is why we tried to set up, Omnigent, so you can have a server and have the security, set up in there. So, like log in with Google or whatever and, like securely share stuff. which. And that’s where we’ve seen a lot of other agents like hit things. Like people think they prototyped an awesome agent, but it’s not allowed to connect to like some really important data or whatever because of the security team. Omnigent Architecture, Open Source, and Common APIs Swyx [00:08:38]: Yeah. Matei Zaharia [00:08:38]: So yeah. Swyx [00:08:39]: Yeah. At this point, so for those watching along on YouTube, we’re gonna putting up a image of the structure here, and we can talk a little bit of the architecture. I think I just want to have people understand, ‘cause like when we’re talking about software, it can be very abstract and like here is what we’re talking about. You’ve worked out in open source this entire platform and there’s a runner component and server component with a uniform API that you’ve, you’ve figured out. any other element and obviously you can plug in all this, persistence layers and compute layers. This is a whole cloud. It’s an agent cloud. Matei Zaharia [00:09:12]: Yeah. It’s, it’s got these components to work with it. The, a lot of the action happens like on the machine where you deploy your agent too. So whatever you’ve got on there, you can run. But yeah, it’s, I think it’s the minimal thing you want to have hosted, like collaborative agents and to have that server. And one of the reasons we open sourced it is, anyone building agents, this gives them an app they can start with and customize, which we were seeing in Databricks too. Like someone would make a nice, agent app and then other teams would ask, “Oh, can I just use yours for my agent?” Swyx [00:09:45]: Yeah, I think we had like five or six different agentic frameworks Matei Zaharia [00:09:48]: Yeah Swyx [00:09:48]: built by every different team. They do all do more or less the same thing. Yeah, you need to. people wanna take something that works in Forkit, and you might as well have something open source. Yeah, which also was another question, which is interesting for Databricks. Like what do you choose to open source? What do you choose to make it proprietary? It’s in. this goes back to Spark, right? Matei Zaharia [00:10:05]: Yeah. Matei Zaharia [00:10:06]: One, so one of the reasons to open source something is if you think it’s a layer that will there’ll be some network effect, it’ll benefit from many, people collaborating, on it. So, for example, with Spark, I don’t know if when Spark came out, we also focused a lot on letting you have libraries on top. So like there used to be different Swyx [00:10:28]: Ecosystem Matei Zaharia [00:10:28]: distributed computing engines for like machine learning and graph computation. We said they should all be libraries that you can compose. And we made it super easy to add connectors to data sources too. And then we benefit because, we don’t have the time to write like connectors to like, 1,000 like different databases and file formats, but we can just use the ones people make, and of course they benefit from joining, this thing. So that’s like one of these as it. Another way to think about it is like imagine, we our thing wasn’t open. We had some agent hosting thing, but it’s not open and then there is an open one. if you’re. Which one’s gonna win in the long run? So like here, because there is this benefit from like people writing integrations, it’ll be, it’ll be that. And then there are other things that like you just can’t, even deliver as open source that are things the company does. Like for example, how do you make sure you’re like streaming, jobs or your Lakebase database doesn’t like, lose all your data at night? Well, that requires an operational team that’s gonna sit there. There’s no way it has to be a service. So like we wanna make sure as a company we’re really good at those infra services and then we’re as open as we can in terms of like what you build on top. Swyx [00:11:42]: speaking from a benefits, I think we are already seeing pull requests Matei Zaharia [00:11:45]: Yeah Swyx [00:11:45]: of all kinds of ecosystem integration, even though it was only released on Saturday. Matei Zaharia [00:11:50]: Yeah, Saturday. Yeah. So someone Swyx [00:11:51]: Let’s see, let’s see what’s going on. Yeah, you can look at the merge ones. I asked Sam Nigon this morning about Matei Zaharia [00:11:59]: 400 merge already? Matei Zaharia [00:12:00]: Yeah. I think Recent quite, I would guess around half are not from our team. but for example, someone added support for running it on Kubernetesrnetes. people added, many cloud sandboxes, so this can launch a cloud sandbox and run your agent in there, which is great for sharing too, ‘cause it’s not, like, on your laptop and someone’s, like, running scary code on there. so yeah, many startups have put those in, and, we expect to see more of them. We also have more agent harnesses already. Cursor, CLI, and Antigravity also. The Modern Data Stack and the Emerging AI Stack Matei Zaharia [00:12:34]: Yeah. That’s all, beautiful. And I, I feel like the last time this happens, there was the rise of the modern data stack. Matei Zaharia [00:12:42]: I don’t know if it’s that useful. I’m, I’m curious in your postmortem. Matei Zaharia [00:12:46]: I think most people Swyx [00:12:47]: Agree Matei Zaharia [00:12:47]: will agree that it is finally dead. but maybe this arises to a new modern AI stack that, like, does the same thing. Matei Zaharia [00:12:52]: I don’t know. Reynold Xin [00:12:54]: I think the modern data stack was a pretty useful thing, probably even up until this day. I think what, maybe for the audience who don’t understand the history, I think the modern data stack is effectively decomposed into you need a layer to ingest the data in, you need a layer to transform your data, and then all of this are run, and then you need a layer to maybe visualize your data. And all of this runs on some data warehouse, or later on, as we’re doing data warehouse or lakehouse. Reynold Xin [00:13:21]: I think that concepts are all very powerful and very useful. They enable a lot of workloads. What people eventually run into is a question of unification and consolidation is, hey, do you really need to chop all this into different pieces and work with so many different vendors and platforms in order to get, like, a very simple visualization done, right? So I think, like, over time, everybody started realizing that customers are pushing us. We started, we can realize that, so we started building more and more capabilities and trying to consolidate. And at the end of the day now, customers don’t have to worry about having me hook up five different systems in order Matei Zaharia [00:13:55]: Yeah Reynold Xin [00:13:55]: produce a chart. But the. I think, honestly, something like this is probably happening, in how many different frameworks do you want to hook up together in order to produce, like do a very simple agent. Matei Zaharia [00:14:06]: Just to be clear, I would say the core of this is this common API on top of all the harnesses. So the API is like, you’ve got an agent session, and you can send in a message or, like, a file. That’s what you can send in, and then you get out, these streams as it’s streaming text or as it’s doing tool calls. And, or the other thing you can send in is you can, like, tell it to cancel a turn. So that’s the API. Now, the thing we did is we could get you that on top of, like, cloud code running in a terminal, Codex, Py, OpenAI SDK, all that stuff. We map them all to that same interface. So that is something that you’d have to maintain yourself if you built your own, like, agent orchestrator, and then whenever cloud changes its API, you gotta, tweak your thing or it’s gonna lose some messages. So that’s the thing that’s valuable to maintain. Then on top of that, like, we built a few apps. I think we built a pretty cool UI and stuff, but that’s, And we built a security and control piece, which I’m excited about. But it’s that common interface, so we don’t. We. That doesn’t try to be a stack. And in fact, you could plug in your own UI on top of this, server. That, and that’s one of the use cases we care a lot about, ‘cause we want to use this in our own products. Compute, Sandboxes, and Databricks Scale Swyx [00:15:20]: Yeah. It should be everywhere. Matei Zaharia [00:15:22]: Yeah. Swyx [00:15:22]: I think one of those things that is really interesting to me is, like, well, first of all, I’ll, I’ll endeavor to do everything and not call it the modern AI stack because like it needs a different name. Matei Zaharia [00:15:32]: Yeah. Swyx [00:15:32]: But like, yes, like, so one of the first people that told me about compute, sandboxing was Nikita from Neon. Swyx [00:15:39]: Because a lot of people think about Neon as like, well, it’s serverless Postgres with, like, the separation of compute and storage and, instant branching and all those things. But every database company is also a compute company. Matei Zaharia [00:15:51]: Yeah. Yeah. Swyx [00:15:52]: And so he was showing to me his whole, his sandboxing solution. I don’t think he have ever launched it. Matei Zaharia [00:15:57]: So our sandbox solution, the reason we could build it so quickly was because we realized if you just take the actual Lakebase architecture Swyx [00:16:05]: Yeah Matei Zaharia [00:16:05]: and remove the database from it, by the coming from Neon Swyx [00:16:08]: Exactly, right Matei Zaharia [00:16:09]: you have this sandbox Swyx [00:16:09]: Every database company has it already, yeah. Matei Zaharia [00:16:11]: Now, there are some differences. For example, in the one to support this particular workflow, it’s important to have local persistence, Swyx [00:16:19]: Yeah Matei Zaharia [00:16:19]: because you want your state to persist. Your libraries, you don’t have to install your library every time, right? Matei Zaharia [00:16:24]: whereas the Neon architecture, because of the separation of storage from compute, you don’t need persistent local disk. Swyx [00:16:30]: Yeah. Matei Zaharia [00:16:30]: So there’s some differences. Swyx [00:16:32]: Yeah. Matei Zaharia [00:16:32]: But the, at the end of the day, yeah, it’s, Yeah, so this is when you run, like, a coding sandbox. Like, if I use it, yeah, we have the dev env internally at Databricks. There’s, like, many, like, tens of gigabytes of data just for, like, all the source code and, like, artifacts and stuff that I built, and I want that to come back next time, so. Matei Zaharia [00:16:51]: Yeah. Matei Zaharia [00:16:51]: But yeah. Matei Zaharia [00:16:52]: Before the show, we was talking about some statistics that might be surprising at the adoption. Matei Zaharia [00:16:56]: It could be internal, it could be external, whatever comes to mind, just to impress people the scale this is happening. Swyx [00:17:02]: So we, on the analytics side, I think we launched Reynold Xin [00:17:06]: Maybe 50 or 60 million virtual machines a day across all three clouds, so we’re one of the biggest compute orchestrators out there. Reynold Xin [00:17:13]: Stuff for sure for CPU compute. Swyx [00:17:14]: Yeah. Matei Zaharia [00:17:14]: Yeah. Reynold Xin [00:17:15]: the. And all of this process, I think exabytes of data, I joked about depending on which time zone you are, typically before you have breakfast, Databricks would have processed exabytes of data already on that day. and on Neon, it’s pretty interesting, too. It’s launching, I think, 13 million databases Swyx [00:17:34]: Yeah Reynold Xin [00:17:34]: a day now. Swyx [00:17:35]: Yeah, to me that was, like, a Reynold Xin [00:17:36]: And that’s just like Swyx [00:17:37]: Like, what do you mean? Matei Zaharia [00:17:38]: Yeah. And that’s the point. Reynold Xin [00:17:40]: And a lot of those were thanks to agent- agents and branching experimentation Swyx [00:17:44]: Yeah Reynold Xin [00:17:44]: because we made it so easy and so quickly, and thanks a lot to Nikita’s team, to launch databases. It’s, the. So it’s changing the way people use databases. Swyx [00:17:54]: Yeah. Okay, we’re gonna go into more database talk in a bit, but I wanna make sure we close up anything on Omnigentt. you mentioned, you were excited about the security Omnigent Security, Contextual Policies, and Spend Controls Swyx [00:18:03]: control side. Matei Zaharia [00:18:04]: Yeah. Swyx [00:18:04]: a lot of companies are figuring that out right now, as well as the spend side. Matei Zaharia [00:18:08]: Yep. Swyx [00:18:09]: what have you found there? Matei Zaharia [00:18:11]: Yeah, so I spent quite a bit of time talking to internal users, developers, security team, managers, and also lots of customers, and there’s a few things. Like, first of all, one thing, that immediately was. became obvious is for security, there’s this tension between, like, usability and security. And, the way people do. Like, a lot of coding agents today have very basic things like you can tell me which tool patterns I’ll allow or disallow or whatever. It’s like yes or no. But that puts you in a very tough spot. So just as an example, like, should my agent be able to read, some confidential documents, or let’s say, should it be able to install new packages from npm, which, maybe it’s compromised. Yes or no? Like, maybe I wanna allow it. Should my agent be able to publish stuff to the company website? Well, if I’m using it to code on the website, yes. But should it be able to do both, so it can, like grab a confidential document and be prompt injected and leak it? Probably not. So the thing we decided we need is stateful or what we call contextual policies where you keep track of the state of that session. It’s not like is it allowed to push to the marketing site or not, but, like, hey, if it did a risky thing, like it installed, a old package from npm, or it read, like, 1,000 confidential docs, then no. Then don’t, don’t do it. Otherwise, maybe it’s okay. That’s one example of, like, moving that trade-off so it’s both more secure and more useful by having a more powerful engine, essentially. This requires tracking sessions. The other piece that was interesting there is, like, there are these very level events it’s doing, and you want some libraries on top that parse them. Like, for example, we have a, MCP server on Google Drive internally. It’s got 60 API calls. like, how do I know which of those, like, will share a document with stuff on the internet and which ones won’t? It’s, it’s annoying. So we designed in Omnigentt the policy layer so that it’s functions and you can have libraries. Like, someone can make something that maps the level events to high-level ones, and then you write a policy about the high-level things that came out. so and that Swyx [00:20:25]: This is related to the Panther, Matei Zaharia [00:20:27]: Yeah, Panther is. will help with that. Panther Swyx [00:20:30]: Yeah Matei Zaharia [00:20:30]: a similar idea on the event processing side, and it’s Python-based versus a weird custom language. this is more, as in real Swyx [00:20:39]: I didn’t even know we were good yeah. Matei Zaharia [00:20:41]: Those things are happening, yeah. Swyx [00:20:42]: Yeah. Matei Zaharia [00:20:42]: So yeah, but these are the cool things. I think the contextual or stateful part, and then the way it can be libraries, and that was another reason to make it open source because others will write libraries and, like, we and our customers can use them. And the final thing, because it’s stateful, one of the states we track is how much you spent in that session. So I can. I’ve had, like, I ask an agent to debug something, and it spent $500 because it decided to read a lot of log files and burn a lot of tokens. but I can literally say, “Okay, launch a agent to do this and cap it to spending $5.” Like, ask me for permission if it needs more. And because we’re counting that within that session, it’ll pop up and tell me, “Okay, you spent five, $5. Do you wanna go on?” Reynold Xin [00:21:27]: So important context here. Matei spent the last five years, a lot of his time was architecting Unity Catalog at Databricks Matei Zaharia [00:21:34]: Yeah Reynold Xin [00:21:34]: which is the governance layer for data. Matei Zaharia [00:21:35]: That’s right, yeah. Reynold Xin [00:21:36]: And he’s combining expertise at that layer together with all the AI governance he knows. Matei Zaharia [00:21:41]: Yeah. Swyx [00:21:41]: Do Matei Zaharia [00:21:41]: But I also spent a lot of time being annoyed by coding agents and getting prompts. Matei Zaharia [00:21:46]: And also as the Reynold Xin [00:21:48]: All the above Matei Zaharia [00:21:48]: I don’t want to end up on the front page as, like, I installed some weird npm package and leaked Swyx [00:21:53]: Yeah Matei Zaharia [00:21:53]: all the code, so I’m especially paranoid. But also I have very little time, so I don’t want to sit there approving, like, do you want to run a 20-line, bash script, yes or no? so that’s why I spend a lot of time figuring out, like, how can I make it as safe as possible and not annoying? Swyx [00:22:10]: Yeah. Is safety and mmm, let’s call it security a bigger concern than token maxing or token budgets? which one is, like Matei Zaharia [00:22:19]: Oh, yeah, they’re both there. I don’t know. I guess it depends on the type of company you are. So I think, some companies, like, the budget is, limited and, they really care about that Swyx [00:22:34]: you can be Uber and still be concerned? Matei Zaharia [00:22:36]: Yeah. Oh, yeah, totally. Yeah. If you have Reynold Xin [00:22:38]: for us, security Matei Zaharia [00:22:39]: Yeah Reynold Xin [00:22:40]: super paramount. Matei Zaharia [00:22:40]: For us, security is absolutely critical as a, cloud provider. It’s, it’s the most important thing, and, token maxing, we’re not so worried about it yet, but I’ve seen the Like, for example, I talked to some consulting companies. They have, like, 100,000 employees who are all coding for customers. If those each spend, like, an extra $1,000 a month, that’s, that’s not fun. Swyx [00:23:04]: Yeah Matei Zaharia [00:23:04]: we have, like, only a few thousand engineers. Swyx [00:23:06]: What’s the policy in Databricks? Is it just unlimited or what’ Matei Zaharia [00:23:08]: It’s, it’s unlimited, but we do. we use our own product to, like, analyze the traces and stuff, and we have a team that’looking to optimize and to see if anyone’s doing something weird. And, we had some really cool insights just from analyzing current traces, like which Swyx [00:23:24]: Yeah Matei Zaharia [00:23:25]: models are better at, say, Rust versus like TypeScript or whatever. So yeah, at least in our code base. Swyx [00:23:31]: Yeah. Amazing. Obviously, I have to ask the token question, obviously. Matei Zaharia [00:23:34]: Yeah. Swyx [00:23:34]: I think it’s Reynold Xin [00:23:34]: Yeah Swyx [00:23:34]: it’s a key thing. But yes, security and control above that, and figuring out a sane layer there you can have some autonomy, but, not too much. Matei Zaharia [00:23:43]: Yeah. Yeah, and we wanna make it super easy. As a engineer, you should set a thing. So in Omnigentt, you can ask your agent, “Set a policy on yourself to do this.” So it can like Swyx [00:23:52]: But if there’s something I should be showing Matei Zaharia [00:23:53]: Yeah Swyx [00:23:53]: I don’t, I don’t see it on the GitHub, but, Matei Zaharia [00:23:55]: Oh, yeah Swyx [00:23:56]: there’s just Matei Zaharia [00:23:56]: Well, in the docs there’s something. Swyx [00:23:57]: Yeah, this is it. Matei Zaharia [00:23:58]: You can look at it later. Swyx [00:23:59]: Okay. Yeah. Matei Zaharia [00:23:59]: Just look in the docs Swyx [00:24:00]: Yeah Matei Zaharia [00:24:00]: contextual policies if you wanna see. Swyx [00:24:04]: I just like to point people Matei Zaharia [00:24:05]: look at the built-in policies. Swyx [00:24:06]: Yeah. Reynold Xin [00:24:06]: Yeah. Swyx [00:24:06]: If you want to, follow up on this is exactly where to look, right? Reynold Xin [00:24:10]: Yeah. Matei Zaharia [00:24:10]: Yeah. yeah, and the story of these is, like, I just wrote, like, I wrote a doc with like 10 ideas for things before as you were working on them. Well, that was, like, my wish list of things people asked, and I told the team, like, “Hey, can you do like at least five of these for the launch?” And then they just got back with all of them, so. Swyx [00:24:29]: Oh, wow. Matei Zaharia [00:24:29]: so you can come up with more, but them- some of them are just meant to be examples. really you can intercept, like, any event the agent is making, and you can then either block or force it to ask the user or, like, allow, and you can update state to keep Swyx [00:24:45]: Yeah Matei Zaharia [00:24:45]: track stuff. Swyx [00:24:46]: Yeah, ‘cause ultimately you’re, I think of you as, like, a systems designer. Swyx [00:24:50]: You let people plug in, right? That’s the whole Matei Zaharia [00:24:51]: Yeah Swyx [00:24:52]: modus operandi of what you do. Matei Zaharia [00:24:53]: Yeah. Swyx [00:24:54]: It’s like Matei Zaharia [00:24:54]: And we care a lot about also composab- like, can someone else write a library that others use, which Swyx [00:24:59]: Yeah Matei Zaharia [00:24:59]: this is meant to. Reynold Xin [00:25:00]: There’s also a batteries included philosophy here Matei Zaharia [00:25:03]: Yes Reynold Xin [00:25:03]: probably very similar to how you did Spark, which is you could just start using. Swyx [00:25:06]: Yeah. Matei Zaharia [00:25:06]: Yeah, that’s right. It has to be good out of the box at certain things, and then you can build your own things on top that, like, we don’t wanna do. But in Spark, if you just wanna like, I don’t know, like read a table or do, like, a aggregation, it should be awesome at that out of the box. Building on Omnigent: Contributions, Startups, and Analytics Swyx [00:25:23]: Yeah. People wanna catch up on Omnigentt, they should watch your keynote. Swyx [00:25:26]: they should go through the GitHub and the docs. If they wanted to contribute, or they want to build on this ecosystem what would you call out as the most high-leverage places get involved? Matei Zaharia [00:25:36]: Yeah, do get involved in the Discord and in GitHub. Our team is there, is monitoring, and, some of the things people ask for we just built ourselves. Some of them, we’re, we’re collaborating with them to build it. and also tell us, like Swyx [00:25:49]: Yeah, they’re gonna be very Matei Zaharia [00:25:49]: how you would like to use it because I think especially for developers, like, everyone wants it to work their own way, and a really good developer tool, like you have to hear the feedback on all the ways and figure out the abstractions and how to let people customize. So we’d love to hear, like, if you think, “Hey, I, I don’t want it to work this way,” tell us. We really just wanna get that compatibility layer across agents and then let you do stuff on top. Swyx [00:26:14]: Yeah. is there any, in terms of like the startup side, I’m, I’m a founder. Swyx [00:26:18]: I want Matei Zaharia [00:26:18]: Yeah Swyx [00:26:18]: I see an opportunity, I wanna get in front of you. What’s your request for, like, a startup that, like, I wish someone Matei Zaharia [00:26:23]: Oh, like you wanna integrate with us? Swyx [00:26:24]: someone was working on this. Matei Zaharia [00:26:26]: Oh, for a startup? Swyx [00:26:27]: Yeah. Swyx [00:26:28]: Like, your, you got your own startup. It’s doing well. Matei Zaharia [00:26:30]: Yeah. Swyx [00:26:30]: But like, if you weren’t working on your own startup, what is, like, obvious that you should You advise many startups too, obviously. Matei Zaharia [00:26:37]: I do think, just as a company with a lot of engineers, like anything that helps me make sense of how people are using Swyx [00:26:46]: Spend Matei Zaharia [00:26:46]: coding agents and, Swyx [00:26:48]: Yeah. Analytics Matei Zaharia [00:26:48]: spend, but also quality or like you should write, you should add this skill, or you should write this thing, or your agents are really horrible at tasks involving this service, so I go spend time. That would be nice. yeah. Swyx [00:27:00]: Yeah. The closest I’ve found is, this team, GitAI. Matei Zaharia [00:27:03]: Oh, cool. Yeah. Swyx [00:27:04]: They started with, like, we will just do, code and human attribution, but they’re building the analytics layer on top of that. Matei Zaharia [00:27:12]: Yeah. Swyx [00:27:12]: I do think, like, there are a bunch of, like, artificial analysis is obviously, Matei Zaharia [00:27:18]: Yeah, they have their benchmarks Swyx [00:27:18]: doing super well Matei Zaharia [00:27:19]: Yeah Swyx [00:27:19]: with their stuff. so there’s, there will be people. I think this is like the domain of consultants first, but then people Matei Zaharia [00:27:26]: Yeah Swyx [00:27:26]: will build software that, let’s say, it’s kinda like the management plane Matei Zaharia [00:27:29]: Yeah Swyx [00:27:30]: for coding agents. Matei Zaharia [00:27:30]: Yeah, I think there’ll be a lot of insights there. You have it in other areas. Swyx [00:27:34]: Okay. Well, and then the other, big thing is your dream engine. LTAP: Lake Transactional/Analytical Processing Swyx [00:27:39]: maybe you wanna tell the story of, LTAP. Reynold Xin [00:27:45]: So, and background with. I’m, I’m gonna make people listen to our Ankur Goyal episode where we talked about SingleStore, HTAP Matei Zaharia [00:27:52]: Yeah Reynold Xin [00:27:52]: and all that history. Matei Zaharia [00:27:52]: Yeah. The LTAP idea is pretty simple. so if people have heard of the, Ankur’s, talk about HTAP, it’s effectively the world of databases. Sorry, there’s like maybe a lot of context needs to be injected here. The world of databases Swyx [00:28:06]: I am happy to be the database podcast that I’m forcing people to, like, learn your databases, guys. Swyx [00:28:11]: You cannot vibe code with just markdown files. Reynold Xin [00:28:13]: Yeah. Swyx [00:28:13]: Like, Reynold Xin [00:28:14]: It’s one of the most important fundamental systems technologies out there. But the world of database effectively split into roughly two halves. There’s what we call OLTP databases, which are transactional, and think of your Postgres, your MySQL, your Oracle databases, and the other side is what we call analytics, and sometime might refer to term OLAP. And the difference is on OLTP, you typically have maybe run some transaction on some event that looks up at one specific row. We update that row, right? It’s a very oriented data structure. And on analytics, you’re trying to reason on the data. You’re trying to compute, “Hey, what’s my revenue per store? What’s my. How’s my website doing every day?” And then you, eventually want to probably end up running anal- machine learning on it to predict, “Hey, how will my maybe sales be going in the future?” they are so very different architecture, and everybody start with OLTP databases. Every app, when you become serious enough, that needs more than markdown files, you need to have a database. You want to lose your data, you want to have some transactional consistency. But once you want to reason on the data, if you only have like- A hundred rows, it’s probably okay to run it on your Postgres or your own, your MySQL database. But once you have more data and want to run more complicated analysis, the very analysis might crush your Postgres database. So you start doing, getting data out of the OLTP database Swyx [00:29:35]: Replication. Reynold Xin [00:29:36]: Replicate them into the analytic systems and just start Swyx [00:29:39]: Yeah, which for people, Elasticsearch is, like, a Reynold Xin [00:29:42]: Yeah. So some of them get into Elasticsearch for, like, blocked analysis. A lot of our customers obviously get into Databricks to run more sophisticated things. Swyx [00:29:51]: Yeah. Reynold Xin [00:29:51]: And there’s this term called CDC, which Matei Zaharia [00:29:54]: Change data capture Reynold Xin [00:29:55]: change data capture. and what it does, it reads the binlog of the database, and if you don’t understand what binlog is, it’s fine. The, but it’s a little delta of the data, and it reconstructs based on the delta, the state of the database, on the analytics side. But CDC is, like, a very painful thing. It’s how standard in the industry, everybody uses it, but, it ends up being. I think many data engineers ends up being waken up at, like, 3:00 a.m, because there’s some pipeline thing. Swyx [00:30:22]: my explanation is, like, Airbyte is like a, became a $5 billion company just doing CDC. Reynold Xin [00:30:27]: Yeah, exactly. Reynold Xin [00:30:28]: CDC is, like, a very Matei Zaharia [00:30:30]: It’s hard. Reynold Xin [00:30:30]: It’s one of the most boring but one of the most fundamental operations, like, powering modern society. Matei Zaharia [00:30:37]: huh. Reynold Xin [00:30:37]: But it’s so brittle that, we joke that it’s, should be called continuous data corruption, because you might change your schema on your OLTP database, and then the CDC pipeline fails to handle Swyx [00:30:48]: Yeah Reynold Xin [00:30:48]: the schema change. Swyx [00:30:49]: Yeah. Reynold Xin [00:30:49]: And then everything goes out. Swyx [00:30:51]: And there’s all sorts of tricks that you can do, like, you add in, like, some versioning or whatever, but yeah. Reynold Xin [00:30:55]: Yeah, but it’s a very, in general, very complicated. Like, I think at my keynote, I asked the audience put up their hand if they love their CDC pipeline. Only, like, maybe two people put it up. So if single store, like, about maybe a decade ago, I think the industry had this idea, hey, what if I built a single database that can handle both workloads? Now I don’t. Swyx [00:31:12]: Which, like, by the way, every database person ever has ever always dreamed about this. Reynold Xin [00:31:15]: Yes. Yes. Reynold Xin [00:31:16]: This is the holy grail of database engineering is why not build a single system that can do both of this? But it ends up just being a lot of compromises. one, I think one of the first issue is that, hey, each. they say Postgres has a massive ecosystem, right? You want to be using the tools that’s built for Postgres. And Spark, for example, had a massive ecosystem. There’s a lot of libraries you want to use. If you were to create now a new thing, you don’t have a ecosystem. You tend to create a new, smaller proprietary API, and you’re lacking both, and it’s also very difficult to make it performance-wise to be, comparable on either side. So it ends up being sucking on both. And our whole idea of LTAP, it’s obviously a wordplay on the term HTAP, is that we think this is HTAP done right. HTAP wants to build a single engine for both. We think you can get 99% of what you need by unifying the storage, and just have a single storage layer. And once you have the single storage layer, if your Postgres databases are writing data in a column-oriented format, everything analytics can just go read that data directly without any delay, right? There’s no pipeline in between, so all the data will immediately be available for reasoning analytics. I think I was telling some customers earlier, hey, when we talked about this is gonna be super useful for agents, I at first didn’t really believe in it myself, even though we wrote that positioning. Lakebase, Agents, and Live Operational Data Matei Zaharia [00:32:39]: Yeah. Reynold Xin [00:32:40]: But then last night I was having dinner with a Australian customer, and they told me, “Oh, hey, one of the big issue we have is we have all these logs from our services, and we see SLA dips and want to investigate. But then there’s no way for those agents to even understand what’s going on in the actual databases themselves. All we see is just, like, product telemetry of the database and the services.” It would make those agents 10 times more powerful if understand, for example, who’s placing those orders, what is happening, what exactly are they doing. So now I’m sold on our own message. Swyx [00:33:13]: Yeah. Reynold Xin [00:33:14]: I think it’s really. It gets you the almost all of the benefits of the HTAP holy grail, which is, hey, make the data available immediately for reasoning analytics Swyx [00:33:26]: Yeah, I think, Reynold Xin [00:33:27]: without compromise Swyx [00:33:28]: in the way that humans are generally intelligent and want to have the ability and access to query anything Reynold Xin [00:33:34]: Yeah Swyx [00:33:35]: while they do the work, they also need history and need context. Swyx [00:33:38]: And, like, where else does they get context? That’s it’s an analytical workload. Reynold Xin [00:33:41]: Exactly. Matei Zaharia [00:33:42]: Yeah. Yeah. And I remember when we had incidents with our databases and engineers said, “Well, I can’t just run a giant query on it to see what’s going on because that’s gonna bring down the database and hoard it even more.” Like, that’s the stuff that this gets rid of, because you spin up a whole separate fleet of machines that’s doing the analytics. You’re not overloading, like, the main database Reynold Xin [00:34:02]: Right Matei Zaharia [00:34:02]: that’s still trying to serve stuff. Reynold Xin [00:34:04]: Yeah. Matei Zaharia [00:34:04]: Yeah. Why LTAP Works Now: Parquet, Postgres, and Lakebase Swyx [00:34:05]: So this has been a dream for a while. what had to get done in order to get to today? Like, Reynold Xin [00:34:11]: Yeah. Swyx [00:34:11]: I feel like, you have announced variants of this several times, but it wasn’t as clear as LTAP. Reynold Xin [00:34:18]: Yeah. Swyx [00:34:18]: I think LTAP is like Like, okay, we’ve got it, guys. Matei Zaharia [00:34:21]: This thing, yeah. Reynold Xin [00:34:21]: I was talking to somebody at Meta, and then he was asking me, “Hey, what’s the catch? Why is it possible now?” And I think the reality is we took a lot of time to work on the Lakebase architecture. obviously a lot of it came from the Neon team, which is a separation of storage from compute. And it turned out it was just a tiny little step away going from that to this LTAP idea, which is, hey, we just. in the Neon architecture and in Lakebase architecture, we’re writing data in oriented format to the open data lake, but in there we’re writing in Postgres pages. Ali and I were spending a lot of time debating, hey, can we just change that to write in column-oriented format? And we’re just debating, and one day, one of our engineers who’s, like, super smart came in, he’s like, “Hey, I just prototyped it. It works.” Swyx [00:35:07]: Wait, it’s, prototype what? Reynold Xin [00:35:09]: Prototype, instead of storing the data in the data lake in the oriented format Swyx [00:35:15]: Column Reynold Xin [00:35:15]: like Postgres pages Swyx [00:35:15]: Yeah Reynold Xin [00:35:16]: write them in Parquet. Swyx [00:35:17]: Yeah. Reynold Xin [00:35:18]: and he just made the observation that, hey, our storage fleet has a lot of extra idle CPUs And we could use those CPUs to do the transcoding from row to column, where row is good for OLTP, but column is good for analytics. so let’s do that transcoding at that time. And as a matter of fact, once you transcode the data compresses better. So from those services writing to, for example, S3 or other data lake, like object stores, you can write them faster ‘cause now they are now smaller. Matei Zaharia [00:35:49]: Yeah. Reynold Xin [00:35:49]: So there’s no overhead, it’s no compromise in performance Matei Zaharia [00:35:52]: Some CPU overhead. Swyx [00:35:54]: Yeah, because, Matei Zaharia [00:35:55]: Yeah Swyx [00:35:55]: we had extra CPUs anyway. Matei Zaharia [00:35:56]: We had that fleet anyway, yeah. Swyx [00:35:57]: so the debate ended. it’s one of the classics of, tech, issue of a lot of debate, but then somebody went ahead and just tried to prototype it and it worked. Matei Zaharia [00:36:06]: But, like, something this strategic Swyx [00:36:07]: That’s right Matei Zaharia [00:36:07]: and important to the company, I expect there to be, like, a kickoff thing, like a design doc. Nothing like that. Swyx [00:36:13]: Nothing like that. Swyx [00:36:14]: He just. We were debating in many meetings Matei Zaharia [00:36:17]: Yeah. Swyx [00:36:17]: and then we’re just debating whether it’s possible or not from first principle. Matei Zaharia [00:36:20]: Yeah Swyx [00:36:20]: and then, somebody just did it. Matei Zaharia [00:36:23]: Yeah, if you set yourself up so people do that’ll be great. And that happened a bit with Omnigentt too. I think if I just had a doc on, like, we can make these together, everyone would, would think, “Oh, what about this? What about this?” But then you. if you try it out, it helps. And then if you have real users and they bash it and, like, it’s still working, or in this case, if you have the workload, what the workload looks like, you can just test the same pattern then. Databricks’ Culture of Fast Prototyping Swyx [00:36:47]: Yeah. Matei Zaharia [00:36:47]: Yeah. Swyx [00:36:47]: Tech aside, which is very cool, this is, like, the most important thing, the culture of innovation, and you don’t have to ask my permission, you don’t have like, do a whole form- formal process, just do it? Matei Zaharia [00:36:59]: Well, especially these days, I think with Swyx [00:37:01]: Yeah Matei Zaharia [00:37:01]: AI, it’s easier to build Swyx [00:37:02]: But so, like Matei Zaharia [00:37:03]: a prototype Swyx [00:37:03]: I think you are very I made a lot of suite of, like, large companies and, like, I think that at scale, things slow down, and I’m sure you felt it already, but somehow you have this core of people that, like, are exempt. How? I think we hire and we work with really good people, and that’s a very important part of it, and empowering them, but also spending a lot of time, maybe us in the trenches matter a lot also. Matei Zaharia [00:37:28]: Yeah, I think, I think first, people can adapt to being in the larger company, so that helps. And we wanna make sure they know that they can try stuff and settle debates and have a lot of examples of how it was done before, or launch a thing in beta or whatever. and then the other thing I do think as a company, like despite the size, we don’t launch that many, like, products. We try to keep it pretty coherent. That’s, that was the whole, like, theory of the company, was like instead of having, like, 20 Amazon services you need to set up, like a analytics and machine learning stack, you just have one, and it’s, like, the same API, the same semantics across all of them, the same copy of the data. So that requires, like, unification. And then we added one more thing at a time. Like, we added storage with Delta Lake. We didn’t used to do any storage. Then we added SQL, we added, machine learning platform stuff. So, but yeah, don’t, don’t do too many, but do those things well and, that also helps, it helps keep it manageable. Reynold Xin [00:38:33]: Yeah. The other thing we encourage a lot is instead of building, boil the ocean for everything, let’s figure out how do we do it incrementally, how do we do it very quickly. Like, many of our products Matei Zaharia [00:38:43]: Yeah Reynold Xin [00:38:43]: they’re built in the span of weeks, and then we go to, hey. Like, usually my first question to whoever team is building is who’s the target customer? Who are you working with? Are you on a first-name basis with them? Are you texting with them? I think having that very tight loop, Matei Zaharia [00:38:59]: Can you bring up another launch that comes to mind when, in this thing? I just want to give examples. Reynold Xin [00:39:04]: Omnigentt itself happened that way. Reynold Xin [00:39:05]: Yeah. Matei Zaharia [00:39:06]: Who’s the customer? That’s a good one Reynold Xin [00:39:34]: storage layer we did. we had, our largest customer at the time said like, “Okay, I need some. I want something in the cloud ‘cause, I. if the rest of our network is compromised, like this thing needs to be separate to store and query the events.” And then, talked to us, he said, “Okay, this is the rate of events per second. This is, like, the freshness I want. Can you do it?” So that was, like, way larger than any workload we had, and we had our, engineer, working on that, Michael Armbrust, and he worked just to make this work. And once it worked for them, it worked for everyone else. Yeah. This was early in the company, probably like four years in or something. Matei Zaharia [00:40:24]: 20- 2018? Swyx [00:40:26]: Yeah, ‘17, ‘18. Matei Zaharia [00:40:28]: Few companies Swyx [00:40:28]: Do you have other examples? Matei Zaharia [00:40:30]: there’ Swyx [00:40:31]: Maybe you have others Matei Zaharia [00:40:31]: yeah, Clean Room, which is how you share data in a way without sharing Swyx [00:40:35]: Yeah Matei Zaharia [00:40:35]: underlying data, but you allow specific operations. Those were done effectively initially just for two customers. I think the industry has a sense of, hey, maybe if you overfit to, like, one or two customers, it’s gonna be really bad for you. But I think the, downside of overfitting is much smaller than the upside itself. And if you try to be too ambitious and boil the ocean, it’s a much bigger problem. Swyx [00:40:58]: Yeah. Yeah. Matei Zaharia [00:40:58]: ‘Cause you might end up having no customer. Swyx [00:41:00]: Yeah, that’s more, that’s the more likely outcome. Matei Zaharia [00:41:02]: Yeah. Tech Companies vs. Enterprises Swyx [00:41:03]: than you can pivot from there. I do think there is such a thing as a bad customer that sometimes you should fire. Yeah. Matei Zaharia [00:41:08]: They could exist sometimes if you drive. well, one of the challenge I think we probably see, and maybe many AI, so newer generation companies are seeing is, so tech companies are very different from tech companies or traditional enterprises. Swyx [00:41:22]: Yeah. Matei Zaharia [00:41:22]: And, if you optimize everything just for tech companies, you might have various challenges Swyx [00:41:27]: Oh Matei Zaharia [00:41:27]: scaling them outside of tech companies. Swyx [00:41:28]: Okay, what like Matei Zaharia [00:41:30]: Yeah Swyx [00:41:30]: what like top three differences that you always think about? Reynold Xin [00:41:33]: Governance is a big one Matei Zaharia [00:41:34]: I think, yeah, a big one is like, yeah, security, data privacy, governance, all that stuff. So usually if you’re building some kinda like B2B or developer tool, like your biggest market is gonna be enterprises, but it’s just very different. A company that’s existed for like, it’s had some form of IT for like 30 years, they have so many legacy systems or they operate in a regulated space. whereas a startup or, even like a, like sorta more recent tech company, all the. everything is new and pristine. So yeah, it’s just different, and if you’ve never worked with enterprises or been in one, you just won’t know about it. Reynold Xin [00:42:13]: Yeah. Matei Zaharia [00:42:13]: Yeah. Reynold Xin [00:42:13]: And the procurement process is probably quite different. There’s far more stakeholders. Matei Zaharia [00:42:17]: Yeah, that is one. Yeah. Matei Zaharia [00:42:18]: Another piece that’s interesting is I think some tech companies, people, will say, “Oh, I can build that myself,” right? I’ll just build that myself. Matei Zaharia [00:42:27]: So then you go, Reynold Xin [00:42:28]: I don’t think people say that about Databricks, but Matei Zaharia [00:42:31]: yeah, it depends Reynold Xin [00:42:32]: They do. Matei Zaharia [00:42:32]: They do? Matei Zaharia [00:42:32]: Yeah, the. Yeah, and it depends on the teams and things. So, but, on the other hand, like many of the enterprises say, “I don’t, I never wanna be in the business of building that.” Like, I don’t want my, whatever, I’m a retailer or something, I never wanna Reynold Xin [00:42:45]: Yeah, sell clothes, Matei Zaharia [00:42:46]: be down because like some weird like nerd like couldn’t get streaming pipelines working. Matei Zaharia [00:42:51]: That is not what I’m doing. Reynold Xin [00:42:53]: Yeah. Reynold Xin [00:42:53]: Yeah. This makes them great customers, to be honest, right? Matei Zaharia [00:42:55]: Yeah. But you have to understand that it’s hard without having worked there and stuff, like you may not appreciate. Reynold Xin [00:43:01]: Look, I think they’re all great. don’t get me wrong, they have different challenges. But the, many of the tech companies, for sure there’s a lot, far more DIY. Matei Zaharia [00:43:10]: On the flip side, you have people who are. they’re very much experts in their domain, like they’re building airplanes, they’re, designing medicines, whatever, and they just want to bridge the technology, where like they don’t wanna learn, databases or whatever. As cool as we think it is, even as interesting as the average software engineer might think it is to read a little bit, like they just never wanna know. They just say, “I have a, giant like, matrix or whatever with my, clinical data, like how do I, how do I like cluster it or whatever?” So yeah. The Dream Engine and Rewriting the Database Stack Reynold Xin [00:43:40]: Yeah. That’s true. Okay, so and then I wanted to build out the dream engine, vision. where does this all lead? So one of the thing we, realized maybe a couple years back is that every single database engine out there, especially on the analytics side, are a decade old. pretty much everything that have reasonable traction are about a decade old. And they all started targeting some very specific narrow use cases, and then over time it’s become more and more successful. They have grown in their ambition, and then they try to support more and more use cases. But the fastest way to support those use cases tend to be hacked around the abstractions that were initially created, that were not for those use cases. Matei Zaharia [00:44:23]: Yeah. Reynold Xin [00:44:23]: And then, but you can support them more or less okay. And before it, after 10 years of organic evolution that way, it becomes a gigantic pile of s**t. Reynold Xin [00:44:31]: the. And, but that includes Databricks. And very few company or very few systems, I think, have the gut to say, let’s go start from scratch. Let’s go back to the drawing board and design, knowing everything we know today after a decade of workloads and probably billions in revenue, let’s attempt to rewrite it from scratch and make sure it will work and it can support all of these use cases. So we started doing that, but it’s a very ambitious project. by the way, you can search on Wikipedia, there’s this thing called second system syndrome. Matei Zaharia [00:45:08]: Yeah, I know that. Yes. Reynold Xin [00:45:09]: Or second system effect. Matei Zaharia [00:45:11]: Every developer must know what a second syndrome is. Reynold Xin [00:45:12]: It’s you built your first thing and it works out great, and the second one’s bound to fail because you become too ambitious. Reynold Xin [00:45:19]: And then you ask so many requirements. Matei Zaharia [00:45:20]: Or like you think everything Reynold Xin [00:45:21]: Yeah Matei Zaharia [00:45:21]: and then you’re like Reynold Xin [00:45:22]: You just Matei Zaharia [00:45:22]: you’re, “I’m gonna design the perfect system this time.” Reynold Xin [00:45:24]: Yeah. And it turned out it’s not perfect, and then it start failing and you’re too ambitious, never launch, and you get killed. The, and the engineering team that started this, they were brilliant. I think we hired some of the best database engineers, on the planet into Databricks, and they were brilliant. Thank God it’s not their second system. Many of them have built more than two in the past. Matei Zaharia [00:45:44]: Ah, nice. Reynold Xin [00:45:45]: But they were still worried about this, hey, building a database engine from scratch, I think the conventional wisdom is gonna take like five years to mature. This would be a very long-term project. It could fail. I think one of the engineers jokingly said, “Hey, maybe we just call it Reynolds Stream Engine.” If we name after a founder, maybe we then may get canceled or killed. But I think they built something pretty remarkable. they went back to. They changed the way the database engines were built from a paradigm point of view. Usually when you build a database engine, you read a lot of academic papers, you try to understand what are the latest algorithms and data structures, and you put them together and see if they work or not. And there’s a high risk of failure there also because whatever that looks really good on paper might work out. might look really good in 70% of the workloads, but then it backfires on the other 30%. they went build a more of a factory for building the database. So they spent more time building this factory, and the factory takes the decade of traces we have. I think they count as like quadrillion data points in the trace table. Matei Zaharia [00:46:47]: You don’t drop anything? Or you see sample? Reynold Xin [00:46:49]: We for sure sample, Matei Zaharia [00:46:50]: Yeah Reynold Xin [00:46:51]: the, there’s like massive amount of things. And the, and they use that to build a model, like a machine learning model. Not an AL, a machine learning model. Machine learning model it can very quickly tell us how any algorithm and how any implementation would perform for any specific type of queries with very high fidelity. And based on that, they can, pick the most likely algorithm and data structure that will help with the different kinds of workloads. Reynold Xin [00:47:21]: Both at runtime as well as at implementation time. Reynold Xin [00:47:25]: Because there’s like unlimited number Matei Zaharia [00:47:27]: it sounds like you want to like route to different data structures Reynold Xin [00:47:31]: Yeah. if you think about Matei Zaharia [00:47:32]: This is not one database Reynold Xin [00:47:33]: a single database has many things implemented Matei Zaharia [00:47:36]: Yeah Reynold Xin [00:47:36]: together. But you want to make sure they all work well Swyx [00:47:39]: Yeah Reynold Xin [00:47:39]: with each other, and then for any given operation, there might be more than one implementation, so we make it run really. reality is things, algorithms that work super well, for example, for very low latency might not work very well for, say, scanning through petabytes of data. Swyx [00:47:54]: Yeah. Reynold Xin [00:47:54]: Right? most often there’s a trade-off there between throughput and latency. Swyx [00:47:58]: What are the key dimensions like scale, throughput, latency? What Reynold Xin [00:48:01]: Yeah, scale Swyx [00:48:02]: anything else? Reynold Xin [00:48:02]: and the distribution of data. Swyx [00:48:05]: Yeah. Reynold Xin [00:48:05]: Right? How sparse the data is. Swyx [00:48:06]: How hard Reynold Xin [00:48:06]: That matters Swyx [00:48:07]: Yeah Reynold Xin [00:48:07]: very a lot. how frequently do you hit the same data? Matei Zaharia [00:48:10]: Yeah, how many distinct values Reynold Xin [00:48:12]: Yeah Matei Zaharia [00:48:12]: and stuff like that. Reynold Xin [00:48:13]: Those things matter a lot. Matei Zaharia [00:48:14]: Yeah. Reynold Xin [00:48:14]: Like number of distinct value impacts the memory consumption of your aggregation, your hash. Like at some point there’s a hash table. Swyx [00:48:20]: Somebody, I’m gonna, in my write-up, I’m gonna try to list all this out because I really want a taxonomy. To me, taxonomies Matei Zaharia [00:48:25]: huh Swyx [00:48:25]: are so helpful because it covers everything that you should think about. Reynold Xin [00:48:29]: I think if you try to list it out, probably like a million different features. Swyx [00:48:32]: I always want like, okay Reynold Xin [00:48:35]: It’s not a trivial Swyx [00:48:35]: give me like 12. Give me. Swyx [00:48:38]: like a, someone did, like I think a Oracle paper in like 40 years ago did like the, these are the eight fallacies of distributed systems. Reynold Xin [00:48:45]: Yeah. Swyx [00:48:45]: Right? That thing is super useful. Matei Zaharia [00:48:46]: Yeah, it is. Swyx [00:48:46]: It’s like, okay, think through these eight. Reynold Xin [00:48:48]: But let me give you a very, weird example, but it has profound implication on performance, which is like is your string just ASCII or does it have Unicode in it? How should you encode it? Swyx [00:48:59]: Strings, strings are the most complex data types. Reynold Xin [00:49:01]: Yeah. So the. And that, like for example, if string is super dense, you could convert every string into a, like imagine you have to do a aggregation. Instead of having a hash table, you could have an array. Because if your string is dense enough, if you only have 256 options, you don’t need a hash table. You can just do array Swyx [00:49:21]: Yeah Reynold Xin [00:49:21]: lookup. Swyx [00:49:21]: Yeah. Reynold Xin [00:49:22]: and that’ll be far fast. Matei Zaharia [00:49:23]: Yeah, if the string is like a country code or something. Reynold Xin [00:49:25]: Yeah. Matei Zaharia [00:49:25]: Yeah. Reynold Xin [00:49:26]: So it’s like probably millions of, features in that model. But using that, they can, one, prioritize the different algorithms that might impact in practice. And many of them are very counterintuitive. These are naturally things that you think, hey, might work super well, don’t work that well in practice. But also more importantly at runtime, you can dispatch the right algorithm and structure. Vector Databases, Query Engines, and LTAP Swyx [00:49:47]: I’m listening to the dream. I feel like Databricks is doing a really good job of the incremental evolution. Do you have to hard cut to a new system at any point? Or like, Reynold Xin [00:49:58]: We designed it in a way that it can be incremental. Swyx [00:50:00]: Yeah. Reynold Xin [00:50:00]: So first we’re releasing a new endpoint. but this goes to the broader ocean versus. what we wanted to do is wanted to by design, this new engine should be able to do everything we’re able to do before and better, right? It’s been particular, the better part refers to very low latency workloads that can finish in 10s of milliseconds. But we want to roll it out incrementally with incremental capabilities so it doesn’t take like five years to see the light at the end of the tunnel. Swyx [00:50:29]: I think that’s a heroic task. I don’t know what other way to say it. I am really interested in any new workload and new databases. obviously I think, if a, I’ve maybe established that I’m a little of a database nerd. The transactional databases, sorry, the accounting databases, like the Tiger Beetles I don’t know if you’ve, seen those. Reynold Xin [00:50:50]: What do they do? Swyx [00:50:51]: Dual entry accounting database. Like it’s just meant to really model like financial accounts or credit systems Reynold Xin [00:50:56]: Oh, I see. Reynold Xin [00:50:57]: it’s like a very specific problem. Swyx [00:50:58]: Very high throughput. Yeah. Reynold Xin [00:50:59]: Yeah. Swyx [00:51:00]: Yeah. No, so when you were talking about how everyone like starts with Matei Zaharia [00:51:02]: Yeah Swyx [00:51:02]: a thing and then they Reynold Xin [00:51:03]: Oh, I see Swyx [00:51:03]: they scale up and then they tack on other things. It’s exactly that. Swyx [00:51:06]: And then, I recently interviewed Simon from TurboPuffer. Reynold Xin [00:51:08]: Yeah. Swyx [00:51:09]: Same thing. Matei Zaharia [00:51:09]: Yeah. Swyx [00:51:09]: Like, well, and Chroma as well, like the, all the vector database companies of 2023 Reynold Xin [00:51:14]: Yeah Swyx [00:51:14]: all are suddenly now just, we’re just generalist, general storage, like blob storage. Matei Zaharia [00:51:18]: Yeah. Reynold Xin [00:51:18]: Vector database should have never been a separate category. Swyx [00:51:21]: I think it used to be a hot take, now it’s like the conventional wisdom nowadays. What should be a separate category? if everything becomes LTAP, like what’s. Reynold Xin [00:51:31]: I think the thesis of LTAP is we’re not collapsing the databases at the actual query layer. We’re just collapsing Swyx [00:51:37]: Indexing layer Reynold Xin [00:51:38]: the storage layer. Swyx [00:51:38]: Yeah. Reynold Xin [00:51:39]: and that’s a, I think, a very important part. And we don’t think it makes sense to collapse the query layer into a single, like HTAP style database. And part of it. By the way, the other thing I think a lot of people had is, hey, it would be nice if there’s only one query language I have to worry about. Instead of worrying about Postgres and maybe Spark SQL, why not just one? But I don’t think that’s an issue for agents. Agents are very eloquent in Postgres or Spark SQL. It’s never gonna get confused. As long as the data is there and it’ Matei Zaharia [00:52:10]: Yeah Reynold Xin [00:52:10]: accessible, agents will do fine. That might have been, Matei Zaharia [00:52:14]: Yeah, Reynold Xin [00:52:15]: five years ago might have been a problem for humans. Matei Zaharia [00:52:17]: That could arise over time also, but it should. And this is, leads to how to do things incrementally, right? Like we realize you don’t need it right now. We don’t need to solve that problem to have a lot of value, from the current LTAP. Swyx [00:52:30]: Yeah. Okay. I’m gonna end the pod with a little bit of more of spicier things. Databricks vs. Snowflake Swyx [00:52:37]: everyone has like, had to receive within a separation of storage and compute and try to build, the clouds. I had the same pitches from Snowflake. Swyx [00:52:47]: How have you succeeded where they failed? Swyx [00:52:50]: That’s rough. Reynold Xin [00:52:52]: Well, Swyx [00:52:52]: respecting that they are a competitor Reynold Xin [00:52:54]: Yeah Swyx [00:52:55]: objectively you have outpaced them. What is the core insight from your point of view that you guys just went different directions? Reynold Xin [00:53:03]: Probably the biggest fundamental difference, both companies started around the same time, both went to the cloud, both focused on storage from compute architecture. But the biggest difference, one is, open. Like Databricks had never had the proprietary format, right? We started with the open ecosystem started with Parquet and then evolved into Delta and Iceberg and all that. It’s like one big thing. I think it matters a lot. The other one is AI. before 2022, October 2022, when ChatGPT came out, we had always pitched Databricks as a machine learning plus data Swyx [00:53:38]: And a lot of the platform were built with machine learning use cases in mind, and obviously AI is a little bit different, and Matei’s, like spent far more time there than I do. But, the whole platform - we never felt, “Hey, we’re just a data infrastructure platform.” Matei Zaharia [00:53:53]: Like, well, it makes only Swyx [00:53:54]: Yeah. Matei Zaharia [00:53:54]: Yeah. Swyx [00:53:54]: We Matei Zaharia [00:53:55]: I think they started with, like, they thought, “Okay, we’ll just manage the most valuable data and try to make it really fast. For that, we’ll have our own storage, which is optimized with the engine, and then we’ll just start at, like, the small amount of data that, like, the managers and whatever, finance people and so on look at and make that super fast to serve.” And, it was a different space. Whereas we started with, like, we’ll do the bulk processing and ingest. Like, you’ve got a bunch of, JSON log files, you’ve got whatever. We do that very large scale stuff ‘cause that’s what Spark was for, the large scale MapReduce-like stuff. And then we’ll keep the data in an open format. Might be slower, but, like, it’s already out there. You can consume it downstream. And, it turned out that, it’s easier to go from that broad thing that’s really good at the scale and ingesting and super low cost and create versions in it that have the speed and features of the, super easy to use, like, smaller data for, business users thing. And there was a Swyx [00:55:02]: So start open, then optimize. Matei Zaharia [00:55:04]: Yeah, start open and start large. Like, in some sense, we started upstream of them. And there was a time when we both, like, listed each other as partners because we said if you used both solutions together, use Databricks for, like, your ingest and compute, and then serve the tables out of Snowflake, you get all the visualization, all the very fast stuff, like, that’s great. And then, we both realized, like, customers were telling us, like, “Why do I need this other thing? Why can’t I just query your tables?” And we said, “No, we’re horrible at that. Like, please use our partner for the SQL warehouse stuff.” And then they realized that, like, wait a minute, so much of the compute is moving upstream into this other thing. Like, we’ve got to stop that Swyx [00:55:43]: You have to go into each other’s territory, yeah. Matei Zaharia [00:55:45]: But I think we did start with, like, the bigger scope, and with the open thing and that’s important architecture. Like, as - again, it goes to enterprises, like, if your company’s existed for, like, thirty years, you’ve experienced, being locked into Oracle and, like, all kinds of, like, crazy things. And if you’re the CTO there and you’re setting up the architecture for the future for your company, you’re gonna wanna pick a foundation that’s open. And you only want, like, one way to manage data in your company, ideally. You don’t want, like, seven different systems. Swyx [00:56:17]: But, the open data format have won. Like, I think now every enterprise wants to put data in open data format. But, it was very controversial, like, back then. I think five, six. When exactly - one of the Snowflake founders wrote a blog called Matei Zaharia [00:56:31]: Yeah Swyx [00:56:31]: Choosing Open Wisely, which argued against Matei Zaharia [00:56:35]: Yeah. Swyx [00:56:35]: I think they might have taken it down. You have to find it on archive now. Matei Zaharia [00:56:38]: Oh, it’s, it’s never going away now. Matei Zaharia [00:56:41]: no, it’s still there. I love the perspective that only you guys will have because obviously you run the company. and I thank you for indulging this. It’s incredible, perspective. We’d love Swyx [00:56:52]: Maybe one last one. Matei Zaharia [00:56:55]: Yeah. Swyx [00:56:55]: As you were talking I think I have to give Ali a lot of credit. Matei Zaharia [00:56:58]: Yes. Swyx [00:56:59]: He’s an incredible CEO. I think he’s the perfect combination of IQ, EQ, technology obsession, execution, business acumen. Swyx [00:57:07]: and he’s also a founder, which makes a lot, make him, a lot easier for Matei Zaharia [00:57:12]: Yeah Swyx [00:57:12]: to, mobilize and execute. I think that’s, Matei Zaharia [00:57:15]: Oh, that was it? so you have Ali, and he, they don’t, like, okay. Swyx [00:57:20]: Well, a couple of other things, but I think Ali play a pretty big role in the, Matei Zaharia [00:57:23]: I Swyx [00:57:23]: Yeah. Matei Zaharia [00:57:23]: I was, I thought he there was, like, gonna be some technical, choice that he contributed to. Swyx [00:57:28]: Oh, no, I, well, Matei Zaharia [00:57:29]: He did for a lot of these. Like, there were forks in the road where he pushed for, like, one way, and then it became clear that, like, that was the right way. yeah. Swyx [00:57:37]: Yeah, there’s a whole book that needs to be written about how, like, the eight of you, like, work together and all that. I think there’s been profiles that people have done. Second one, not a cleared, question again. Mosaic, DBRX, Genie, and Specialized Models Swyx [00:57:48]: Mosaic. Matei Zaharia [00:57:49]: Stats are there. Oh. Swyx [00:57:50]: Mosaic. Matei Zaharia [00:57:50]: Yeah. Swyx [00:57:51]: A lot of people in our community are in, are curious on, like, what’s the the model story of Databricks, right? Swyx [00:57:56]: Like, when you guys bought Mosaic, like, the thing was like, “Okay, well, we’re gonna do fine-tuning. We’re gonna house model,” ‘cause they had, the Mosaic models. And it seems like you’re, you’re not doing that, and it seems like you’re going towards more of the, LTAP and, the harness stuff. What’s the story there? just Matei Zaharia [00:58:14]: Yeah. I guess when Mosaic started, I think it was well known or became most well known for releasing open source LLMs early on, and they were general models. before that, they were doing other things. They were about optimizing, training systems. So they had the fastest, like, image model training stack in the world and stuff like that. And then they decided to do LLMs, which was smart. They moved into it before ChatGPT, so they had some of the first open source LLMs. Swyx [00:58:43]: Yeah. Swyx [00:58:43]: We interviewed John Franco Matei Zaharia [00:58:45]: Oh, yeah Swyx [00:58:45]: Abi for 7B. Matei Zaharia [00:58:46]: Yeah, exactly. Yeah. Oh, yeah, very cool. Yeah. Yeah. So we, decided, even though we did launch a open source model DBRX and, we went up to, like, above the Llama Three scale, we decided that we really wanna focus on there’ll be so many people releasing models, and, instead of doing the general model where, like, a big part of the recipe is just throw in a lot of compute and just scale, we wanna focus on, like, the next step also of, let’s say you have the very smart model, how do you make it, useful? for us, it was a lot about automating, like, how. Like, making it very good at querying data. That’s the first party agents we have called Genie. so it’s like a virtual data scientist. Imagine, there’s someone who already knows all the stuff in your company inside out and knows all the machine learning libraries, all the data libraries, all the stuff on the web, and you can ask them questions? That’s, that’s what we wanted to do first. So that meant, like, let’s not focus as much on, like, let’s just train some frontier model, but let’s build a system using either external models or, fine-tuned, customized components. we’re still doing quite a bit of model training though, and in fact, we’re always, we’re procuring, like, lots of GPUs and stuff all the time to do it. and there’s a few places where we’re doing it. One is, there are many high volume use cases where if you have a specialized model, it’s just so much better than any of the general models you get. A nice example of that is understanding, like, documents, like PDF, Word documents, stuff like that, parsing them. If you’ve ever tried to do that, it’s frustrating ‘cause you send it to, like, like, Claude, Fable, or whatever, it, like, almost gets it, but it gets some things wrong, and it’s super expensive. You just burnt a huge amount of tokens plopping in an image into there. So our team, built this, document, vision model that takes a page and gives you back a nice JSON with all the components, and it’s very competitive. It’s like- Probably like 100X cheaper than those, frontier models and still better. Swyx [01:00:57]: Yeah. Matei Zaharia [01:00:57]: And that’s done by one of the researchers who came from DeepMind, was a founder of Adept, like very early scaling person, but focused on this. likewise we have, we’re doing specialized agents for part of what the coding agent does. And if you’ve seen the stuff on advisor models, Swyx [01:01:17]: Yes Matei Zaharia [01:01:17]: from Harvey, also from Swyx [01:01:20]: Anthropic has been putting Matei Zaharia [01:01:20]: Anthropic Swyx [01:01:20]: Commission also. Matei Zaharia [01:01:21]: Yeah. Swyx [01:01:21]: Yeah. Matei Zaharia [01:01:22]: And UC Berkeley one of my grad students there, wrote a paper called Advisor Models, I think before those came out. I’m sure others had the idea at the same time Swyx [01:01:30]: Yeah Matei Zaharia [01:01:30]: but that’s, something that helps a ton. So yeah, we showed some stuff just today at the keynote on Swyx [01:01:38]: Is it Parth? Oh, Parth? Matei Zaharia [01:01:39]: Parth, yeah. Parth Swyx [01:01:39]: Oh, he’s speaking at my thing. he’s doing Matei Zaharia [01:01:41]: Oh, nice Swyx [01:01:41]: continual learning bench. Matei Zaharia [01:01:42]: Yes. Matei Zaharia [01:01:43]: Yeah, I’m one of his advisors, at Berkeley. Swyx [01:01:44]: Oh, yeah. Matei Zaharia [01:01:45]: Yeah. Swyx [01:01:45]: We interviewed his brother, Chai. Matei Zaharia [01:01:47]: Oh, okay. Swyx [01:01:47]: ‘Cause he’s also at Abridge. Matei Zaharia [01:01:48]: Yeah. Cool. Swyx [01:01:49]: that, their family’s very smart. Matei Zaharia [01:01:51]: Yeah. Matei Zaharia [01:01:51]: Yeah. They’re, they’re awesome, yeah. So yeah, so we’re doing some of that and as we get experience with these in the first party agents, we’re also doing them with customers. So my feeling is, like, customizing models is gonna get way easier over time. That’s what we’re finding, ‘cause the base models are smarter, so they generate better traces in RL already, and then RL is about learning from your own past traces. And then synthetic data generation is way better, way easier now. we have pipelines just using open source models, like the same model generates training environments and trains itself and beats like Opus and GPT 5.5 and stuff at a task. So I do think it’s gonna pick up, like. The thing is, the ease of training the algorithms is only gonna go up over time. There’s a question of when it crosses into mainstream. Like, instead of this like, specialized document parsing thing we did where like you need a hardcore LLM researcher, when does it get easy enough that anyone can like plop in some stuff and describe a task? Swyx [01:02:53]: Yeah. Matei Zaharia [01:02:53]: Yeah. Swyx [01:02:53]: Well, what makes it easy? Interfaces. Matei Zaharia [01:02:56]: Yeah. Swyx [01:02:56]: And, unified APIs. Matei Zaharia [01:02:57]: Yeah. Swyx [01:02:57]: ‘Cause obviously if it’s not interoperable, then you cannot switch. Matei Zaharia [01:03:00]: That’s what we’re seeing with these like, with Omnigentt and Swyx [01:03:04]: Yeah Matei Zaharia [01:03:04]: composable agents, like you can have agents or, with specialized models, and then you can train the whole thing. I think that’ll help a lot too. Context, AI Runtime, and RL Fine-Tuning Swyx [01:03:11]: Yeah. The last thing I was gonna leave, this, I’m sequencing this, so I’m proud of myself. Satya, is, talking about this. I interviewed him at, Microsoft Build Matei Zaharia [01:03:22]: Yeah Swyx [01:03:22]: a couple weeks ago, and then he wrote this essay, which I’m sure you’ve seen Matei Zaharia [01:03:25]: Yes Swyx [01:03:26]: which is, talking about building frontier ecosystem. He sounded, when I was talking to him, more like a Databricks CEO than I’ve ever Matei Zaharia [01:03:32]: huh. Swyx [01:03:35]: is there a this thing presumably went viral in my circles. I don’t know if it’s in your circles. Swyx [01:03:41]: What’s the theory of like, I guess tokens as IP, building up the context? He said everything but data is the new oil or context is the new oil. Some version of that that you guys have heard before. Matei Zaharia [01:03:54]: Yeah, I agree. I think the data you have, as you get better technology around it, like you can just do more in your domain with it. It’s not even just about AI. Even when people, started collecting stuff in real time, like I remember all the power companies put like the smart meters and stuff, and all the car manufacturers started putting like sensors and cameras and stuff. Any technology like makes data more valuable and can give you some advantage, anything that helps you do something with it and make some decisions, and AI is the same way. Like you had all this stuff that’s just sitting there, now you can have an agent automatically tell you. Like for example, instead of I discovered as a, what feature in my product is broken ‘cause a customer complained, the agent tells me, “I noticed no one is like uploading files anymore ‘cause they get errors or whatever.” And as you saw with like Reyden, like as a database company, because we have all these, the history of all the queries and all the table layouts and like how they worked, we can build a new engine very quickly that, is good and we’re confident that it’s gonna be good. So I think this is right. I think the question is exactly how it will, land, but I do think like custom, model customization, which Satya talked about, is gonna get easier over time. Swyx [01:05:09]: Yeah. Swyx [01:05:10]: Which is why, by the way, I brought up the model thing, ‘cause they have their MEI things and you guys don’t. That’s the, that was the, to be the mental question. Matei Zaharia [01:05:17]: Yeah. We do have, We’re doing like RL fine-tuning as a service and, with a bunch of customers. We don’t have like. we have like preview customers, and we have a general, something called AI Runtime that’s like we get you GPU clusters on demand with a software stack in there that makes it easy to do training. So we didn’t like launch Swyx [01:05:38]: Do fancy name, yeah Matei Zaharia [01:05:39]: but that’s existed for a while. We’ve had like GPU compute for a while, and that’s where a lot of the Mosaic, stack went Swyx [01:05:46]: Yeah Matei Zaharia [01:05:46]: to help scale that. But yeah, we found that the engagements, like some of the. There’s two types of customers. There’s some who just want GPUs and libraries to like get data in and out and monitor, so that’s what AI Runtime is. And then there’s some that say, “Hey, can you work with me, build evals, build synthetic data, and create-” Swyx [01:06:05]: Yeah. The more forward deploy solutions architects. Matei Zaharia [01:06:07]: Yeah. And then that’s what we’re doing and as. And more things will transition from like being custom to not, but, that’s how it is today. Data, Agents, Security, and Customer Platforms Reynold Xin [01:06:15]: Going back to your original question, I think one of the thesis we have is the, once you can get the data in the right place, the AI models are becoming pretty good. The generic agents are fairly. Ali talked about Matei Zaharia [01:06:27]: Yeah Reynold Xin [01:06:27]: AGI is already here. They have pretty good reasoning capabilities. I think many of the traditional software will be rewritten, with this new paradigm, which is just get the data to be there, and then just slap some agent on top. Reynold Xin [01:06:40]: Magic will come out. Matei Zaharia [01:06:41]: Yeah. Reynold Xin [01:06:42]: but without the right data, you can’t really do that. And it’s our approach going to security and our approach going to the, customer data platform space Matei Zaharia [01:06:51]: Yeah Reynold Xin [01:06:51]: is, like we launched two products Matei Zaharia [01:06:54]: Yeah Reynold Xin [01:06:54]: at Data and AI Summit, one targeting security teams and the other one targeting marketing teams. And those all are, have a lot of existing technologies out there, and our, I think our approach is just, hey, once you get the data in, everything is a lot easier with agents on top. Matei Zaharia [01:07:09]: Yeah. Reynold Xin [01:07:10]: Well, and you guys have been fantastic guests. I just love this discussion. I just love the ability to dive in on the tech side, but also culture and strategy. I hope this isn’t the last time we chat. Like, congrats on all the success so far. Matei Zaharia [01:07:23]: Thank you. Reynold Xin [01:07:24]: Yeah. Matei Zaharia [01:07:24]: Congrats on your success also. Reynold Xin [01:07:27]: Yeah. Yeah. Databricks is supporting my, event, which is, so I Matei Zaharia [01:07:31]: Yeah Reynold Xin [01:07:32]: the AI engineer conference, and it is. I was, I’ve been an attendee of Data AI Summit for a long time, and I noticed that it was like. this was back in 2022. It was like 90% data and then 10% AI. Matei Zaharia [01:07:43]: Yeah. Reynold Xin [01:07:44]: And I was just like, “Well, okay, like we need a, we need the community thing that is like just 90% AI.” Matei Zaharia [01:07:49]: Yeah. Reynold Xin [01:07:50]: Which like now everybody is. Matei Zaharia [01:07:51]: Yeah. No, we’re excited to support. Reynold Xin [01:07:52]: so yeah. So Databricks will be at the conference. and I know, I just, it’s just amazing to see you guys, build out the most like interesting like cloud that I have I’ve seen outside of like the, the big three. And like it’s amazing how far you’ve grown. Like, Matei Zaharia [01:08:07]: Thank you Reynold Xin [01:08:07]: one of the, one of the most, insightful, like, I don’t, I’m not a VC, but I play one on TV. Reynold Xin [01:08:12]: like Ben Horowitz like when he was talking to you guys, advising you on just like where is this company going, he was like, “Don’t sell it to 100 billion,” or some some version of that story, right? Matei Zaharia [01:08:22]: Yeah, it was like the company should be worth a trillion dollars. You’re underselling it for 10 billion. Reynold Xin [01:08:26]: And like he doesn’t do that for everyone? Like for some reason, like, I think he saw the vision, but also, the infinite runway that you have. Matei Zaharia [01:08:36]: We’re lucky to have Ben. Yeah. Reynold Xin [01:08:37]: Yeah. Matei Zaharia [01:08:37]: He’s a big supporter. Reynold Xin [01:08:39]: Yeah, amazing. Okay, well thank you so much. Matei Zaharia [01:08:41]: All right. Thank you so much, Swyx. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Matei Zaharia

Co-founder & CTO

Databricks

Reynold Xin

Co-founder

Databricks

SaaStr 863: The Enterprise AI Reality Check: From Dashboard Graveyards to 30-Day Migrations with Databricks' Co-Founder and SVP of Field Engineering Every Fortune 500 CEO has told their team that if they are not using AI, they are behind. So now every employee is token-maxing, spend is going up, and almost nobody can tell you what they are getting out of it. That is the reality Databricks sees from the front lines, serving more of the Fortune 500 than any other data and AI company on the planet. In this episode, Databricks Co-Founder and SVP of Field Engineering, Arsalan Tavakoli, sits down with SaaStr CEO and Founder, Jason Lemkin, to cut through the Twitter noise and talk about what enterprises are actually doing, what is still broken, and why the next 24 months will fundamentally change who wins and who loses in every major software category. You'll learn: Why the BI dashboard is dead and what replaces it - including how a car manufacturer just onboarded 70,000 non-technical users to query their own data in plain language with no analyst in the loop What "context" actually means for enterprise AI and why it is harder to solve than the data problem, using a framework that explains why agents fail even when the underlying data is clean Why no software monopoly survives the next 24 months, and how collapsing migration costs and low-end AI competitors are about to give every incumbent a pricing problem they cannot ignore How Databricks now completes enterprise-grade migrations in 30 days or less using LLMs to analyze, convert, and reconcile legacy systems that previously took years and cost more than the savings Why the murky middle is the most dangerous place to be in enterprise software right now, and how to know which side of the AI budget divide your product actually sits on

Arsalan Tavakoli

Co-Founder and SVP of Field Engineering

Databricks

Ryan welcomes Bryan Clark, director of product for Lakebase at Databricks, to discuss what happens when AI agents become the primary creators and users of databases; why agents are “sloppy” about cleaning up infrastructure; and how database branching, scale-to-zero, and centralized access control can help teams keep up with agent-driven development. Episode notes: Databricks Lakebase is a Postgres-compatible operational database built around fast branching, separated compute and storage, and tight integration with the Databricks lakehouse. Connect with Bryan on LinkedIn and X. Congrats to Populist badge winner Benjamin Merchin for earning the badge for their answer to JSX element class does not support attributes because it does not have a 'props' property.ts(2607). See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Bryan Clark

Director of Product, Lakebase

Databricks

In this BG2 guest interview, Altimeter partner Apoorv Agrawal sits down with Ali Ghodsi (Databricks) and Arvind Jain (Glean) for a candid, operator-level discussion on what’s actually working in enterprise AI—and what isn’t. They unpack why 95% of AI projects fail, why LLMs are rapidly commoditizing, and why durable advantage is shifting to proprietary data, agentic systems, and workflow integration. The conversation dives deep into real-world use cases across finance, healthcare, and retail; the debate over whether we already have AGI; and how AI spend, CapEx, and valuation bubbles will realistically play out. A must-watch for builder, and investors navigating the AI transition inside real organizations. Timestamps: (00:00) Intro (01:00) Consumer AI vs. Enterprise Reality (02:15) Why 95% of AI Projects Fail (04:15) RBC, Merck, and 7-Eleven Use Cases (06:45) What Actually Makes AI Work (07:00) LLMs Are Commodities—Data Is the Moat (08:45) Failed AI Bets at Databricks & Glean (11:00) RPA vs. Generative AI (14:15) Advice for CIOs Planning AI Budgets (16:00) AI CapEx and the Revenue Math (18:00) The Three Camps of AI (21:00) Making AI Useful Inside Enterprises (24:30) Why Apps Capture the Value (30:00) The Future of UI, Voice, and Data Entry (37:30) Rapid Fire: Winners, Bubbles, Long/Short Produced by Dan Shevchuk Music by Yung Spielberg Available on Apple, Spotify, www.bg2pod.com Follow: Apoorv Agrawal @apoorv03 https://x.com/apoorv03 BG2 Pod @bg2pod https://x.com/BG2Pod

Ali Ghodsi

Co-founder & CEO

Databricks

Arvind Jain

Founder and CEO

Glean

Apoorv Agrawal

Partner

Altimeter Capital

Naveen Rao is cofounder and CEO of Unconventional AI, an AI chip startup building analog computing systems designed specifically for intelligence. Previously, Naveen led AI at Databricks and founded two successful companies: Mosaic (cloud computing) and Nervana (AI accelerators, acquired by Intel).  In this episode, a16z’s Matt Bornstein sits down with Naveen at NeurIPS to discuss why 80 years of digital computing may be the wrong substrate for AI, how the brain runs on 20 watts while data centers consume 4% of the US energy grid, the physics of causality and what it might mean for AGI, and why now is the moment to take this unconventional bet.   Stay Updated: If you enjoyed this episode, please be sure to like, subscribe, and share with your friends. Follow Naveen on X: https://x.com/NaveenGRao Follow Matt on X: https://x.com/BornsteinMatt Follow a16z on X: https://twitter.com/a16z Follow a16z on LinkedIn:https://www.linkedin.com/company/a16z Follow the a16z Podcast on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Follow the a16z Podcast on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details, please see http://a16z.com/disclosures. Stay Updated: Find a16z on YouTube: YouTube Find a16z on X Find a16z on LinkedIn Listen to the a16z Show on Spotify Listen to the a16z Show on Apple Podcasts Follow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Naveen Rao

Co-Founder and CEO

Unconventional AIDatabricksMosaicMLIntel

Ben Horowitz founded Loudcloud in the middle of the dot-com bust and sold it for $1.6 billion, then led Andreessen Horowitz from its founding to $46 billion in committed capital. Ali Ghodsi co-founded Databricks, stepped in as CEO during a crisis, and led it to a valuation of over $100 billion. In this episode of “Boss Talk”, Ben and Ali join a16z General Partners Sarah Wang and Erik Torenberg to share founder war stories, how to hire and make deals, how to keep culture intense without burning employees out, and why founders should raise their ambitions even higher.   Resources: Follow Ali on X: https://x.com/alighodsi Learn more about Databricks: https://www.databricks.com/ Follow Ben on X: https://x.com/bhorowitz Follow Sarah on X: https://x.com/sarahdingwang Follow Erik on X: https://x.com/eriktorenberg   Stay Updated:  If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://x.com/a16z Subscribe to a16z on Substack: https://a16z.substack.com/ Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Podcast on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Podcast on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Stay Updated: Find a16z on YouTube: YouTube Find a16z on X Find a16z on LinkedIn Listen to the a16z Show on Spotify Listen to the a16z Show on Apple Podcasts Follow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Ben Horowitz

Co-founder and General Partner

Andreessen Horowitz

Ali Ghodsi

Co-founder & CEO

Databricks

Ron Gabrisko is the Chief Revenue Officer at Databricks, where he joined in 2016. Under his leadership, Databricks has scaled from $0 to $3.7BN annualized revenue. He has grown the sales team from 0 to over 1,000 globally, leading expansion into enterprise, government, and international markets. Ron previously held senior sales roles at Cloudera and IBM, bringing deep experience in data and AI infrastructure. His tenure at Databricks has been defined by hypergrowth, multi-product adoption, and world-class GTM execution. Agenda for Today: 00:04 – The Databricks Origin Story: Ali, Ben Horowitz & 7 PhDs 00:08 – Ali vs JPMorgan: Turning Down $10M to Stay Cloud-First 00:13 – Prospecting Day: How Ron Scaled the GTM Culture 00:16 – Why Databricks' Pricing Model Was Its Secret Weapon 00:19 – Enterprise vs SMB: The Risky Bet That Paid Off 00:23 – From $2M to $13M ARR: How Ron Built the First Sales Engine 00:29 – Can AI Replace Salespeople? Ron's Brutally Honest Take 00:36 – How to Get Your First Million-Dollar Rep (and Keep Them) 00:42 – The Culture Secret Behind Scaling to 5,000 Sales Reps 00:45 – Why Databricks Waited Until $500M ARR to Go International 00:52 – What Makes a Great Sales Meeting? Ron's Gold Standard 00:58 – The Snowflake Wars: Why Ron Says Databricks Is 5 Years Ahead

Ron Gabrisko

Chief Revenue Officer

Databricks

In the latest episode ofThe New Stack Agents, Naveen Rao, VP of AI at Databricks and a former neuroscientist, reflects on the evolution of AI, neural networks, and the energy constraints that define both biological and artificial intelligence. Rao, who once built circuit systems as a child and later studied the brain’s 20-watt efficiency at Duke and Brown, argues that current AI development—relying on massive energy-intensive data centers—is unsustainable. He believes true intelligence should emerge from low-power, efficient systems, more aligned with biological computing. Rao warns that the industry is headed toward “model collapse,” where large language models (LLMs) begin training on AI-generated content instead of real-world data, leading to compounding inaccuracies and hallucinations. He stresses the importance of grounding AI in reality and moving beyond brute-force scaling. Rao sees intelligence not just as a function of computing power, but as a distributed, observational system—“life is a learning machine,” he says—hinting at a need to fundamentally rethink how we build AI. Learn more from The New Stack about the latest insights about the evolution of AI and neural networks:  The 50-Year Story of the Rise, Fall, and Rebirth of Neural Networks The Evolution of the AI Stack: From Foundation to Agents Join our community of newsletter subscribers to stay on top of the news and at the top of your game.

Naveen Rao

Co-Founder and CEO

Unconventional AIDatabricksMosaicMLIntel

This episode is sponsored by Oracle. OCI is the next-generation cloud designed for every workload – where you can run any application, including any AI projects, faster and more securely for less. On average, OCI costs 50% less for compute, 70% less for storage, and 80% less for networking. Join Modal, Skydance Animation, and today's innovative AI tech companies who upgraded to OCI…and saved. Try OCI for free at http://oracle.com/eyeonai What if you could fine-tune an AI model without any labeled data—and still outperform traditional training methods? In this episode of Eye on AI, we sit down with Jonathan Frankle, Chief Scientist at Databricks and co-founder of MosaicML, to explore TAO (Test-time Adaptive Optimization)—Databricks' breakthrough tuning method that's transforming how enterprises build and scale large language models (LLMs). Jonathan explains how TAO uses reinforcement learning and synthetic data to train models without the need for expensive, time-consuming annotation. We dive into how TAO compares to supervised fine-tuning, why Databricks built their own reward model (DBRM), and how this system allows for continual improvement, lower inference costs, and faster enterprise AI deployment. Whether you're an AI researcher, enterprise leader, or someone curious about the future of model customization, this episode will change how you think about training and deploying AI. Explore the latest breakthroughs in data and AI from Databricks: https://www.databricks.com/events/dataaisummit-2025-announcements Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A.I. on X: https://x.com/EyeOn_AI

Jonathan Frankle

Chief AI Scientist, Databricks

DatabricksMosaicMLMassachusetts Institute of TechnologyMIT

SaaStr 784: From $1M to $3B ARR: Databricks CRO Ron Gabrisko on Scaling a Revenue Rocket Ship In the latest episode of SaaStr's CRO Confidential Series, our host Sam Blond sits down with Ron Gabrisko, CRO of Databricks, to unpack the journey from $1M in ARR to crossing $3B ARR at the end of January 2025. With Databricks now one of the largest pre-IPO technology companies, with $10 billion of expected non-dilutive financing and a valuation of $62 billion, Ron's insights are gold for any revenue leader looking to scale. Dive into the journey of Databricks, from its early days with less than a million in ARR to becoming one of the largest pre-IPO technology companies. Ron shares his insights on the importance of having a technical background in sales, strategic hiring, and the evolution of Databricks' pricing and customer acquisition strategies. Learn how leveraging unique business advantages and building strong community ties with customers can drive exponential growth. Whether you're a startup founder or a seasoned revenue leader, this episode is packed with valuable takeaways. -------------------------------------------------------------------------------------------- SaaStr hosts the largest SaaS community events on the planet. Hey everybody - thanks to the 10,000 of you who came out to SaaStr Annual. We had a blast and big news -- we'll be back in MAY of 2025. That's right, the SaaStr Annual will be a bit earlier next year, May 13-15 2025. We'll still be back in the same venue, in the SF bay area at the 40+ acre sprawling san mateo county events center. Grab your tickets at saastrannual.com with code JASON50 for an extra discount on our very best pricing. --------------------------------------------------------------------------------------------

Ron Gabrisko

Chief Revenue Officer

Databricks

Today, we’re bringing you a bonus episode zeroing in on DeepSeek, the Chinese AI lab that’s recently taken over the news and the app stores, beating out OpenAI's ChatGPT. Max Zeff is talking about it all with Ion Stoica, Professor of Computer Science Division at UC Berkeley and the cofounder and executive chairman of software startup Databricks. Listen to the full episode to hear more about: Why Stoica believes the future of AI lies in "doubling down on open source." Microsoft's decision to host DeepSeek on Azure. What the U.S. can do to foster accelerated innovation – with a look back at SB-1047 and a look ahead to 2025. The controversy surrounding claims that DeepSeek used OpenAI’s models to train its own. Equity will be back next week, so stay tuned! Equity is TechCrunch’s flagship podcast, produced by Theresa Loconsolo, and posts every Wednesday and Friday.  Subscribe to us on Apple Podcasts, Overcast, Spotify and all the casts. You also can follow Equity on X and Threads, at @EquityPod. For the full episode transcript, for those who prefer reading over listening, check out our full archive of episodes here. Credits: Equity is produced by Theresa Loconsolo with editing by Kell. We’d also like to thank TechCrunch’s audience development team. Thank you so much for listening, and we'll talk to you next time. Learn more about your ad choices. Visit megaphone.fm/adchoices

Ion Stoica

Co-founder

DatabricksUniversity of California BerkeleyAnyscale

Berkeley professor Ion Stoica, co-founder of Databricks and Anyscale, transformed the open source projects Spark and Ray into successful AI infrastructure companies. He talks about what mattered most for Databricks' success -- the focus on making Spark win and making Databricks the best place to run Spark. He highlights the importance of striking key partnerships -- the Microsoft partnership in particular that accelerated Databricks' growth and contributed to Spark's dominance among data scientists and AI engineers. He also shares his perspective on finding new problems to work on, which holds lessons for aspiring founders and builders: 1) building systems in new areas that, if widely adopted, put you in the best position to understand the new problem space, and 2) focusing on a problem that is more important tomorrow than today. Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital Mentioned in this episode:  Spark: The open source platform for data engineering that Databricks was originally based on. Ray: Open source framework to manage, executes and optimizes compute needs across AI workloads, now productized through Anyscale MosaicML: Generative AI startups founded by Naveen Rao that Databricks acquired in 2023. Unity Catalog: Data and AI governance solution from Databricks. CIB Berkeley: Multi-strategy hedge fund at UC Berkeley that commercializes research in the UC system. Hadoop: A long-time leading platform for large scale distributed computing. VLLM and Chatbot Arena: Two of Ion’s students’ projects that he wanted to highlight.

Ion Stoica

Co-founder

DatabricksUniversity of California BerkeleyAnyscale

Ali Ghodsi, CEO of Databricks (now a $61B company), has become one of the top minds in AI, evolving from researcher to leading one of the biggest AI-centered private companies. In our conversation, he broke down the AI hype cycle in detail and made bold predictions for the future. He also shared his biggest operating lessons from scaling Databricks to a market leader and pivotal moments in his journey.   (00:00) Introduction to AGI and AI Challenges (01:00) Ali Ghodsi's Journey in AI (01:38) The Evolution and Impact of AI (03:59) AI in Enterprises and Consumer Adoption (05:25) Specialized AI Systems and Data Advantage (07:32) The Hype Cycle and Future of AI (08:22) Scaling Laws and Compound AI Systems (19:52) AI Use Cases and Industry Applications (28:29) Early Days of Databricks and Spark (38:43) Taking Over as Interim CEO (39:00) Focusing on Enterprise Sales (40:42) Innovating in Open Source (41:08) Challenges and Pivots in 2015 (44:31) Building and Scaling New Products (47:57) Leadership Evolution and Advice (54:45) Hiring and Promoting Leaders (01:10:44) Future of AI and Databricks Executive Producer: Rashad Assir Producer: Leah Clapper Mixing and editing: Justin Hrabovsky Check out Unsupervised Learning, Redpoint's AI Podcast: https://www.youtube.com/@UCUl-s_Vp-Kkk_XVyDylNwLA 🎥 Subscribe on YouTube: https://www.youtube.com/channel/UCugS0jD5IAdoqzjaNYzns7w?sub_confirmation=1 Follow on Socials 📸 Instagram - https://www.instagram.com/theloganbartlettshow 📱 X - https://twitter.com/loganbartshow 🎬 Clips on TikTok - https://www.tiktok.com/@theloganbartlettshow About the Show Logan Bartlett is a Software Investor at Redpoint Ventures - a Silicon Valley-based VC with $6B AUM and investments in Snowflake, DraftKings, Twilio, and Netflix. In each episode of The Logan Bartlett Show, we sit down with the people behind today’s most important startups and extract the tactics, lessons, and frameworks they’ve learned the hard way. Conversations span hiring to GTM, product, growth, fundraising and everything in between - collectively forming the ultimate playbook to make you a better CEO, investor or board member. Tap follow and enable notifications to stay ahead of the game.

Ali Ghodsi

Co-founder & CEO

Databricks

This Week in Startups is brought to you by… OpenPhone. Create business phone numbers for you and your team that work through an app on your smartphone or desktop. TWiST listeners can get an extra 20% off any plan for your first 6 months at https://www.openphone.com/twist⁠ LinkedIn Jobs. A business is only as strong as its people, and every hire matters. Go to https://www.linkedin.com/twist to post your first job for free. Terms and conditions apply Beehiiv. Power your newsletters with AI tools, referral programs, and ad network features—all in one platform. Get 30 days free and 20% off your first 3 months at ⁠https://www.beehiiv.com/twist * Timestamps: (0:00) Databricks’ Naveen Rao joins Alex Wilhelm (8:54) Regulatory challenges and open-source AI (10:02) OpenPhone - Get 20% off your first six months at https://www.openphone.com/twist (13:57) Meta's Llama models and reinforcement learning (19:46) LinkedIn Ads -  Get a $100 LinkedIn ad credit at http://www.linkedin.com/thisweekinstartups (20:55) Open vs. closed source AI models and advances (28:09) Precision in AI and enterprise applications (30:20) Beehiiv - Get 30 days free and 20% off your first 3 months at https://www.beehiiv.com/twist (52:08) Tech incumbents' evolution and AI competition (54:55) AI model sizes and synthetic data quality (57:29) AI inspiration from biology and future innovation (1:03:06) Databricks' growth trajectory and potential IPO (1:04:25) Elliot Easterling of bonbon joins TWIST for a Jam with JCal. (1:05:16) Bonbon.tech and the rewards platform overview (1:14:44) Gamification strategies and user engagement (1:22:18) Exploring growth opportunities and market expansion * Subscribe to the TWiST500 newsletter: https://ticker.thisweekinstartups.com Check out the TWIST500: twist500.com Subscribe to This Week in Startups on Apple: https://rb.gy/v19fcp * Check out Databricks: https://www.databricks.com/ Check out Bonbon: https://bonbon.tech Link to Newsom vetoes SB 1047 article: https://www.bbc.com/news/articles/cj9jwyr3kgeo The short guide to understanding data intelligence from Databricks: https://www.databricks.com/blog/short-guide-understanding-data-intelligence Mosaic AI Agent Framework post: https://www.databricks.com/blog/announcing-mosaic-ai-agent-framework-and-agent-evaluation Data Intelligence Platform product tour: https://www.databricks.com/resources/demos/tours/horizontal/introducing-databricks-intelligence-platform?itm_data=demo_center Mosaic AI new features and demo: https://www.youtube.com/watch?v=acIBdCOj88E Unity Catalog GitHub: https://github.com/unitycatalog/unitycatalog “Thinking Fast and Slow” by Daniel Kahneman summary on Medium - https://www.penguinrandomhouse.ca/books/89308/thinking-fast-and-slow-by-daniel-kahneman/9780385676533 * Follow Naveen: X: https://x.com/naveengrao LinkedIn: https://www.linkedin.com/in/naveen-rao-bba5b01/ * Follow Elliot: X: https://x.com/e_easterling LinkedIn: https://www.linkedin.com/in/elliotteasterling * Follow Alex: X: https://x.com/alex LinkedIn: ⁠https://www.linkedin.com/in/alexwilhelm * Follow Jason: X: https://twitter.com/Jason LinkedIn: https://www.linkedin.com/in/jasoncalacanis * Thank you to our partners: (10:02) OpenPhone - Get 20% off your first six months at https://www.openphone.com/twist (19:46) LinkedIn Ads -  Get a $100 LinkedIn ad credit at http://www.linkedin.com/thisweekinstartups (30:20) Beehiiv - Get 30 days free and 20% off your first 3 months at ⁠https://www.beehiiv.com/twist * Great TWIST interviews: Will Guidara, Eoghan McCabe, Steve Huffman, Brian Chesky, Bob Moesta, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland * Check out Jason’s suite of newsletters: https://substack.com/@calacanis * Follow TWiST: Twitter: https://twitter.com/TWiStartups YouTube: https://www.youtube.com/thisweekin Instagram: https://www.instagram.com/thisweekinstartups TikTok: https://www.tiktok.com/@thisweekinstartups Substack: https://twistartups.substack.com

Naveen Rao

Co-Founder and CEO

Unconventional AIDatabricksMosaicMLIntel

It’s return guest season here at Latent Space! We last talked to Kanjun in October and Jonathan in May (and December post Databricks acquisition): Imbue and Databricks are back for a rare treat: a double-header interview talking about DBRX from Databricks and Imbue 70B, a new internal LLM that “outperforms GPT-4o” zero-shot on a range of reasoning and coding-related benchmarks and datasets, while using 7x less data than Llama 3 70B. While Imbue, being an agents company rather than a model provider, are not releasing their models today, they are releasing almost everything else: * Cleaned-up and extended versions of 11 of the most popular NLP reasoning benchmarks * An entirely new code-focused reasoning benchmark * A fine-tuned 70B model, built with Meta Llama 3, to identify ambiguity * A new dataset of 450,000 human judgments about ambiguity * Infrastructure scripts for bringing a cluster from bare metal to robust, high performance training * Our cost-aware hyperparameter optimizer, CARBS, which automatically and systematically fine-tunes all hyperparameters to derive optimum performance for models of any size As well as EXTREMELY detailed posts on the infrastructure needs, hyperparameter search, and clean versions of the sorry state of industry standard benchmarks. This means for the FIRST TIME (perhaps since Meta’s OPT-175B in 2022?) you have this level of educational detail into the hardware and ML nitty gritty of training extremely large LLMs, and if you are in fact training LLMs of this scale you now have evals, optimizers, scripts, and human data/benchmarks you can use to move the industry forward together with Imbue. We are busy running the sold-out AI Engineer World’s Fair today, and so are unable to do our usual quality writeup, however, please enjoy our show notes and the excellent conversation! Thanks also to Kanjun, Ashley, Tom and the rest of team Imbue for setting up this interview behind the scenes. Video pod Timestamps * [00:00:00] Introduction and catch up with guests * [00:01:55] Databricks' text to image model release * [00:03:46] Details about the DBRX model * [00:05:26] Imbue's infrastructure, evaluation, and hyperparameter optimizer releases * [00:09:18] Challenges of training foundation models and getting infrastructure to work * [00:12:03] Details of Imbue's cluster setup * [00:18:53] Process of bringing machines online and common failures * [00:22:52] Health checks and monitoring for the cluster * [00:25:06] Typical timelines and team composition for setting up a cluster * [00:27:24] Monitoring GPU utilization and performance * [00:29:39] Open source tools and libraries used * [00:32:33] Reproducibility and portability of cluster setup * [00:35:57] Infrastructure changes needed for different model architectures * [00:40:49] Imbue's focus on text-only models for coding and reasoning * [00:42:26] CARBS hyperparameter tuner and cost-aware optimization * [00:51:01] Emergence and CARBS * [00:53:18] Evaluation datasets and reproducing them with high quality * [00:58:40] Challenges of evaluating on more realistic tasks * [01:06:01] Abstract reasoning benchmarks like ARC * [01:10:13] Long context evaluation and needle-in-a-haystack tasks * [01:13:50] Function calling and tool use evaluation * [01:19:19] Imbue's future plans for coding and reasoning applications * [01:20:14] Databricks' future plans for useful applications and upcoming blog posts Transcript SWYX [00:00:00]: Welcome to the Latent Space Podcast, another super special edition. Today, we have sort of like a two-header. John Frankel from Mosaic Databricks, or Databricks Mosaic, and Josh Albrecht from MBU. Welcome. JOSH [00:00:12]: Hey, glad to be here. SWYX [00:00:14]: Thank you for having us. Hey, so both of you are kind of past guests. Jonathan, you were actually one of the most popular episodes from last year talking about MPT7B. Remember the days when we trained large models and there was 7B? JONATHAN [00:00:30]: Yeah, back when reproducing LLAMA1-7B was considered a huge accomplishment for the field. Those are the good old days. I miss that. SWYX [00:00:38]: As the things have accelerated a lot. Actually, let's do a quick catch up and Josh, you can chime on in as well. So Databricks got acquired. I talked to you at New York. JONATHAN [00:00:45]: Mosaic got acquired, although sometimes it feels like Mosaic acquired Databricks because, you know, we're having a lot of fun being here. But, you know, yeah. SWYX [00:00:52]: Yeah. I mean, you are chief scientist now of Databricks. JONATHAN [00:00:55]: Chief AI scientist. Careful with the title. As much as I would love to understand how Spark works, I'm going to have to defer that to much smarter people than me. SWYX [00:01:03]: Got it. And I don't know about like what you would highlight so far as a post-acquisition, but the most recent news is that you guys released DBRX. Is that the thing that most people should be aware of? JONATHAN [00:01:13]: Actually, that's no longer the most recent news. Honestly, the most recent news, we announced this, but it was at our Data and AI Summit last week. So it was announced among like 100,000 other things, is that we finally released our text to image model, which has been a year in the making through a collaboration directly with Shutterstock. There was a lot of work put into finding a dataset that we were comfortable with working on and trying to build a model that honestly, I felt like I could trust and that others might be able to trust to put out in the world. So that model was released last week. It's unfortunately just available via API due to the fact that the data is quite sensitive and quite valuable. It's Shutterstock's entire business in a lot of ways, but I'm still really excited that there's now a model that is trained on a dataset where the provenance of every single image is known, and it's a damn good model. So I'm really proud of the team on that. SWYX [00:01:55]: Yeah, amazing. Josh, do you have any thoughts on image model questions? JOSH [00:01:59]: That is not my area of expertise, but I was excited to see the release of it last week as well, and very happy that you guys did a nice job on the data side of everything there. So that was cool to see. SWYX [00:02:09]: I think what's unusual is like, I think Shutterstock's doing multiple deals in multiple labs. So what is the Shutterstock model? Like, I guess, is this the house model for Shutterstock? Is this Databricks' version of the Shutterstock model? Like, what is this? JONATHAN [00:02:22]: The way that I would think about it is that Shutterstock is doing an amazing business in AI across the board. Their dataset is kind of widely known to be the best stock photos dataset in the world, the most comprehensive, the biggest. When you think about like, what dataset am I going to train a multimodal model on? You call Shutterstock. And I, at least I've heard in the news, like OpenAI, Google, Meta, Apple have all called Shutterstock and made those deals. So a lot of models have had Shutterstock data incorporated into them. But this is the only model I know of so far where it was, you know, exclusively and specifically trained just on the vanilla Shutterstock data. There was nothing else mixed in. We didn't go and scrape the web and find other data or combined datasets or anything like that. And so this is, in some sense, the house blend. But the other piece is that it's just a dataset where the provenance of every image is known in public. Where did the data come from? It is the Shutterstock collection. That's it. You know, nothing less, nothing more. And certainly being at Databricks, if I've learned one thing, I've learned about enterprise customers and what they want out of AI. And one of the things they ask for most is just, what can you tell me about the data the model was trained on? And here, especially for text to image models, where images are just tricky subject matter, there's been a lot of kind of legal conversation about images, especially. It's nice to just have something where I can point to it and say, you know, if you want to know where the images came from, these are what they are and this is how they got there. SWYX [00:03:36]: I will talk a little bit about Databricks because it's relevant to the rest of today's episode. So Databricks, sorry, I keep misspeaking. It's DBRX. JONATHAN [00:03:46]: DBRX, actually, there's been a pronunciation update. It is now D-B-Rex. So we have decided to add a dinosaur mascot because what model doesn't like a mascot? So literally, I wish I could pull it up. There is a little plush dinosaur that we had made. It's like the world's cutest dinosaur, but it is the official mascot of D-B-Rex. And there's a little dinosaur logo that, you know, you'll probably see around a little bit more because DBRX is a mouthful, but D-B-Rex, like, you know, it's just kind of... SWYX [00:04:13]: Rolls off the tongue. I love mascots. Like every company should have a mascot. And I think Hugging Face got it right. You need an emoji mascot because that's the minimal viable image. JONATHAN [00:04:21]: I probably shouldn't talk at all about, you know, Velociraptor, but, you know, that's a, maybe that's something we can talk about later in the summer. I'll just leave it at that. SWYX [00:04:28]: Okay. That's a hint to names. I feel like your names leak a lot of alpha. So just to quickly cover the headline details, DBRX, as Make Sure Experts model, that's fairly big, 132 billion total parameters, so 36 billion active on any input, pre-trained on 12 trillion tokens of text and code, and did really well on evals to the point where you had to dye your hair blue. That's my high level conclusion. JONATHAN [00:04:53]: Never make a bet with your team two weeks out from model launch, even when, you know, human eval is looking quite bad. Because if you set some bar, even if it's arbitrary and you think there's no way in hell they're going to hit it, apparently money doesn't motivate people anymore. Humiliating their boss motivates people. So Josh, you should really take a hint from this. You know, you cannot pay someone enough money to make up for you dyeing your hair blue. JOSH [00:05:15]: I'll keep that in mind for our next model. SWYX [00:05:17]: It works. So speaking of Imbue's next model, perhaps Josh, you want to actually just say hi to the general sort of latent space audience and talk about what we're releasing today. Yeah. JOSH [00:05:26]: I'm Josh, CTO of Imbue, and we're not releasing the model. We're not releasing the weights, but we are releasing a bunch of different things that should make it easier for other people to make their own models. So I think right now, training foundation models from scratch is like a very difficult, time-consuming, expensive, kind of risky endeavor, especially for smaller companies. And the things that we're releasing hopefully make that at least a little bit easier. So the things that we're releasing fall into kind of three different buckets. One is infrastructure and scripts for dealing with the kind of hardware and hardware failures and understanding how well is the actually lowest level of thing actually working so that you can actually do your training at all and at a reasonable speed without having to constantly restart, etc. So infrastructure and training scripts. A second set of things is around the evaluation. So after you've trained it, like how well is this actually working and how do you know how well it's working? We're releasing a whole bunch of different data there, a new benchmark about code, reasoning, understanding, as well as our own private versions of 11 different open source benchmarks. So things like pool queue or ANLI, where we've gone through and kind of cleaned up the data as much as possible by looking at all the ones that models get wrong or that are flagged for ambiguity and also our own kind of private reproductions of those where we've done like a kind of clean room black box, like, okay, this is what the data set is supposed to be. Here are some examples. Let's make our own version of this to make sure that there is no data contamination, etc. To make sure that we're actually, you know, not testing on train. And then I think a final thing that we're releasing there is around 450,000 human judgments about ambiguity and question quality, which we used in the process of cleaning these evaluations and we also hope will be helpful for other people training kind of similar models. And then the third thing is CARBS, our hyperparameter, our cost-aware hyperparameter optimizer, which was especially helpful for being able to experiment at much smaller scales and then scale those experiments up to the much larger scale kind of on the first try without having to retry it. You don't want to be training, you know, 10, 20 different 70B models. You really want to get these larger models SWYX [00:07:30]: right on the first try. JOSH [00:07:30]: And so the ability to kind of tune things very precisely and learn scaling laws, not just for, you know, the like data and flops, but also for learning rate and all the other hyperparameters and see like how should you scale these things up was extremely valuable to us as we were training the larger models. Yeah, that's a lot of stuff. SWYX [00:07:49]: Yeah, exactly. So there's a bunch of stuff JOSH [00:07:50]: we'll have to go through all of it. JONATHAN [00:07:52]: Yeah, I just want to throw in how excited I am about this. This is the stuff that nobody ever talks about. That is the difference between success and failure in this stuff. Like, can you get your cluster to run? Can you get software on your cluster? Can you figure out what broke? Because fault tolerance is still not really built into any of the fundamental primitives of training models. And so if something breaks, you have to go figure out what broke, your job stops, you have to restart your job. It is a nightmare just to get to the point where anything can train on the cluster. A basic MPI hello world that has the GPUs talk to each other is hard enough, let alone actually training a model, let alone getting good performance out of the GPUs, let alone actually getting a model that converges to anything interesting. There's so many levels of things you have to accomplish. This is the kind of stuff that matters. I think to a point that Josh made earlier, before we got on here, there are plenty of weights out there. Nobody's released this. JOSH [00:08:46]: Yeah, that was part of the motivation actually is that there are lots of other things that are complimentary, but I have not seen nearly as much discussion about some of these other things that we think are pretty important. I mean, in some sense, SWYX [00:08:56]: I'm very excited to have Jonathan on because this is a little bit, you're a bread and butter with Mosaic. And I think you've released some part with Composer. And I think it's just really interesting to see like a different take, basically a full stack take that's kind of open source today. JONATHAN [00:09:18]: Yeah, it's really kind of, it's been an ordeal to figure this out. And every time something changes, whether it's a new GPU or even a new driver update, you get new creative errors and new things go wrong. And, you know, we've dealt with the weirdest things from, you know, our InfiniBand cables getting stolen from the data center twice, like in boxes before they arrived at the data center. Like, you know, Porch Pirate basically had stolen our InfiniBand cables back when those were hard to come by. To like, you know, weird recalls of switches to like the strangest stuff has happened. I have my favorite GPU failures I've seen, like ones where the GPU doesn't fail, it has a correctable memory issue and the memory correction causes the GPU to become a straggler and hold up the whole job. Like weird stuff happens and figuring out how to not just identify all of that, but then eventually productize it, is in some sense, the entire story of Mosaic and now Databricks in terms of our ML offering. Really, the thing we offer is we have gone through this suffering and figured out how to even productize that. It has been a pain in the butt. SWYX [00:10:20]: Yeah, it's a lot of work. JOSH [00:10:20]: I think my favorite failure was GPU is just giving wrong math. Like if they give errors, great, because you can see the errors, but if they just give you the wrong math back, not so fun. SWYX [00:10:30]: When did they give you wrong math? JOSH [00:10:32]: Like literally you could just, you know, add two things. For example, the numbers come back. They're not the numbers that they're supposed to be. JONATHAN [00:10:40]: I think it's important to say at this stage, just because like it, I think it goes without saying for Josh and I, but it's worth saying here, this isn't to say that like anything is wrong with us. It's not like NVIDIA did a bad job or, you know, Mellanox did a bad job or the like the server builder, the data center operator, the cloud provider, like the million other parties that are involved in building this. We are running these insane chips that are huge and complicated and built on tiny transistors at insane frequencies with insane heat in data centers that for the most part, were not built remotely for this kind of power or heat and have been retrofitted for this. Like failures happen on a good day with normal CPUs. And this is not a good day and not a normal CPU for the most part. It's fun to joke about all the weird things we see. This is not to say anybody's done anything wrong. This is just kind of part and parcel of working on a massive cluster running at multiple megawatts of power at a time. SWYX [00:11:32]: It's crazy. Yeah. JONATHAN [00:11:33]: So optical cables, like all sorts, like everything. SWYX [00:11:37]: I'll take the opportunity to start going to the sort of infra piece. There's just like a description of the infra just to give people a sense of what we talk about when we talk about massive clusters. So I'm just going to read off the blog post here. This post is about one cluster that has 4,092 H100 GPUs spread across 511 computers. They use unified fabric manager nodes, which manage the infinite band network. And you talk a little bit about your networking. Is there anything unusual about this setup that you'll call out to people? JOSH [00:12:03]: Yeah, actually this particular cluster is a little bit non-standard. The normal, like vanilla setup for these large clusters as vanilla as it can be is what's normally like a 127 node cluster. So closer to like 1024 GPUs instead of 4,000. Here we have a larger cluster. As you start to get into the larger clusters, the networking becomes a little bit more custom. It's a little bit more, it's a little bit trickier. It's a little bit more difficult to get these things to all be able to talk to each other at the same speed. And so this has, in this particular case, this is a three tier network architecture instead of two tiers, kind of the normal one. So most of the clusters are a little bit smaller. As you get to even larger scales, then this becomes even much more complicated, SWYX [00:12:43]: much more expensive. JOSH [00:12:43]: So we chose this particular scale, kind of knowing our own workloads and kind of what we wanted to do. This was kind of the right size for us. But yeah, I think it's not exactly vanilla already. It's already getting into kind of the custom territory. SWYX [00:12:54]: So my understanding is that there, and is there any part of this that comes with the Voltage Park deal that you guys had? Is that part of the hardware that you got from the deal with them? JOSH [00:13:04]: Yeah, so we worked really closely with Voltage Park to set up all their clusters and infrastructure and everything and kind of decide even like what to order, how should the networking work? Like we were very involved in kind of the construction and bring up of this. And that's what this post is about, is about that process of like bringing up all these, there's like different clusters in different places of different scales. So in this particular post, we're talking about this one 4096 GPU, but there are other clusters that they have as well. And we were very closely involved with figuring out the exact architecture and kind of the trade-offs that go along with picking, you know, those exact components. You really don't want to like place the wrong order because it takes months to get it and it's very expensive. So yeah, we were happy to help out with that. JONATHAN [00:13:43]: And then your bit of good cables get stolen. SWYX [00:13:44]: Yeah, yeah, exactly. JOSH [00:13:47]: We wanted to make sure that we ended up with compute that would work for us and that would also work for their other customers. And so we kind of helped design something so that we would get exactly what we were looking for. We knew that these kinds of details would be super important and that getting down to the level of the hardware and like having these good scripts and everything was going to be a core part of like actually getting this to work. I'm very glad that we did that. I don't think that most companies kind of take that full stack approach, but for us, it certainly paid off. SWYX [00:14:12]: Yeah, it's basically sort of built to spec. It's interesting that relationship because you usually, for the rest of us who don't operate at your scale, we take whatever we can get from cloud providers, but you are basically co-designing from the single machine up. And you described that a little bit. Do you want to take us through the process that you described here? JOSH [00:14:27]: Yeah, so for the actual, like the blog post and kind of bringing these machines online. SWYX [00:14:32]: Yeah. JOSH [00:14:32]: So yeah, I think the process, as we have it broken down in the blog post, there's kind of a few different layers. First is like getting the individual machines to work at all and then getting the machines to actually be able to talk to each other. So getting the InfiniBand networking to work and then getting to a point where, you know, not just the machines are working and they can talk to each other, but everything is actually working correctly. There's a big gap between like it's working at all to it's working perfectly correctly. And then after you have all this stuff working perfectly correctly, nice and healthy, then now you get into kind of the software data, like training issues. And then after that, you're still not done. Like now, even once you're training at full speed, things are going to fail over time. Things are going to change. There's going to be new, you know, firmware updates. Like how do you kind of deal with this change and flux over time without going crazy SWYX [00:15:16]: and pulling your hair out, JOSH [00:15:16]: trying to like reproduce things or understand why there were regressions. And so there's a lot of work to kind of automate the infrastructure tooling as well. And kind of the first step, like bringing these things online in the first place, you know, you have hundreds of machines at this point. So you don't necessarily want to be like walking around with like a CD-ROM or a USB drive, like plugging it in with your keyboard, like hitting next, next, next on the OS install. That's not how this works. You do that for one machine. And then you use, we use this thing called Metal as a Service to bring up all the other machines. So it's a kind of server that can kind of install the operating system on these other machines. So most like when you're talking about these machines, like each machine is, you know, on the order of hundreds of thousands of dollars. So they usually come with a kind of out-of-band management interface as well. So they don't, they have their InfiniBand networking. They have their normal 100 gigabit per second Ethernet networking. These are like dual, redundant, et cetera. And then you also have this extra out-of-band management network. So you can log in and you can see like the boot screen or you can see the blue screen of death. You can like get in there and actually see what was wrong, which is pretty fun. And it makes it like possible to automate a lot of this work. So the beginning of that, and the blog post goes into much more detail about like exactly how we set these up and kind of the other errors that we ran into. When you're bringing these online, you'll definitely have failures. Even if they all worked in the factory, they get shipped, some parts come loose, something fails, something goes wrong. So when you're bringing them online, there'll be some that don't quite work for all sorts of reasons. As you start to be working with machines at this scale, like if something happens one in a thousand times, you're like pretty likely to see it. And so you can get pretty rare, weird things, especially since we had fairly early builds and fairly early versions of this hardware. Like these are some of the like first machines that were ever produced, some of the first GPUs. So you've got some extra special things there. We definitely worked with Dell, for example, on making fixes in the firmware level to be like, okay, like this thing is wrong. Like we need to update this at the firmware to like actually fix this particular thing. So we worked pretty closely with Dell and Nvidia. Yeah, that's what I'm saying. Like this stuff gets complicated. And the thing is like, you know, taking a step back, the whole reason we're doing this, right, is that we knew that this was going to be complicated. There would be these kinds of failures. And if we're just using, you know, AWS or some other cloud provider, these errors are still gonna be there and you're gonna have no way to know and no way to debug this and no way to diagnose what's going wrong. And so we would much rather be able to like call up Dell and say, hey, this isn't working. And they're like, yep, okay, cool. Let's debug it together. Oh, I see. Yeah, cool. We'll ship a firmware update and actually fix this for you. That was a much better experience than like, great, just magically fails. I guess we restart and hope that that machine goes away. Like that's not a very good place to be. So yeah, that's kind of the first place is getting to a place where like GPU training is working on your single node machines. You can observe stuff. We have tons of tooling around like, you know, Prometheus and all sorts of other tools for understanding what's going on in these machines because you don't want to be like logging into each one and looking at the temperature or something you really need to have tooling to collect all these metrics, et cetera. Unfortunately, all of the scripts that we have for this are like for this entire cluster and for all this infrastructure are a little bit like special purpose for our particular thing. So it's not that every script that we have, it's not that you can just like take this and plug this in. Even if we did open source all the tooling that we have, you'd still have to do like a lot of work to open source it. What we are releasing is as many of the things that we can that are going to be useful for other people. You're still going to have to have some way of kind of managing these things, making your own like logging aggregators, et cetera, et cetera. So that's kind of bringing them up to the like, you know, the single nodes that are working. From there, it goes into, I'm happy to keep going if you want. Well, I just want to leave the opportunity for John SWYX [00:18:53]: to comment if there's anything that's different from how he runs things. JONATHAN [00:18:57]: Oh, I mean, all I'll say is I'll endorse this and say this s**t is hard. Like this is really, really hard. And, you know, I have a special props to, you know, the folks in Vue because they were building this from the ground up. You know, at Databricks and at Mosaic, we typically work with cloud providers because some of this stuff is just, there's too much to handle. It's complicated. There's a lot to deal with. And this doesn't even get into things like physical security, you know, securing power if you're the data center operator. Like this gets infinitely complicated and you have to abstract somewhere. Like, you know, and then you get to the folks who are literally building their own custom chips and like, good God. SWYX [00:19:36]: Like, oh my God, that's, you know, JONATHAN [00:19:38]: if you're one of those folks, you're having, you know, pour one out for the infra people at some of the AI chip startups who are having a really, really interesting time right now. But this stuff is really hard. And I don't think we talk about it much because there's so many other things that are hard. But the other hard things, I think everybody's becoming pretty familiar with at this point. This is something that I don't think there's ever really been a comprehensive discussion of, at least not that I've seen. SWYX [00:20:00]: Yeah, so my impression is that you guys, Mosaic, have your own software for sort of spinning up and down machines, just like Imbue had to build. But Imbue probably, it sounds like Imbue, you guys went fuller stack. I don't know how to describe it. Like Mosaic is not working with Dell on like their firmware. JONATHAN [00:20:21]: No, no, we're typically working with like, you know, pick your cloud provider on their Dell firmware or what have you. Like, it's kind of, I think one of the things, I don't know, Josh, you can correct me on this. It's kind of impossible if you're doing training to not go all the way through the entire stack, regardless of what happens. Like somehow I'm still chatting with cloud providers about power contracts, even though the whole point of dealing with the cloud provider is not to have to think about power contracts. Somehow I'm still asking them about which InfiniBand provider they used this time to see if this is part of the bad batch of cables I encountered on that cloud provider or what have you. Or like, we're still talking about a firmware update from pick your provider. You can't not do this. It's convenient that they have data center staff who are worrying about what to send back to which provider when, and they have people who can go and wait for the InfiniBand cables so they don't get stolen outside. But, you know, it's kind of, it's impossible not to really go full stack if you're thinking about the infrastructure at all. I don't know, Josh, correct me. No, I think that's right. JOSH [00:21:17]: That's what we expected from the beginning as well, is that we would inevitably have to get into the details here. And I'm glad that we kind of just planned for it. I think it made it a lot easier from our perspective to have direct control over this. Instead of having to go to the cloud provider that goes to the data center, that goes to the supplier, we could just go direct to NVIDIA or Dell SWYX [00:21:37]: or the data center, JOSH [00:21:37]: whoever was responsible and be like, hey, this thing needs to change. And they're like, oh, okay. Yeah, that is our responsibility. Great, we can fix that. So it was just a lot easier for us to fix these bugs than if we had to go through an extra layer of email. SWYX [00:21:48]: Something we discussed in the pre-show was that you had a rule of thumb for your cluster of reliability. You say here in the post, by and large, you expect around 3% of your machines to break every week. So you're basically going to turn through all your machines in a year. JOSH [00:22:04]: As it says in the post. So that would be true if it was a uniform failure like that. But as it says in the post, it's usually these kind of problematic nodes. And to be clear, that is the number that we've heard from other people is like they're having about 3%. I don't think we're experiencing failure rates that are that high. I think ours is actually quite a bit lower than that, probably because we've taken the time to like dig into a large, maybe larger number than we should have of these failures and get to the root cause of it and be like, oh, okay, like that's exactly what's going wrong. SWYX [00:22:33]: How do we fix this? JOSH [00:22:33]: How do we prevent this from happening? How do we make automated checks for this so that if it does happen, it just goes back to whoever owns that particular part of the process and they can fix it immediately. SWYX [00:22:43]: And that's part of what you're also open sourcing, which is the health checks, right? You got the NIC health checks, GPU health check, this space health check, Docker D message. I don't know what that is. JOSH [00:22:52]: That one is just a lot of stuff. SWYX [00:22:54]: Yeah. JOSH [00:22:55]: That one is one where we realized that actually like when these machines boot, sometimes they wouldn't actually boot cleanly all the way. Or when they rebooted, they had problems that they didn't have when they were working before, which was kind of frustrating. Like usually if you restart your computer, SWYX [00:23:08]: it gets better. JOSH [00:23:08]: Here you restart. It did not get better. SWYX [00:23:10]: It got worse. JOSH [00:23:10]: That was very frustrating. So this health check looks at every particular line we've ever seen from the boot, like in D message, like every single log line that your computer emits SWYX [00:23:21]: and says like, JOSH [00:23:21]: have we ever seen this before? SWYX [00:23:23]: Is this expected? JOSH [00:23:23]: Is this in the right order? Or is there something out of place? If there's anything out of place, let me say, okay, great. Like now it goes into this, like longer, more triage list of like, all right, great. Like, is this acceptable? SWYX [00:23:33]: Should we flag this? JOSH [00:23:33]: Like, should someone take a look at this? So we're looking down at a very, very granular detail level, what's happening on these computers to make sure that nothing is out of place. And that's critical because without that, if you're running your training, as Jonathan said, and this thing is slow, like what are you supposed to do? Right? SWYX [00:23:49]: Like you really, JOSH [00:23:49]: you really want to be very certain that like all 4,000 of these GPUs are working like they're supposed to. SWYX [00:23:54]: We know that. JOSH [00:23:54]: And so if it's slow, it's because like we messed up the config or something else and not because of this earlier thing that's like really hard to detect in software later. JONATHAN [00:24:01]: Yeah. I think the, I'm just curious to ask, SWYX [00:24:03]: like, you know, JONATHAN [00:24:03]: suppose you were to set up another, let's say another H100 cluster and it were at a different data center. And instead of the vendor being Dell, it was super micro or what have you. How much of this would be repeatable? And how much of this would you have to redo? I, you know, I genuinely don't know. SWYX [00:24:18]: A decent amount. JOSH [00:24:19]: I think it would go a lot faster the second time. I think there's lots of learnings that we had. And also the blog post, SWYX [00:24:24]: you know, yes, JOSH [00:24:24]: we are releasing the health checks, releasing some scripts, but a lot of the valuable stuff is also in the blog post itself, in the details and kind of the, you know, the learnings that we've had and the sort of errors that we run into. We tried to as much as possible surface those to other people SWYX [00:24:36]: could learn from those JOSH [00:24:36]: and avoid the same mistakes or failures as well. But I think it would go a lot faster. SWYX [00:24:41]: Although, yes, JOSH [00:24:41]: there would certainly be some things that'd be a little bit different. I mean, there'd probably be different CPUs SWYX [00:24:46]: or whatever, JOSH [00:24:46]: but I think a lot of that stuff is less, SWYX [00:24:49]: it's less, JOSH [00:24:49]: that's the like, that's less variable. I think most of it would apply the second time around. Although I'm sure next time SWYX [00:24:56]: we're building one, JOSH [00:24:56]: it'll probably be, you know, at a scale that's 10x as big with a different chip or something like this. SWYX [00:25:00]: And then who knows? JOSH [00:25:01]: Yeah, with Kinect X8, JONATHAN [00:25:02]: that will have its own fun behavior and all that good stuff. Yeah. SWYX [00:25:06]: Perhaps there's something that people don't discuss about, and you don't even talk about this in the blog, but I always wonder is what is the timeline that's like kind of reasonable for this amount of work, at least the initial stages? And also what does the team composition look like for setting up a cluster, right? Like what are the mix of skills that you typically would require to get all this going? JOSH [00:25:27]: I'm, I can't really speak to typical. One thing I am very proud of is how much we accomplished with such a ridiculously small team. Like our infrastructure team is like, you know, fluctuates from week to week, depending on like how many things are on fire and how much we need to build. But it's like between like three and six people, like it's small. It's not like some huge team of like tons and tons of engineers. But those people are very, very good at what they do. And so that has allowed us to get a lot of mileage out of out of these things. I think it's not that we're building everything, right? It's not that three to six people build this whole thing. I definitely want to like, you know, say thanks very much to Dell and H5 and NVIDIA and the other people that have done a lot of the work, like to bring up this cluster, you know, with 4000 GPUs and three tier networking, networking architecture, you have 12,000 cables. So that's 24,000 things that need to be plugged in. Like that's just a lot of stuff to plug in, right? And you don't want to mess it up. Like each one needs to be done correctly. Like it's a little bit loose. Like it doesn't really work. SWYX [00:26:23]: If you break it, JOSH [00:26:23]: you need to replace it. Like there's a lot of work SWYX [00:26:26]: that goes into this. JOSH [00:26:27]: Yeah. SWYX [00:26:28]: And then, you know, JOSH [00:26:28]: that's just like that's it. That's if you were to do everything right the first time. SWYX [00:26:32]: And if you didn't JOSH [00:26:32]: have to fix anything. But inevitably, you know, you will have to replace something, which means like taking all the wires out, pulling the thing out, taking all the GPUs out, going and fixing some cable, putting it all back correctly, putting it back in, doing this every time. So there were a lot of people at Dell, NVIDIA and at H5 that all helped a ton with this stuff. I don't know the exact size of the Dell team. It also fluctuated over time. SWYX [00:26:55]: Yeah, excellent. And then, you know, you so you have all the hardware set up and now you're firing it up for a single node. There's a long description that you guys have about just like monitoring the MFU, right? And what each situation might look might be indicative of. One of the most interesting things to me that I saw from here is like, you know, if training immediately starts off at 60 to 80% MFU, something's wrong. SWYX [00:27:24]: But like, you know, like what what are like, you know, some anecdotes or, you know, notable scenarios here that you might you might call out as maybe counterintuitive or super interesting. JOSH [00:27:36]: There's just so many of them. I mean, one of them, which I think is probably pretty common, like common knowledge by this point. But like we did have a sort of like SWYX [00:27:46]: which one was this exactly? JOSH [00:27:47]: I think for the MFU, like gradually getting worse over time. I think that one, when we saw that the first time we were like, what the heck is going on? Like, why does it get just like a little bit worse? This is so strange. Like, what is it getting lazy or tired or something? Like, is it heat? Like what's going on? And in this particular case, it was memory fragmentation. Because you have hundreds of machines, they're doing garbage collection slightly different times. And then they get slightly further apart and slightly more and more jittered until eventually they're all happening kind of at random times. And just like really messing up each one of your steps. So you just turn off garbage collection and call it a day, basically, SWYX [00:28:20]: to be honest. JOSH [00:28:20]: There's other things you can do if you want to be a little bit more sophisticated about it. But you can also just manually JONATHAN [00:28:25]: have it all garbage collect on some interval. Like that's what we've done. We just have a garbage collection callback that just runs. But I've seen the exact same thing. JOSH [00:28:33]: Yeah, yeah, exactly. So I thought that one was kind of funny. And we did trace that one down and look and we did find the actual call. Like, again, this goes to like having good tools. So we had really good tools where we could look at a bunch of like actual traces in C and be like, OK, cool. This is the thing that's taking a lot of time. Or like, you know, this is the thing that doesn't quite line up here. Like, oh, I guess it's garbage collection. OK, cool. SWYX [00:28:52]: Interesting. JOSH [00:28:52]: Yeah, let's just try taking it off. SWYX [00:28:54]: OK, great. JOSH [00:28:54]: That's what it was. Now we can fix it. So for each of them, like basically bugs are not hard if you have good tools. But if you don't have good tools, bugs can be very, very hard. So similarly for like heat, another thing that we saw was like, oh, you know, the CPU is getting throttled. OK, well, it's easy to see if you're monitoring the CPU throttling or monitoring the heat. If you're not monitoring that, it's really hard to know why it's just suddenly one of them is going slower. I noticed also in the piece SWYX [00:29:17]: that you mentioned FSDP with 0.3. Actually, we met, I went to iClear and Guanhua from the DSP team was there presenting 0++. I was wondering if you want to make any call outs to, you know, particular open source or open library or open whatever implementation teams that were super helpful in your process. I think we ended up actually JOSH [00:29:39]: pulling from a whole bunch of different ones to pull things in into our own particular pipeline. So we use things from NVIDIA's, you know, Megatron stuff. We use stuff from probably DeepSpeed. I think we pulled in a bunch of different pieces from a bunch of different places. So it was really nice to see all these working open source like examples. I think I really appreciate all the effort that has gone into actually tuning these things because you can tune them, but it's a lot of work to like tune this stuff and do all this stuff from scratch. It's really nice to have like a working example. I think those are probably the two biggest ones, DeepSpeed and Megatron alone, but there are probably other ones as well. SWYX [00:30:13]: Is there a particular thing in the ecosystem where you would call out as like, you know, there should be something here that is open source, but like it's not really, it's like everyone kind of builds it on their own. I want to say something with the file system because everyone talks about the file system eventually. JOSH [00:30:28]: The file system actually was, SWYX [00:30:30]: I mean, we did something JOSH [00:30:31]: kind of dumb there. Like we have our own sort of local mirror so that we can, you know, like a crappy version of S3 SWYX [00:30:38]: that's local, JOSH [00:30:38]: but it's just a pretty simple script, right? SWYX [00:30:41]: Like I think we run like JOSH [00:30:41]: a little web server that just like serves files and then, you know, it can upload them SWYX [00:30:45]: and download them. JOSH [00:30:45]: Okay, great. And part of the reason we did that is that our internet connection SWYX [00:30:50]: in the beginning JOSH [00:30:50]: was not the like full speed SWYX [00:30:52]: one that we would JOSH [00:30:52]: eventually have. And so we are a little bit more kind of bottlenecked in terms of internet bandwidth. And so we had this. I think we looked at a bunch of services out there like Minio and some other ones, but a lot of these like come with a lot of extra overhead and maintenance. And since we already have so much infrastructure SWYX [00:31:09]: to deal with, JOSH [00:31:09]: we kind of didn't want to, you know, bring in a whole other like cloud provider, virtualize something, something. SWYX [00:31:14]: We just wanted something simple. JOSH [00:31:14]: So we went with that, which has been quite helpful. Like our tools SWYX [00:31:19]: are usually quite simple. JOSH [00:31:19]: It's like Bash and Python and SSH and Docker. Like we'd like to keep things simple so that's easier to debug, like less layers of infrastructure, less layers of abstraction, make it a lot easier to work with. Like we don't use Kubernetes, SWYX [00:31:30]: for example, JOSH [00:31:30]: and we just directly launch these things. And it's just been much easier to debug this way. One tool actually that does come into mind that I will call out is Kraken from Uber. That was great. We love that tool. We were a little bit skeptical. What is it? SWYX [00:31:44]: I'm sorry. Yeah. JOSH [00:31:45]: So Kraken is this, yeah, it's a distributed like Docker registry, basically, that uses BitTorrent to like transfer things between the machines in a sort of nice optimal way. Like in the very beginning, the naive way is like you have this one Docker registry, which was outside of the cluster. So every time we change an image, you know, there's many gigabytes that each of the 500 machines needs to download. SWYX [00:32:07]: So that just takes JOSH [00:32:07]: a really long time. So what this thing does is like just one of them downloads it and then like they all sort of broadcast all the pieces to each other. And it was just like a really nice, fast way of getting these images down. And it was very robust. SWYX [00:32:19]: Like there's a lot JOSH [00:32:19]: going on under the hood, but I think it's a pretty cool tool that we haven't really had any bugs with it at all. Amazing. SWYX [00:32:26]: Yeah. I mean, that's all my questions, I guess, for the info piece. I don't know if, John, you had something that you were sort of burning to ask or. JONATHAN [00:32:33]: No, all I can say is just same SWYX [00:32:36]: in a lot of places, like, you know, and they're done that JONATHAN [00:32:38]: seeing this plus one. I think the one big difference, you know, perhaps in philosophies is we've tried to basically standardize on as much commodity stuff as possible, just because, you know, I think the reason I asked about trying to do this SWYX [00:32:50]: on multiple different JONATHAN [00:32:50]: pieces of infrastructure is like, I think we're running on like six or seven different clouds right now. And everybody has done something slightly different. And my gosh, the little differences add up as you know, you've seen. And so, you know, SWYX [00:33:04]: our philosophy has been like, whatever the hell JONATHAN [00:33:05]: we can standardize, please let's standardize it. Like vanilla off the shelf FSDB. SWYX [00:33:10]: And like, you know, JONATHAN [00:33:10]: we wrote our own data loader, but we've tried to make that as much of a standard as we can across our infrastructure and in Databricks, because things just start getting really complicated SWYX [00:33:18]: or like we use JONATHAN [00:33:18]: Kubernetes extensively because it at least gives us a uniform set of APIs. Like that's our hardware abstraction layer to a certain extent for everything else. So it's just, you know, a difference in philosophy there. But otherwise, like, yeah, this stuff is really, really hard. And I feel like we take for granted how much of this, you know, is done for us when you go and you just query chat GPT, for example. Like, oh my God, everything going on underneath that, you know, it's kind of a miracle that the machines boot up, let alone that you can like query a giant language model that's probably doing inference across multiple machines and was trained across thousands of machines. Like, you know, minor miracle. SWYX [00:33:54]: Yeah, it is an awesome amount of power that we invoke with a single API call that we take for granted these days. It's absurd. Yeah, I mean, like Kubernetes, like that point about Kubernetes, I will say as a former AWS employee, like it seems like it would be ideal for imbue to at some point make it more abstracted or agnostic because you're going to want to, you know, replicate your setup. We do have our own JOSH [00:34:19]: sort of replacement. It's just a much simpler version of Kubernetes. Kubernetes is really designed for running services, not for running experiments. Like that's not its like main architecture. And so for us, like we have everything that's like, cool, you're going to run an experiment. So you want it to run to completion, right? SWYX [00:34:34]: OK, great. JOSH [00:34:34]: Like the primitives are sort of built around a slightly different style. And that makes it a lot easier, like just a lot simpler to fit that the nature of like these machines are going to disappear. They will need to be rebooted for infrastructure upgrades. They will like something will happen to the GPUs. Failure is like baked into this as like a core part of our infrastructure. So it's not that we don't have an abstraction. It's that it's a sort of simpler, more tailored abstraction for the particular work that we're doing. JONATHAN [00:34:58]: Yeah, I think it all depends on what your goals are. And like, I think the challenge in a lot of the deep learning stuff right now is that people are trying to like, people often build things that are more complicated than necessary to get the job done. And the complication is the enemy of everything. You know, don't use a fancier parallelism strategy than you have to. Don't use a fancier set of libraries than you have to. SWYX [00:35:18]: Don't do anything JONATHAN [00:35:18]: that you don't have to do because it's hard enough as it is. Like, don't overcomplicate SWYX [00:35:23]: your own life. JONATHAN [00:35:23]: Don't try to bring in more tools or more fancy architecture tweaks if you absolutely don't have to. SWYX [00:35:29]: Like getting to the minimum JONATHAN [00:35:30]: necessary to get the job done. And it's really tempting to want to try to use everything. So like, I totally understand that one. SWYX [00:35:37]: I think the last piece I'll maybe call out is that I'm just going to weave this in just because I see the opportunity to do it. Are there any infrastructure shifts that need to be, that need to rise because of changing architecture? So I think, for example, SWYX [00:35:57]: you're announcing a dense model, a 70B dense model, whereas John just worked on DBRX and the image-to-text model, which presumably has different bottlenecks. JONATHAN [00:36:10]: That's correct for us. You know, we train both dense and mixture of expert models. The one we happened to, you know, kind of get permission to open source was a mixture of expert model. And those models are very demanding when it comes to network bandwidth, at least if you're training them in kind of FSTP 03 style, where there's just a lot of parameters getting shuffled back and forth. And your ratio of kind of compute to amount of data that you have to shuffle back and forth becomes a lot worse because you're now, you know, you're only using a fraction of the parameters for every token instead of all the parameters. And so we had to really push the envelope on getting all the stuff to the right places on time. And so actually the networking part of DBRX was the single hardest thing, I think, of the entire process. Just get MOE training, working at scale across a big cluster. We still managed to, I think, do it all with commodity parts, which was very exciting. You know, we were using FSTP and we eventually used HSTP so that we could have HSTP as a version of FSTP where you have multiple smaller replicas and you're doing data parallel within those replicas. And that helped a lot with network latency issues that we were running into just because we were transmitting so much data, you know, for every single part of the process. I think it actually, like, it was instructive for how Google designs their hardware and software together personally. Their training, as far as I understand, using kind of a 03 style of training and have been for a while. They also train mixture of expert models. TPUs have a very different network bandwidth to compute ratio. They have a lot more bandwidth just objectively. And TPUs per chip tend to be a little bit less compute intensive and have a little bit less memory. You know, it's just a different design choice. So the ratio of flops to bandwidth is very different. And that means that it's much easier for Google to be able to pull off SWYX [00:37:54]: some of this stuff. JONATHAN [00:37:54]: They also have interesting, you know, Torus style network architecture or Torus style, like, literal network architecture SWYX [00:38:00]: is not like the model, JONATHAN [00:38:00]: but the network. SWYX [00:38:02]: Is this the sort of block attention? I forgot what you call it. So this is just more or the, JONATHAN [00:38:07]: yeah, this is more, not the ring attention, but these are the ring all reduces. Like you have three different dimensions of rings because they kind of put you in these three dimensional Toruses from what I understand. And so like, you know, Google's infrastructure in some sense is kind of, I wouldn't say built for this, but maybe the way that Google trains models is built for a slightly different bit of infrastructure they have. And it's kind of neat to think about that. You know, as one thing that I think NVIDIA announced for, you know, for, for both the GH200 and the GB200 is this hybrid networking where you'll have blocks of NVLink network chips. I think for the GB200, I think it's like groups of 72 GPUs will all have NVLink to each other. So higher bandwidth, then you'll have normal networking of some kind, InfiniBand or Rocky or what have you between these blocks. And that's kind of a, you know, it's a change due to the fact that, you know, it's hard to build really high bandwidth networks over very large groups, but it is now a blocked networking. And you have to think about how you architect your model and your parallelism differently. You also have to think about fault tolerance differently because it now matters where you lose a GPU, whereas it didn't before. So, you know, it's, it's, it's just all really interesting and really fun speaking personally, but it's going to mean new nightmares when we all move to that generation and have to think about, you know, new versions of these problems. JOSH [00:39:20]: As you go up to larger scales, it gets quite different. Like right now, you know, if you're experiencing, let's say, for example, you experience a GPU failure every day, that's fine. SWYX [00:39:31]: Just restart. JOSH [00:39:31]: If you make your thing 24 times as big, now it's once an hour. Now it stops being quite as easy to just restart, right? So now you have to kind of break, like bake in this sort of redundancy that you didn't have before. So I think as you go up in scale, you end up running into like a lot of really interesting problems that also inform the, the actual like design. Yeah, I mean, as an orchestration guy, SWYX [00:39:52]: this is why I always emphasize like very cheap storage or very fast storage. So you can checkpoint more, but I don't think that's probably not the best solution to for fast, you know, training. JONATHAN [00:40:05]: Which works fine when you're doing language and then you move to vision or video. And then, you know, you have multi petabyte datasets SWYX [00:40:12]: and getting, you know, JONATHAN [00:40:13]: cheap, fast multi petabyte storage starts to bite. Like I've certainly encountered issues where the literal data center where my GPUs were did not have enough, you know, object store to fit the datasets that people wanted to bring into that data center from whichever users were, were trying to bring them in. And then you get to a whole SWYX [00:40:31]: different world of hurt JONATHAN [00:40:31]: where you have to keep your data in a different region because the region is just out of storage. So things get fun really fast. SWYX [00:40:39]: Speaking of vision, Josh, actually, you know, Embu is an agents company, but you're only, you're announcing a text-only model. What, where does, where does the vision side come in? JOSH [00:40:49]: I think we've actually done a lot of work in the past and people can see kind of our blog posts about sort of self-supervised learning and some other kind of vision-related stuff in the past as well. So we're very familiar with, with that stuff. But I think our main focus right now is on kind of, as we say, coding and reasoning. And there, there's certainly a visual component to some problems. But, you know, it's not necessarily required for all problems. And actually we found that for most of the kind of like code writing and, and reasoning problems that we care about, the visual part isn't really a huge important part of it. Sometimes if you really need to, you can maybe describe SWYX [00:41:24]: the thing. JOSH [00:41:24]: There are other like, you know, multimodal models that you can use off the shelf to sort of plug in for those particular pieces SWYX [00:41:30]: that you need, right? JOSH [00:41:30]: Like if something is driving a browser or whatever, like you can sometimes get away with not having to have that baked into the original model. So our folk were, you know, in a sense, we kind of do a lot across the stack. We're working on our own infrastructure and pre-training and RL and fine tuning and products and everything. But in another sense, we're very narrowly focused on the application side. So all of the stuff across the stack is kind of going toward a very particular purpose. And so that particular purpose right now doesn't really need vision. So we think that people are going to make all sorts of really cool image models SWYX [00:42:00]: like Jonathan, right? JOSH [00:42:00]: And all sorts of interesting multimodal models into the future. We'll let them go do that. That's great. We'll take advantage of that, partner with those people in the future. And right now we're really focused on kind of the core reasoning and coding capabilities and aspects of the model. SWYX [00:42:14]: I wanted to go into carbs since that's kind of the next layer of the stack. We talked about carbs in the first episode with Kanjin because you've actually had a blog post about it like a couple of years ago. Maybe let's introduce it. JONATHAN [00:42:26]: Has that been a couple of years now? JOSH [00:42:28]: No, it must have been at least one year. Hopefully it's not multiple years. SWYX [00:42:32]: Sorry, I'm counting AI time. Yeah, yeah. Yeah, I was going to say JONATHAN [00:42:35]: you're making me feel really old right now. SWYX [00:42:39]: I count everything before the generally intelligent rename as like, you know, prehistory. Yeah. And now sort of modernity, right? So I actually thought carbs was more about hyperparameter optimization in a sense of like sort of parameters, hyperparameter search. Whereas, you know, when you introduced it, especially in this blog post, it's more about scaling laws and predictability of like, are we sort of in the right ballpark before we scale things up? Maybe sort of recount the history of carbs. JOSH [00:43:10]: Yeah, so it really is a little bit of both. So carbs is, it's maybe a backronym, but it's for cost aware Pareto region Bayesian search. So this is about technically how it works, but carbs is like, you know, we like pastries and stuff. SWYX [00:43:26]: So great, why not? But the point is that JOSH [00:43:29]: it's a cost aware hyperparameter tuner. So most hyperparameter tuners, you kind of say, OK, here's this objective function. I want you to make this number as big as possible or as small as possible, whichever direction you want to go. So yeah, just go make this number, you know, as small as possible. OK, so it'll try a bunch of different SWYX [00:43:46]: hyperparameters, JOSH [00:43:46]: a bunch of different configurations SWYX [00:43:48]: to figure out, like, JOSH [00:43:48]: how do I tweak your network and architecture, et cetera, to get the kind of best performance I possibly can. That's usually saying, like, you know, almost all of these hyperparameter configurations are, let's say they're all going to use the same number of GPUs or the same number of nodes. SWYX [00:44:01]: So it's going to run JOSH [00:44:01]: for the same amount of time. SWYX [00:44:03]: So you can do that. JOSH [00:44:03]: You can get a number out and that's great. But what carbs does is it says, SWYX [00:44:07]: OK, actually, JOSH [00:44:07]: what if we relax that constraint? What if we say each of these different points, we're going to model how expensive it will be to sample this configuration. So if what if we train with just one one hundredth of the data? Like, how well can we do? SWYX [00:44:19]: What if we train JOSH [00:44:19]: with one tenth of the data? What if we train with all the data? That way you can understand, like, as we get more and more data, as we spend more and more compute, SWYX [00:44:26]: as we make a bigger JOSH [00:44:26]: and bigger network, how does performance change with these things that change? Like how expensive it is to even explore this data point. So by doing that, we can see the scaling laws for not just, you know, SWYX [00:44:36]: the scaling laws JOSH [00:44:36]: from like the, you know, Chantilla paper, the scaling laws for all parameters. We can see how does how does the number of layers change with this? How does the, you know, the learning rate change? How do the like, you know, various types of regularization change? So you can see these nice scaling laws. And as you're going across costs, like how should this be changing as you're scaling up your model? So that, coupled with the kind of metric that we chose, which is a very precise way of measuring performance, allowed us to really like hone in on parameters that worked really well SWYX [00:45:05]: and understand, like, JOSH [00:45:05]: how do we want to scale those up, especially as we're changing SWYX [00:45:08]: things about the network? JOSH [00:45:08]: Like one of the things that we did is we used a custom tokenizer. As we change this tokenizer, changes a bunch of other things about the model. So how should we scale up this entirely new tokenizer? Like no one has ever made a model this large with this tokenizer before. And so how do we want to SWYX [00:45:22]: change all these things? JOSH [00:45:22]: Harps kind of shows you, like, look, as you change these parameters, like these other ones are kind of dependent on this. SWYX [00:45:28]: Like this is the, these are JOSH [00:45:28]: the relationships between them. So you can better understand, like, OK, if I'm going to scale this up 10x or 100x, like, where do I want to be? I can only go so far. And so, you know, we did run, like, I think maybe it was like a 14b one or something SWYX [00:45:40]: like that to check. JOSH [00:45:41]: But and so we had a bunch of like 1b or 14b and then at 70b. I don't think we had a, I think we just did like one at 14b. So you can, we get to check that like, oh, is this on the curve? Like, is this where we expect? It was like right there. So then great, go on to the next one. Yeah, I mean, that makes a lot of sense. SWYX [00:45:56]: I wonder if, so one of the key questions, and correct me if I'm wrong, but like usually people do search or do their evals just based on loss. But you actually evaluate based on, you know, the sort of end state evals that people might expect, like HellaSwag and Lombata, whatever. What is the norm here? Is there a norm? JOSH [00:46:20]: Yeah, I don't know if there's a hundred percent. SWYX [00:46:21]: I don't know. I only see loss on most people's reports. JOSH [00:46:25]: I think it's easy to, like, loss is very nice because it's very precise. It will tell you, like, very fine grained differences between like really small changes in your hyperparameters or network architecture. Whereas, especially at the smaller scales, if you're looking at like accuracy, it's very noisy. Like it might be zero or a hundred or like, you know, fluctuating by like 10 or 20 percentage points, which makes it really hard to tell, like, did that change actually mean anything? So our loss is sort of a combination of these two. Instead of saying, like, let's just look at perplexity, we say, let's look at perplexity on the tasks that we care about for multiple choice questions effectively. SWYX [00:47:00]: So we're saying like, yes, JOSH [00:47:00]: this is formulated as a multiple choice question, and we're going to look at the, like, you know, the loss of perplexity for this particular answer token. And that ends up being something that's like both targeted to what you actually care about and also very precise. The nice thing about this though is that it's independent of the data that you train on. One thing that's annoying about perplexity or about loss is that as you change your data set, this is really obnoxious because now it fundamentally changes your loss, right? And so you can't tell, like, how do I tweak my data set? But because we have this held out evaluation data set where we're looking at perplexity, we can actually change the data mix. And so CARBs actually control what is the mix of data that we want to see, like how much code, you know, how much internet text, et cetera, in order to figure out what is the best optimal mix of data and we could do that because we have this other metric. So that was one of the things that was really, really helpful. SWYX [00:47:46]: I think there is a trend overall about changing data mix as training goes on. I don't know how, you know, we're deciding not to talk about data sets in this podcast, but what have you observed about the changing data mix question? JOSH [00:48:06]: We did some experiments SWYX [00:48:08]: and we've actually talked JOSH [00:48:08]: to a bunch of researchers who are doing work here as well SWYX [00:48:11]: and looking at kind of JOSH [00:48:12]: their experiments on this. And we were originally pretty hopeful because it sounds like something that should work and make sense, right? Like, oh, cool. Like maybe you would have your model, like learn the basic features SWYX [00:48:22]: and then over time, JOSH [00:48:22]: it could get really good at these complicated math problems or coding or something, right? But it just turns out that like, it's just not the way it works. Like we've done so many experiments and you can get like a tiny, tiny little boost from this, but it just is not like, it's just not the important thing, at least in the experiments that we've seen. So yeah, we've kind of, we're letting other people SWYX [00:48:40]: explore that more JOSH [00:48:40]: if they want, but that just doesn't seem like the most promising direction for us. JONATHAN [00:48:44]: We've had some surprisingly good luck with this. We just released a paper on it. The details matter a lot and it really matters what you're trying to do with the model. SWYX [00:48:53]: Yeah. JONATHAN [00:48:53]: But it's been quite effective for us depending on the setting. And certainly when we're thinking about domain-specific models, this helps a ton. You know, to a certain extent, you can always think of this as like early fine tuning. But yeah, I like, there've been little glimmers of this in the literature for years. Like especially, I think the Gemini 1.5 paper mentions this. And I don't remember whether the Llama 3 paper mentions this, SWYX [00:49:15]: but it's kind of, JONATHAN [00:49:16]: it's one of those, like people have different ways to get to these endpoints. SWYX [00:49:20]: I think, you know, JONATHAN [00:49:20]: there are the architectural tricks that each lab has to mitigate loss spikes or what have you. And everybody's got, you know, their own bag of tricks and it leads to kind of sometimes this contradictory information. It's not contradictory. People are just kind of exploring SWYX [00:49:33]: different parts of the space JONATHAN [00:49:33]: in some sense. And there are lots of ways to get a great model. But certainly for us within our config, and it seems like, I guess for the folks at Google, within kind of the part of the world they live in, changing the dataset has helped, but the details matter a lot. And it's really hard to get those details right for the reasons Josh, SWYX [00:49:48]: you know, just mentioned. JONATHAN [00:49:48]: Like there's a lot of search involved and you essentially have to make hard choices about SWYX [00:49:52]: what parts of the space JONATHAN [00:49:52]: you're going to search and which ones you're going to leave be. And so, you know, some people have done an amazing job. Like I think the, who is it? The Deep Seek folks have done an awesome job looking at like batch size warmup. And that's been really, really fruitful for them. You know, other people are looking really hard at things like data mix, but it just gets tricky to look at everything. JOSH [00:50:09]: Yeah, I think we've found that like we could get some things that looked like gains from datasets. But one of the things that I like about carbs is that when we applied carbs to like properly tune things, then a lot of those kind of evaporated. Whereas like, like if we just tune these other parameters, actually we can get almost the same gains without having to do this more complicated thing. So at least in the experiment and in the settings that we've, like in the particular metrics SWYX [00:50:34]: that we care about, JOSH [00:50:34]: we haven't seen these kind of like pan out or scale up in quite the same way. But not to rule it out. And I think you're right, Jonathan, SWYX [00:50:41]: that there probably are JOSH [00:50:41]: a lot of like details that go into like exactly what is the metric, exactly what is the dataset, exactly which, like what schedule are we using for this. And I certainly wouldn't rule it out working. SWYX [00:50:52]: Quick question about emergence. Doesn't emergence throw a spanner into a theory of carbs? Ah, so there is a paper JOSH [00:51:01]: of which I really liked and I think informed SWYX [00:51:05]: a little bit of how JOSH [00:51:05]: we thought about this, which is are emergent properties of language models a mirage? And I think if you look at that paper, it actually makes a relatively compelling case that in fact, you know, this emergent behavior that you're seeing is not really emergent behavior, but is really a function of the evaluation metrics that we're using. So if you look at accuracy as a metric, what's happening is that accuracy is actually going up continually over training, but it's in log scale. So it starts out at 0.001%, 0.1, 0.1, 10. SWYX [00:51:35]: Only when you're going JOSH [00:51:35]: between 10 and 90 do you see this happen, right? When you go from one in, you know, SWYX [00:51:40]: a thousand getting right JOSH [00:51:40]: to one in a thousand getting wrong, like there's many orders of magnitude happening here. SWYX [00:51:44]: So when you're looking JOSH [00:51:44]: at this in perplexity, then you just see this nice straight line. And so that's actually what carbs is exploiting. Like since we're, since our metric is in this kind of like perplexity log space, like you can see like, oh, it's just like getting better as you make it bigger in this nice, very predictable way. So that, and that is exactly what we saw. Like these things were really, really bad at, you know, predicting the multiple choice answer, just always guess A. OK, it's so terrible at it, but it was like learning to be less confident about that. SWYX [00:52:09]: Yeah. One trick I saw from one of the papers recently was just like, just randomize the order of the multiple choice questions. And if you, if, if, if they, if they over, if that hits the performance a lot, then they're just basically memorizing the test set, which makes a lot of sense. JONATHAN [00:52:28]: Yeah, this is, I, I mean, you know, I, I completely agree with what Josh said. SWYX [00:52:32]: I think the, you know, JONATHAN [00:52:32]: my bigger lesson is that anything can look however you want it to look. If you put it on a log scale to a certain extent and log, we love our log scales and deep learning for various reasons. Everything looks very clean on a log scale until everything looks very flat on a log scale. Um, I don't know. I like log scales always mix me up. That's, that's all I can say. SWYX [00:52:51]: Great. I think the, the last thing I was, I was going to mention on, uh, carbs. Oh, well, I mean, let's, let's just kind of go right into evals because I think that's going to be, uh, the, the sort of crowd favorite. Um, so carbs, we already mentioned, um, you know, leans heavily on, uh, the sort of end evals that we would typically eval LLMs on, except that you had to make your own. Um, there are a lot of documented problems with many of the common evals out there and you fixed all of them. It sounds like, I don't know JOSH [00:53:18]: about fixed all of them, but, uh, I think in the same way that we like to dig into the infrastructure and hardware and understand, like what actually is going SWYX [00:53:27]: wrong? JOSH [00:53:27]: Like what is the actual error on this machine with this GPU? SWYX [00:53:31]: And why did that happen? JOSH [00:53:31]: And how do we fix it? We take the same approach to the evaluations. So when we looked at the evaluations and actually looked at the data sets, you know, what we did is SWYX [00:53:39]: like, okay, if we're going JOSH [00:53:39]: to be, you know, evaluating natural language, understanding and reasoning, like, let's look at all the data sets that are out there. Let's actually look at a bunch of the examples and say, like, is this a good data set that we should use for evaluation? That's kind of how we selected the evaluation data set that we had. Uh, and then when we looked at the actual examples in there, we noticed like a lot of these are very messy. Like some of them messy SWYX [00:54:00]: to the point of like JOSH [00:54:00]: incoherence and some of the ones that we didn't choose. Uh, but even the ones that we chose, like people tried pretty hard on SWYX [00:54:06]: these data sets. JOSH [00:54:06]: They did try and clean them, but there's just a lot of data points in there and it's just easy to SWYX [00:54:10]: make mistakes. JOSH [00:54:10]: Right. And so, you know, it's not that they have a SWYX [00:54:13]: hundred people looking JOSH [00:54:13]: at every question, like that's just way too SWYX [00:54:15]: expensive. JOSH [00:54:15]: So you end up with questions that just don't make sense. SWYX [00:54:18]: Somebody didn't really JOSH [00:54:18]: see this. Somebody just clicked the wrong box for the answer. Uh, or the question makes sense in your head. When you write it, we've often seen this, it's not even like malice or SWYX [00:54:26]: incompetence. JOSH [00:54:26]: It's really just like, you know, you write this, SWYX [00:54:28]: you're ready. JOSH [00:54:28]: You're like, this makes SWYX [00:54:29]: sense to me. JOSH [00:54:29]: You show it to another person like that makes SWYX [00:54:31]: sense. JOSH [00:54:31]: You show it to a third SWYX [00:54:32]: person. JOSH [00:54:32]: They're like, this makes no sense at all. SWYX [00:54:34]: That's because you're JOSH [00:54:34]: kind of, you know, using a different meaning of SWYX [00:54:36]: the word. JOSH [00:54:36]: And then when they say that, you're like, Oh, SWYX [00:54:38]: wow, you're right. JOSH [00:54:38]: That is actually really confusing. It's easy for things to SWYX [00:54:41]: kind of make sense in JOSH [00:54:41]: our own head. So what we did for the evaluations is really dug into the details of each of these data sets and tried to ask, like, what makes a good SWYX [00:54:50]: question? JOSH [00:54:50]: What makes a good answer? SWYX [00:54:52]: Like, what does it mean JOSH [00:54:52]: for it to be ambiguous? We had a whole, like, SWYX [00:54:55]: we looked at lots of JOSH [00:54:55]: data, broke this down, asked lots of people SWYX [00:54:58]: about all these JOSH [00:54:58]: different questions to build a model of this and help us kind of clean these data sets. That was sort of one big piece of it. A second big piece was making sure that our data that we're training on is not data that we're testing on. So there we kind of took a step back and said, like, OK, well, let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves. And just make sure that this data is definitely not in the, you know, the training set. So we did that. And then we're able to, like, now be confident about, like, our performance of our model and also performance of other open source and other closed source models. Yeah, there's a lot there. SWYX [00:55:33]: You had 11? I don't know how many data sets. I think so. One, two? Yeah. Any one you want to call out in particular to dive deeper on? Some of these are very famous, like HelloSwag, MitoGrand. Some are less famous, like Race. I don't know if... Race is a great data set. JOSH [00:55:50]: See that one? SWYX [00:55:51]: Yeah. Yeah. Just, you know, anything that's interesting you want on specific data sets? I think there are JOSH [00:55:57]: a few asterisks in there. You know, definitely read the whole paper SWYX [00:56:02]: as you're looking at JOSH [00:56:02]: some of these, like the GSM8K one is a little bit weird. I think one that was SWYX [00:56:06]: kind of funny, JOSH [00:56:06]: it was, like, low performance on ethics from some of the more recent models. I think that was a SWYX [00:56:11]: little bit funny JOSH [00:56:11]: because the models, you know, SWYX [00:56:13]: I think there was JOSH [00:56:13]: a reaction to, like, oh, no, like, you know, SWYX [00:56:16]: the models are saying JOSH [00:56:16]: bad things. SWYX [00:56:17]: And so they went way, JOSH [00:56:17]: way in the other direction. And now, like, on the ethics data set, SWYX [00:56:20]: it's always like, JOSH [00:56:20]: this is totally unethical, even though it's really fine. So they've just been tuned to, you know, make sure they don't make any PR disasters. SWYX [00:56:28]: I thought that was JOSH [00:56:28]: a little bit funny. Not to say that it's necessarily like a flaw of the model, but just kind of like, you know, political or tuning opinion. I think the main takeaway, I was just going to say SWYX [00:56:38]: the main takeaway JOSH [00:56:38]: for many of the, like, actual performance is, like, once you fix these ambiguous examples, a lot of these benchmarks are really saturated. Like, I think it's SWYX [00:56:48]: important to look at, JOSH [00:56:48]: like, you know, SWYX [00:56:50]: like when you're JOSH [00:56:50]: talking about performance on ANLI or race or pool queue or something, what you're really talking about is, like, performance on questions that make no sense. Like, it's just like, did it guess the answer in this, like, really weird scenario? Like, those are the ones that are left. SWYX [00:57:03]: Like, when you look JOSH [00:57:03]: at the performance on the ones that actually make sense to everyone, all the models agree. SWYX [00:57:07]: We agree, like, JOSH [00:57:07]: everyone's on the same page, which I think is kind of a really interesting result. SWYX [00:57:11]: The question then becomes, you know, what are the new, like, set of evals that would be like the next frontier that often embeds with it your idea of what reasoning is, because it's obviously you're super interested in reasoning. And yeah, I mean, like, where does this, where does the state of evals go from here? JOSH [00:57:30]: This work and this blog post is talking mostly about the public evaluations SWYX [00:57:34]: and the things JOSH [00:57:34]: that we can release. We do have our own internal evaluations. For example, one of them that we are releasing is the code understanding evaluation, which is about predicting, SWYX [00:57:44]: you know, JOSH [00:57:44]: what will this variable be or asking questions about code, et cetera. And that is one of the early benchmarks that we made that we can release. We can partly release it because we can generate an almost infinite amount of this data because these are programmatically generated. And so, you know, we're not really worried about there being like corruption in the kind of the training or test sets. So that makes it a little SWYX [00:58:03]: bit easier for us. JOSH [00:58:04]: But I think it's, you know, we have built other data sets as well that we can't release. Some of them, you know, SWYX [00:58:09]: for example, JOSH [00:58:09]: because they maybe use other open source code and so we can't redistribute it necessarily. Other ones, because, you know, that's, I think evaluations and data are like a core, important part of, you know, the business. And I think we take evaluations very seriously and are spending a lot of effort in terms of like, what exactly do we make as part of the evaluation set? How do you evaluate these things? We've done a lot of other stuff, you know, since these evaluations. But I think a lot around like code understanding for us, since that's our main focus. And it's a nice place to explore reasoning as well. SWYX [00:58:40]: It sounds like you talk a little bit about like code understanding as like sort of variable level, like sort of very micro context. Is there a sense of like larger code context as well? I don't know what I mean by that, by the way. It's mostly just like if I told the senior engineer to go look at a code base, they would understand at a broad level, the architecture, but also the design decisions and be able to tell me that. I don't know if that's useful or not, but I mean, that's useful to me as a, as someone who might be working with them. Yeah. JOSH [00:59:06]: This particular dataset is like the more low level code understanding, SWYX [00:59:10]: like just literally JOSH [00:59:10]: what happens in this code. And this is mostly because, you know, SWYX [00:59:13]: this is part of the JOSH [00:59:13]: carbs tuning metric, etc. SWYX [00:59:15]: Like we care about JOSH [00:59:15]: the low scale version SWYX [00:59:17]: of this as well. JOSH [00:59:17]: We want smaller scale models to be able to do something on this. And so that's kind of the focus for this. SWYX [00:59:22]: And hopefully this is more JOSH [00:59:22]: useful for other people. But yes, SWYX [00:59:25]: those other questions JOSH [00:59:25]: are also quite interesting. They get a lot harder to evaluate, like, is this a good architecture or not? Like you and I could probably debate for a while on, you know, different architectures. And so it becomes a lot trickier to do these evaluations as they become more realistic. So I think that's one of the things that we've been playing around with a lot, especially around like code generation. SWYX [00:59:44]: So if you're saying, JOSH [00:59:44]: you know, implement this function, okay, it can be kind of objective, but, you know, even MBPP, we've made our own internal version of this data set, right? SWYX [00:59:52]: Where we've taken like JOSH [00:59:52]: every single example SWYX [00:59:54]: and looked at it and been like, JOSH [00:59:54]: does this actually make sense? Like, what is the type signature? Like, can we remove all ambiguity, et cetera? SWYX [01:00:00]: So you basically like reviewed every single question on, I mean, that's impossible for like HelloSwag, right? Yeah, yeah. JOSH [01:00:05]: We didn't do that for HelloSwag, but this is for MBPP, which is only like a few hundred. So we just sat down and did it. Yeah. JONATHAN [01:00:12]: I'm so excited to get to look at this data set. Like this is such a resource for the community. I absolutely can't wait. We should probably do the, JOSH [01:00:19]: I don't know. I don't know if we were planning on doing the healed MBPP one, SWYX [01:00:23]: but hopefully we can do JOSH [01:00:23]: that one in the future. Did you look at SweetBench? SWYX [01:00:26]: It's the sort of hot new data set of the summer. JOSH [01:00:28]: Yeah, I've taken a quick look SWYX [01:00:29]: at SweetBench. JOSH [01:00:29]: It's really interesting. I like that it's a much more difficult kind of coding, code related task for bug fixing. I think it gets into some of these problems where it is a lot harder to evaluate these things once they get more realistic. Like we were looking at the AgentBench paper, I think just last week for our paper club and one of the things SWYX [01:00:49]: that we noticed JOSH [01:00:49]: is that actually like both of the examples in the appendix that are given as like traces where it got it right. This is actually not the right solution. And it's OK. You know, it's fine. Like it did make it past the test. That's what the metric is. SWYX [01:01:02]: That's what the benchmark JOSH [01:01:02]: is about, right? But like it just said, SWYX [01:01:05]: you know, like, JOSH [01:01:05]: you know, dot encode ASCII. Like, well, that's not the right way to do this. Like it just dropped all the other edge cases that you actually would have cared about in production for this thing. SWYX [01:01:14]: And there is like JOSH [01:01:14]: a better way of doing it. SWYX [01:01:16]: And you know, JOSH [01:01:16]: that's what the real golden patch was. But, you know, that's OK. But then how do you test all of that? SWYX [01:01:21]: Like as you start to do JOSH [01:01:21]: more realistic things, the test coverage, like getting test coverage over all possible ways of solving these bugs is really hard. Evaluation is the single JONATHAN [01:01:28]: hardest part of the whole thing. Like I spend a shocking amount of time just telling our customers SWYX [01:01:34]: we need to find a way JONATHAN [01:01:34]: to measure what you actually want out of the model before you should ever touch a GPU. And, you know, trying to convince my team and me to follow our own advice a lot of the time on that. And I think everybody like on the one hand, SWYX [01:01:46]: it's easy to laugh JONATHAN [01:01:46]: at the state of the evaluations that we have. None of them are good. Like if you go read these eval benchmarks, you'll always come away SWYX [01:01:52]: disappointed. JONATHAN [01:01:53]: And yet they've given us useful hills to climb. And we do seem to be making progress and measuring SWYX [01:01:58]: progress in the field. JONATHAN [01:01:58]: And I think anecdotally, models are getting better year to year. So I feel like people tend to go and get into one situation or the other, like evals don't matter. I'm just going to look at loss SWYX [01:02:07]: or like, you know, JONATHAN [01:02:08]: the evals matter a lot and they're all broken. So what do I do? And I think like a lot of things in deep learning, we have to make peace with just complete imperfection. Like the most successful scientists I see are the ones who are OK operating in a world SWYX [01:02:20]: where everything's JONATHAN [01:02:20]: going to be broken. SWYX [01:02:22]: And yet we can still JONATHAN [01:02:22]: cobble things together and make something SWYX [01:02:24]: interesting happen. JONATHAN [01:02:24]: I mean, we were just discussing that with literal infrastructure. And now we're all the way SWYX [01:02:28]: up to like, JONATHAN [01:02:28]: how do we measure whether a model performed a complex coding task correctly? And everything is broken. SWYX [01:02:34]: And yet we're still able JONATHAN [01:02:34]: to make huge amounts of forward progress. SWYX [01:02:36]: I think that's right, Jonathan. JOSH [01:02:38]: And that the challenge SWYX [01:02:40]: isn't necessarily JOSH [01:02:40]: making perfect evaluations. I think our blog post here is about going really into the weeds on these to figure out like, what does that look like? And I think one thing is like, you know, SWYX [01:02:49]: as you said, JOSH [01:02:49]: we have been able to make a lot of progress without making these perfect. SWYX [01:02:52]: That's great. JOSH [01:02:52]: You don't have to have perfect evaluations. And, you know, the more interesting work is the stuff that we can't necessarily publish about, which is the imperfect evaluations that we have for actual coding tasks, for example. SWYX [01:03:04]: Like, what does this JOSH [01:03:04]: really mean as a person? And there, as you said, it's much messier. SWYX [01:03:08]: So it's a lot harder JOSH [01:03:08]: to put it out and say like, hey, everybody use this because there's so many SWYX [01:03:12]: rough edges. JOSH [01:03:12]: It's so hard to like even say, oh, is this even the right task? Is this even the right way to do it? And there's a lot of judgment. SWYX [01:03:19]: There's a lot of intuition JOSH [01:03:19]: that it comes down to. But yeah, I think that's where it's critical to do SWYX [01:03:23]: if you actually want to JOSH [01:03:23]: make these systems work. JONATHAN [01:03:24]: Yeah, you have to make peace with with living in that in between. SWYX [01:03:28]: Yeah. JONATHAN [01:03:28]: And I think that in some sense, SWYX [01:03:30]: when I hire researchers, JONATHAN [01:03:30]: that's the number one quality I look for. Like, can they be at peace living in a house that is neither clean nor messy, SWYX [01:03:36]: but it's just kind of JONATHAN [01:03:36]: somewhere in between? And are they OK with that? Are they OK with a few dishes being out on the table and a few clothes SWYX [01:03:42]: being on the floor? JONATHAN [01:03:43]: Or will that drive them insane? Or will they just end up with all the clothes on the floor and like all the dishes out all the time? Like, it's kind of I'm looking for that perfect balance because, you know, we have to operate in this imperfect world. Like, yeah, go ahead and give me the perfect evaluation for programmers SWYX [01:03:58]: or for an LLM JONATHAN [01:03:58]: that is a program assistant tool. Like there is no perfect evaluation. But clearly we've made progress. And so the most important part SWYX [01:04:06]: is just are we JONATHAN [01:04:06]: climbing the right hills? And so this is why I'm so excited to see the ambiguity aspect of this. We often think we have more room to climb on these benchmarks. It turns out we don't. Or it turns out that actually we're climbing, getting good at the benchmark and not actually getting good at the task we care about underlying the benchmark anymore. SWYX [01:04:21]: Maybe the model, JONATHAN [01:04:21]: like this is the famous example where if you get 100% at MNIST, your model must be broken in some way because there are four examples mislabeled, you know, it's it's that all over again. Welcome to this. SWYX [01:04:33]: Yeah, it's the accidental canary canary in this. I think one thing that's JOSH [01:04:37]: actually really interesting about this also is that, yes, like the ambiguous examples are sort of, you know, not that great from the perspective of these particular tasks that we're evaluating. SWYX [01:04:46]: But actually, one thing JOSH [01:04:46]: that we're very interested in is ambiguity itself. Like, can we detect whether a task from a user is ambiguous or whether you've, you know, completed a task successfully? Like these are actually hard, messy problems, but are really important from like the user experience of using these models. I would much rather have a coding agent that will give me back a thing. And, you know, it's it's actually the code doesn't work like 10% less of the time than some other model, but it will tell me 100% of the time like when it's not sure. Like that's so much more useful if it can communicate like, I'm not really sure about this or maybe there's some errors here. Then just like, here's some code. I have no idea if it works. And so these kind of like, you know, detecting ambiguity and detecting correctness SWYX [01:05:25]: or uncertainty, JOSH [01:05:25]: I think are really interesting problems SWYX [01:05:27]: that we're really like JOSH [01:05:27]: digging into quite deeply. SWYX [01:05:29]: I want to touch on maybe a couple of hot topics in evals, maybe tangentially related, but we're on the evals train right now. So I'm just going to get on that. So ArcAGI, Francois Chollet's hot new thing, it's sort of my take on it is basically it's trying to measure reasoning through an abstract IQ test. Effectively, I noticed that you don't use it. There's a lot of community debate, pro and con about it. What are your thoughts on just more abstract reasoning and maybe ArcAGI specifically? JOSH [01:06:01]: I think we purposely stayed away from the very, like there's BigBench, for example, that has a lot of, I think, to me, feels sort of similar types of tasks that are like very unrealistic. Like, oh, you know, we have books of different colors and then you're going to shuffle them and like which book is furthest to the left or something like, OK, cool, I guess it's neat. It's neat, I think, for us to explore in terms of like an agent reasoning in a larger loop. And we do care about these types of evaluations there. The types of evaluations we're talking about in the blog post here are for getting at, like, does this model in a base model sense, is this working at all? There's no chain of thought in these evaluations. These are just like, go straight to the answer. Does this make sense? SWYX [01:06:42]: Like, is this a thing that JOSH [01:06:42]: you can answer very quickly? That's what we were selecting for with these evaluations. This is not to say that these are the only evaluations we have. I think the Arc ones are like a little bit too, probably, visual for us to really be able to integrate with. SWYX [01:06:56]: But I think some of the JOSH [01:06:56]: BigBench ones are... You can tokenize it. SWYX [01:06:59]: Yeah, but, you know, JOSH [01:07:00]: I think it's not really... I think you can spend a lot of time getting really good at these kinds of benchmarks without making, like, kind of more general purpose progress. And so I think we're a little bit leery of going too far in that direction. Similarly, like, coding competitions. Like, we do a lot of code generation, but we don't really do a lot on, like, code competition problems for the very, very hard ones. SWYX [01:07:20]: So I think you can go JOSH [01:07:20]: very far down that route SWYX [01:07:22]: and make something that's, like, JOSH [01:07:22]: really good at those problems, but not actually that useful as, like, a programmer day to day. SWYX [01:07:26]: Yeah. JONATHAN [01:07:27]: Take a different tactic, which is, like, at the end of the day at Databricks, I have 12,000 customers, or I think that's the latest number, all of whom are trying to do something with, you know, LLMs or AI or machine learning. And those things don't look like these tasks. I don't think I have a single customer that's asking to, you know, have AI solve abstract reasoning problems. Things are pretty, like, they can be ambiguous, SWYX [01:07:53]: they can be challenging, JONATHAN [01:07:53]: they can be really interesting, SWYX [01:07:55]: but none of them look quite like this. JONATHAN [01:07:56]: And so, you know, I think to Josh's point, like, it's really about asking, why are we doing this? Even if you're trying to build AGI, and that's not personally my purpose, and I, you know, Josh has much more interesting things to say about that than I do. I don't even know if this is the kind of intelligence I would get excited about or care about personally, or if I would consider, you know, to Josh's point, this to be the indicia of intelligence. SWYX [01:08:17]: It's neat. JONATHAN [01:08:17]: But, you know, for me, it's, like, more down to earth things, like having a model that can have a conversation with you about data SWYX [01:08:24]: that on the backend JONATHAN [01:08:24]: is running SQL queries on your literal data. That's a much more interesting task to me. That's something that really matters day to day for my customers and, you know, different perspectives, but, you know, I think Josh and I would probably say the same thing, SWYX [01:08:36]: even though I would, JONATHAN [01:08:36]: I'm guessing, I don't want to put words in your mouth. You would say that you're pursuing more general intelligence in your own way. And I would say that I'm very happy with narrow intelligence. Like, I'm very happy with my little SQL bot and building 12,000 of those because they're moving the needle for a lot of folks every day. JOSH [01:08:51]: Yeah, I think we're, you know, we're not as far away in our position as it might seem. I think we're also excited about, like, SWYX [01:08:58]: how do you actually JOSH [01:08:58]: make these things useful? And that does end up being pretty narrow. I think these other tasks can be interesting as, like, ways to explore these more abstract reasoning questions or like, OK, how could an agent actually work through this? But it's important to keep in mind that it's like a toy, not a real problem. It's like it's a scientific tool to tell us something about the models. SWYX [01:09:16]: It's not something we should JOSH [01:09:16]: be optimizing for necessarily. SWYX [01:09:18]: The one thing I'll point out is, you know, as a kid, I was graded into a gifted program based on my ability to solve these exact type of problems. And then I entered college based on my ability to solve SATs, which, again, have nothing to do with my college experience, but whatever. So, you know, we have a history in the humanity of doing correlated IQ tests to general capability. OK, so the two more, two more viral evals, and then, you know, I just want to be mindful of your time. Needle in a haystack, long context utilization. Oh, for the love of God. Something, well, OK, like, let's just assume that, you know, on our podcast, we've discussed the, you know, baseline problems with needle in a haystack, but just generally long context, right? It's a useful thing for agents. I assume. And it's something that, you know, it's out there. Like, we don't know, don't really know what the best way to utilize memory is. But like, I assume it's important, right? What I'll say is like, you know, JONATHAN [01:10:13]: I spend a lot of time thinking about RAG these days. And RAG, you know, in one sense, you know, the way that I think about RAG is it's the world's simplest agent. It is an agent that basically, you know, there's at least more than one thing happening in the process of building models, at least a system. If you give the model the ability to decide when it wants to retrieve data from a context or retrieve data from a database, then we're talking about an agent. So RAG kind of, I think, like toes that boundary really nicely. There are a lot of reasons why you do genuinely need a long context. Like, I don't think long contexts are problematic in and of themselves. I know there's some controversy even about that. I love the idea of doing like thousand shot tasks as an alternative to fine tuning. I love the idea of pulling in lots of data into the context. I love the idea of once you get in a multimodal land, you're just going to end up SWYX [01:10:54]: with giant context. JONATHAN [01:10:54]: It's kind of unavoidable. The flip side is I don't know of anyone who like is hiding a secret passphrase in a book and needs the model to find it. Needle in a haystack is, it's interesting. The challenge with long context to my mind, and Josh, SWYX [01:11:08]: I'm curious what you think, JONATHAN [01:11:08]: is simply that annotating long context evals is really hard and really expensive, you know, intrinsically, because you need someone to read 10,000 tokens or 100,000 tokens, or like you need someone to read a 1,000 page book or the equivalent thereof in order to measure those long context benchmarks. I don't know if a human could solve these tasks, let alone that a human could do this in any amount of time where you're willing to pay the money to get the data annotated. And so any long context eval SWYX [01:11:33]: has to, in some sense, JONATHAN [01:11:33]: be correct by construction. And you have to, you know, the, you have to know the answer before you've created the example. And needle in a haystack is kind of the simplest way SWYX [01:11:41]: of doing that. JONATHAN [01:11:41]: I think the problems of needle in a haystack are well known, you know, it doesn't measure anything real. You're not even testing the model's ability to holistically use the context just to identify one part of the context. So you can do some wacky things to your model, like quantize the hell out of the KV cache and still get needle in a haystack to work quite well because it's not trying to holistically take advantage SWYX [01:11:59]: of things. JONATHAN [01:12:00]: You know, I have some thoughts on things that I like more that are also still correct by construction. Like, I really like the idea of doing thousand shot tasks where you can look at the scaling as you go from 10 shot to 100 shot to thousand shot to fine tuning on that data instead. And I like that as a way to, you know, have something that's correct by construction, or at least where you have SWYX [01:12:19]: a nice baseline JONATHAN [01:12:19]: that you can compare to automatically. So I'm typically looking for like contexts that are situations where long context is one way to solve the task, but not the only way SWYX [01:12:28]: to solve the task. JONATHAN [01:12:28]: And we have some other strong baseline floating around personally. But yeah, needle in a haystack, not my favorite thing in the world, to say the least. JOSH [01:12:35]: Yeah, I mean, I agree with most of what Jonathan SWYX [01:12:38]: said, I think. JOSH [01:12:38]: I think one other thing that I will call out SWYX [01:12:40]: is that, you know, JOSH [01:12:40]: from like a coding application perspective, it's useful to have long context because the lazy thing of just like throw the whole repo in the context is like, SWYX [01:12:48]: OK, cool. JOSH [01:12:48]: Like, you know, you can just get started with that. But then in, you know, in real scenarios, you don't necessarily want to put the whole thing in there. You can have code bases SWYX [01:12:56]: that are bigger. JOSH [01:12:56]: You probably want to filter down to the stuff that's relevant anyway to not be confusing. Like you probably even if you did have a lot of context, SWYX [01:13:02]: you might want to sort it JOSH [01:13:02]: in some way to say this is more important than this other stuff. So and, you know, you don't want to wait for you don't want to be wasting all this time and compute SWYX [01:13:09]: on inference and like JOSH [01:13:09]: doesn't really matter. So, yeah, I don't know that it's the most important thing. SWYX [01:13:15]: I think people will find creative use cases. And like Jon said, I think the multimodality examples will naturally lend themselves to long context. Cool. And then one last one on just general sort of agent related capabilities that we didn't really talk about in the eval section is function calling and tool use. There's a recent trend, I think, basically led again by OpenAI on parallel function calling. There's always there's been a limit on how many tools you can call from four to now, I think, 128. And I think theoretically, Claude and Jem and I support a lot more. JOSH [01:13:49]: So just generally, SWYX [01:13:50]: how do you think about evaling tool use? Is that super important for you guys? We're thinking about it JOSH [01:13:55]: in a slightly different way, which is, yes, you can have this like hard coded list of tools. But if only you could have like this really large open set of like tools, maybe they would be like functions that you could call if only there was like a language or like a programming thing, like being able to write code. I think for us, it's like, well, look, if we can write code, like now you have all these tools accessible at the end of the day, SWYX [01:14:16]: like function calling JOSH [01:14:16]: is just a function invocation, like literally in code. I think our approach to this is like SWYX [01:14:21]: instead of worrying about JOSH [01:14:21]: like weird hard coded agents using tools, like let's just make them SWYX [01:14:25]: able to actually JOSH [01:14:25]: write code robustly and make that code work and be able to debug that code, know if that code is safe to run, like get really good at the like code writing and execution part of things, because that will open up the action space like far more than, you know, 128 tools, like just everything is at your fingertips, especially I think over the next few years, like we already have so many really good APIs. As we get better and better at writing code, we'll be able to make APIs to things that don't even have APIs today. That's kind of how we think about it is less as like a special purpose thing SWYX [01:14:52]: and more as like JOSH [01:14:52]: this is one of the reasons to focus on code. SWYX [01:14:55]: On my end, JONATHAN [01:14:55]: the way that I think about this is, you know, I think a lot about how models interact with data. SWYX [01:15:00]: And so for me, JONATHAN [01:15:00]: tool use is really a question of how do you take models SWYX [01:15:04]: that are really built JONATHAN [01:15:04]: for unstructured data SWYX [01:15:06]: and have them interact JONATHAN [01:15:06]: with structured data? So, you know, and I get the question a lot from my customers, SWYX [01:15:10]: like what do I do JONATHAN [01:15:10]: with tabular data? Or what do I do with like, you know, JSON? Or what do I do? I mean, you name it, like even what do I do SWYX [01:15:17]: with a PDF? JONATHAN [01:15:17]: Because PDF parsing is still an unsolved problem, even in 2024. And the answer, or even just the basic question SWYX [01:15:24]: of like, should I bother JONATHAN [01:15:24]: to structure my data anymore? Shouldn't I just toss the table? Shouldn't I flatten it SWYX [01:15:28]: and just throw it JONATHAN [01:15:28]: into the LLM context and like let the model SWYX [01:15:30]: figure it out? JONATHAN [01:15:30]: Answer is no. We've built all these fun APIs and fun languages SWYX [01:15:36]: and paradigms JONATHAN [01:15:36]: for dealing with structured data over the years. Just use them. SWYX [01:15:40]: Have your model use them. JONATHAN [01:15:40]: Train a model that can interact SWYX [01:15:42]: with these things JONATHAN [01:15:42]: in a meaningful way. Like text to SQL SWYX [01:15:45]: is still, JONATHAN [01:15:45]: or like having a model be able to make SQL calls in the backend is actually like one of the single SWYX [01:15:51]: most useful things JONATHAN [01:15:51]: for my customers. It sounds really boring. Models are really good at it. And it moves the needle day to day. SWYX [01:15:57]: So tool use for me JONATHAN [01:15:58]: really is that like, how do you just interact with structured data sources and take advantage of the fact that you have some SWYX [01:16:05]: prior knowledge JONATHAN [01:16:05]: about the structure of your data that an LLM would completely flatten away. In many ways, this is kind of one of the, one of my biggest frustrations with the fact that LLMs work well with code. We have decades and decades and decades SWYX [01:16:17]: of understanding JONATHAN [01:16:17]: about the structure and interpretation of programs. Like I think that's literally the name of a book on programming, if I remember right. And, you know, we have all this theory. We know everything there is to know about programming languages if they're well-formed languages and have the right properties. And yet when we have an LLM SWYX [01:16:31]: work with them, JONATHAN [01:16:31]: we literally just turn it into a token stream. SWYX [01:16:33]: Despite the fact that we know JONATHAN [01:16:34]: how to parse it. We know, you know, how to do all sorts of, you know, SWYX [01:16:38]: reference, you know, JONATHAN [01:16:38]: disambiguation and things like that. We're still just flattening it into a model and making the model relearn all of these things from scratch. And it frustrates SWYX [01:16:45]: the hell out of me. JONATHAN [01:16:45]: I don't have a better answer when it comes to code, but I really appreciate that with a lot of data sources that have structure to them. Tool uses and function calling SWYX [01:16:53]: are just, JONATHAN [01:16:53]: in my mind, SWYX [01:16:55]: So I think basically what you're saying is like code is the God tool for Jonathan. Like, you know, SQL is so much the right abstraction for accessing all this data. One thing I do spend a lot of time thinking about is for the stuff that doesn't fit in a SQL table, you know, is knowledge graphs the answer? I think a lot of people are exploring that and I think every now and then people get a bout of knowledge graph religion and then it kind of doesn't work out. So I wonder, I wonder what the end state is. Like, is this an idea where it's a mirage? Or is this the idea where it sometime is going to work? It's about having the right tools JOSH [01:17:27]: for the problems, right? Like as Jonathan was saying, SQL is sometimes definitely the right tool. Like you've got your, you know, order table or something and you want to know, you know, number of sales last month. Like you should be using SQL sum that column. OK, great. You're all set. Knowledge graphs also, SWYX [01:17:40]: you know, JOSH [01:17:40]: are sometimes the right tool for a particular problem. You have some like weird question about relationships between entities SWYX [01:17:46]: that are modeled JOSH [01:17:46]: on some particular ontology that you actually understand and it's like math to the real world. Great. Use a knowledge base. Like use a knowledge graph. This is fine. But I think in the real world, it gets a lot messier than like knowledge graph style of things where it's like, well, is there a relationship between these two nodes? Like, I don't know. SWYX [01:18:04]: Like, is are these JOSH [01:18:04]: two separate nodes? Like those kind of messy borders, I think, prevent it SWYX [01:18:08]: from being a tool JOSH [01:18:08]: that can like solve everything forever. And so I think it'll always be good for certain problems, just like SQL is good SWYX [01:18:14]: for certain problems. JOSH [01:18:14]: Like different abstractions are good for different problems. And yeah, I think this is why I'm excited about code. Like code lets you SWYX [01:18:20]: kind of pick the right, JOSH [01:18:20]: like let's use this library for this problem. SWYX [01:18:22]: Let's use this library JOSH [01:18:22]: for this other problem. JONATHAN [01:18:24]: I think Josh said it and you said it well, like code is kind of the God tool. It unlocks literally everything. The challenge for me is always like, SWYX [01:18:31]: you know, sometimes JONATHAN [01:18:31]: unlocking too much power can sometimes inconvenient things can happen. And so it's all about balancing that SWYX [01:18:37]: in some sense, JONATHAN [01:18:37]: language is the God tool. SWYX [01:18:39]: If only, you know, JONATHAN [01:18:39]: we knew how to interpret it all the time. So code is has the really nice property SWYX [01:18:44]: that at least you can JONATHAN [01:18:44]: always execute it. And sometimes you just literally want your model to be able to do SQL calls and nothing else. And setting those boundaries properly for the problem, SWYX [01:18:52]: I think is going to be, I think at least a lot of my customers JONATHAN [01:18:54]: are going to be thinking very hard about that. SWYX [01:18:56]: Like, should I give JONATHAN [01:18:56]: the model access to the web? SWYX [01:18:58]: Is that actually helpful JONATHAN [01:18:58]: for this problem? It sounds great to just like flip yes on all the tools. SWYX [01:19:02]: Is that actually going to mean JONATHAN [01:19:02]: I'm going to get better solutions to my problems? SWYX [01:19:04]: So I want to be mindful of time. I think that's basically our sort of recap of our discussion based on Imbue's releases today. I wanted to leave some time for what's next for both of you guys. Maybe Josh, as a guest of honor, you want to go first as to what happens next. JOSH [01:19:19]: We have these releases. We're happy to put these things out. I think there's a lot of stuff SWYX [01:19:22]: that we haven't released. JOSH [01:19:22]: Like, this is not the only thing we've been working on. Most of our actual focus has been on kind of coding and reasoning. In particular, like the things that we're excited about are can we make these things useful? Like Jonathan is saying, right? Like, it's not about toy problems. It's like, can we use these today in our day-to-day workflow and actually have them accelerate us? And I think we have some kind of internal product prototypes and things that we're excited about. And so we're excited to share more about this in the coming, you know, months to quarters as we get it to a place where like other people could maybe get value out of this as well. But that's kind of our real focus right now is like, how do you take these really cool capabilities that are out there that our models have, et cetera. And like, make sure that they're actually useful today for us, like when we're doing real work and then for other people as well. In particular, focused on generating code, understanding code, testing code, verifying it, like starting with the like robust creation of software. Excellent. SWYX [01:20:13]: Jonathan? JONATHAN [01:20:14]: I never like to talk too much about the future because I think you've heard this from me before. I like for us to speak SWYX [01:20:19]: through our work. JONATHAN [01:20:19]: And so I don't, I don't like to tease too much. Our mission is, to Josh's point, to make this stuff useful to 12,000 customers. And not a lot of that ends up making it into the public eye SWYX [01:20:30]: and not a lot of that JONATHAN [01:20:30]: ends up getting released open source. So for this kind of forum where really, you know, SWYX [01:20:34]: where we're talking JONATHAN [01:20:34]: to the community, I'm asking myself right now, like, you know, what exciting things SWYX [01:20:38]: are we going to have JONATHAN [01:20:38]: to offer the community in the next little while? I think the most exciting part is just we're writing a lot of blog posts right now. We're trying to share more and more of our science because I feel like SWYX [01:20:47]: we've been doing JONATHAN [01:20:47]: these big pushes to create these really giant models. SWYX [01:20:50]: I think, Josh, JONATHAN [01:20:50]: I'm sure you had SWYX [01:20:51]: the same experience. JONATHAN [01:20:51]: It's exhausting and all-consuming and you get to the end SWYX [01:20:54]: and you're like, JONATHAN [01:20:54]: oh, I have all this stuff SWYX [01:20:56]: I want to talk about. JONATHAN [01:20:56]: Now I need to find the time to talk about it now that I've survived this huge push. And we're definitely in that mode right now. So there's going to be a lot of that coming in in the next little while. And, you know, we're always cooking up fun new models. I think the real question is, you know, releasing models open source is not our day-to-day bread and butter. It's kind of a fun reward that we get to do sometimes when we have something really cool to share and a little bit of time and spare GPUs in our hands. But for the most part, SWYX [01:21:20]: everything is going JONATHAN [01:21:20]: toward customers. You know, I think the joke is Databricks has been 18 months away from IPO for five years. So I guess Databricks SWYX [01:21:26]: is 18 months away JONATHAN [01:21:26]: from IPO still. But 18 months away from IPO means there's a lot of pressure to deliver for customers. And we're going to keep working on that. But I think you'll see hopefully some cool, interesting things SWYX [01:21:36]: get dropped over the course JONATHAN [01:21:36]: of the summer and into the fall. We'll find out when we get there. SWYX [01:21:39]: I think that's the right way JONATHAN [01:21:39]: to put it. I know we were talking earlier about kind of Abracadabra and Alakazam. And all I'll say is that, you know, the DBRX small model that we still haven't released yet was called Abra. DBRX was called Kadabra. And there's a third Pokémon in that evolution. And that's all I'll say for now. Cool stuff kind of popping up sometimes on Chatbot Arena. And, you know, keep your eyes out. Yep. SWYX [01:21:59]: I'll leave the links and the hints in the show notes. That was a very fun way to leave some breadcrumbs for people to follow. Cool. I'll leave everything to sort of some calls to action. We're going to be releasing this next week. So I'll be deep in my conference, the AI Engineer World's Fair. So people can just go to AI.Engineer and livestream it. Do you guys have any other calls to action before you wrap? JOSH [01:22:20]: The only one is, you know, we're definitely hiring. So if you're interested in working on coding, reasoning, interested in working on all of this stuff, you know, from the ground up and really deeply understanding not just how does the hardware work, but how do the models work and also designing these, you know, systems to actually be useful for yourself day to day, come say hi. JONATHAN [01:22:36]: The only thing I'll say is, you know, and I like saying it these days, it feels like the field is so crowded and, you know, it requires so many resources to do impactful work. And, you know, on some days it feels like everything's been done or somebody else is doing everything before you can. At least I remember that feeling every single day of my PhD and even more so now. But I hope like what you heard from Josh today tells you there is so much enormously impactful work to do in the field. If only you take a step back and take a fresh look at some of these things and just talk about what you're doing. There's a huge amount left to do here and a huge amount of exciting work happening every day. And for those who are certainly feeling that exhaustion right now, and I count myself among those folks many days, it's refreshing to see these kinds of drops and see that there is so much more even in things that people feel like they understand how to set up a cluster. My God, you know, even in these evals that we think we understand, there is still more to understand and still more work to do. I hope everybody's keeping at it. SWYX [01:23:32]: All right. Keep on keeping on. Well, thanks so much for your time, guys. That was a great discussion and we'll put the links in the show notes for people to read more. Thanks. Thanks a bunch. JOSH [01:23:40]: Thank you so much. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Kanjun Qiu

Co-founder & CEO

Imbue

Jonathan Frankle

Chief AI Scientist, Databricks

DatabricksMosaicMLMassachusetts Institute of TechnologyMIT

In April 2023 we released an episode named “Mapping the future of *truly* open source models” to talk about Dolly, the first open, commercial LLM. Mike was leading the OSS models team at Databricks at the time. Today, Mike is back on the podcast to give us the “one year later” update on the evolution of large language models and how he’s been using them to build Brightwave, an an AI research assistant for investment professionals. Today they are announcing a $6M seed round (led by Alessio and Decibel!), and sharing some of the learnings from serving customers with >$120B of assets under management in production in the last 4 months since launch. Losing faith in long context windows In our recent “Llama3 1M context window” episode we talked about the amazing progress we have done in context window size, but it’s good to remember that Dolly’s original context size was 1,024 tokens, and this was only 14 months ago. But while understanding length has increased, models are still not able to generate very long answers. His empirical intuition (which matches ours while building smol-podcaster) is that most commercial LLMs, as well as Llama, tend to generate responses most of the time. While Needle in a Haystack tests will pass with flying colors at most context sizes, the granularity of the summary decreases as the context increases as it tries to fit the answer in the same tokens range, rather than returning tokens close to the 4,096 max_output, for example. Recently Rob Mulla from Dreadnode highlighted how LMSys Arena results prefer longer responses by a large margin, so both LLMs and humans have a well documented length bias which doesn’t necessarily track the quality of answer: The way Mike and team solved this is by breaking down the task in multiple subtasks, and then merging them back together. For example, have a book summarized chapter by chapter to preserve more details, and then put those summaries together. In Brightwave’s case, it’s creating multiple subsystems that accomplish different tasks on a large corpus of text separately, and then bringing them all together in a report. For example understanding intent of the question, extracting relations between companies, figuring out if it’s a positive / negative, etc. Mike’s question is whether or not we’ll be able to imbue better synthesis capabilities in the models: can you have synthesis-oriented demonstrations at training time rather than single token prediction? “LLMs as Judges” Strategies In our David Luan episode he mentioned they don’t use any benchmarks for their models, because the benchmarks don’t reflect their customer needs. Brightwave shared some tips on leveraging LLMs as Judges: * Human vs LLM reviews: while they work with human annotators to create high quality datasets, that data isn’t just used to fine tune models but also as a reference basis for future LLM reviews. Having a set of trusted data to use as calibration helps you trust the LLM judgement even more. * Ensemble consistency checking: rather than using an LLM as judge for one output, you use different LLMs to generate a result for the same task, and then use another LLM to highlight where those generations differ. Do the two outputs differ meaningfully? Do they have different beliefs about the implications of something? If there are a lot of discrepancies between generations coming from different models, you then do additional passes to try and resolve them. * Entailment verification: for each unique insight that they generate, they take the output and separately ask LLMs to verify factuality of information based on the original sources. In the actual product, user can then highlight any piece of text and ask it to 1) “Tell Me More” 2) “Show Sources”. Since there’s no way to guarantee factuality of 100% of outputs, and humans have good intuition for things that look out of the ordinary, giving the user access to the review tool helps them build trust in it. It’s all about the data During his time at Databricks, they had created dolly-15k, a dataset of instruction-following records written by thousands of their employees. Since then, no other company has replicated that type of effort even though the data wars are in full effect. It’s been clear in the last year that the half-life of a model is much shorter than the half-life of a dataset. The Pile by Eleuther (see Datasets 101) came out in 2020 and is still widely used; if you had trained an LLM in 2020, you would have definitely replaced it by now as they have gotten better and cheaper. On the age old “RAG v Fine-Tuning” question, Mike shared a great example that we’ll just quote: I think of language models kind of like a stem cell, and then under fine tuning, they differentiate into different kinds of specific cells. I don't think that unbounded agentic behaviors are useful, and that instead, a useful LLM system is more like a finite state machine where the behavior of the system is occupying one of many different behavioral regimes and making decisions about what state should I occupy next in order to satisfy the goal. As you think about the graph of those states that your system is moving through, once you develop conviction that one behavior is useful and repeatable and worthwhile to differentiate down into a specific kind of subsystem, that's where like fine tuning and specifically generating the training data, like having human annotators produce a corpus that is useful enough to get a specific class of behaviors, that's kind of how we use fine tuning rather than trying to imbue net new information into these systems. There are a lot of other nuggets in the episode around knowledge graphs extraction, private vs public data, user intent extraction, etc, but we only have so much room in the writeup so go listen! And if you’re interested in working on these problems, Brightwave is hiring 👀 Watch on YouTube We like Mike. The camera likes Mike. Our audience loooves Mike. Show Notes * Brightwave * Mike Conover * Mike on Latent Space #1 * Nature paper on S&P 500 talent movement * Dolly announcement * Dolly 15K dataset * Bard blog post on double-checking generation * RLHF 201 episode * David Luan Episode * Red Pajama * Snorkel * Renaissance Timestamps * [00:00:00] Introductions * [00:02:40] Social media's polarization influence on LLMs * [00:04:09] What's Brightwave? * [00:05:13] How to hire for a vertical AI startup * [00:09:34] How $20B+ hedge funds use Brightwave * [00:11:23] Evolution of context sizes in language models * [00:14:36] Summarizing vs Ideating with AI * [00:18:26] Collecting feedback in a field with no truth * [00:20:49] Evaluation strategies and the importance of custom datasets * [00:23:43] Should more companies make employees label data? * [00:25:32] Retrieval for highly temporal and hierarchical data * [00:30:05] Context-aware prompting for private vs. public data * [00:32:01] Knowledge graph extraction and structured information retrieval * [00:33:49] Fine-tuning vs RAG * [00:36:16] Anthropomorphizing language models * [00:38:20] Why Brightwave doesn't do spreadsheets * [00:42:24] Will there be fully autonomous hedge funds? * [00:47:58] State of open source AI * [00:53:53] Hiring and team expansion at Brightwave Transcript Alessio [00:00:01]: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO in Residence at Decibel Partners, and I have no co-host today. Swyx is in Vienna at ICLR having fun in Europe, and we're in the brand new studio. As you might see, if you're on YouTube, there's still no sound panels on the wall. Mike tried really hard to put them up, but the glue is a little too old for that. So if you hear any echo or anything like that, sorry, but we're doing the best that we can. And today we have our first repeat guest, Mike Conover. Welcome Mike, who's now the founder of Brightwave, not Databricks anymore. Mike [00:00:40]: That's right. Yeah. Pleased to be back. Alessio [00:00:42]: Our last episode was one of the fan favorites, and I think this will be just as good. So for those that have not listened to the first episode, which might be many because the podcast has grown a lot since then, thanks to people like Mike who have interesting conversations on it. You spent a bunch of years doing ML at some of the best companies on the internet, things like Workday, you know, Skipflag, LinkedIn, most recently at Databricks where you were leading the open source large language models team working on Dolly. And now you're doing Brightwave, which is in the financial services space. But this is not something new, I think when you and I first talked about Brightwave, I was like, why is this guy doing a financial services company? And then you look at your background and you were doing papers on The Nature Magazine about LinkedIn data predicting S&P 500 stock movement, like many, many years ago. So what are some of the tying elements in your background that maybe people are overlooking that brought you to do this? Mike [00:01:36]: Yeah, sure. Yeah. So my PhD research was funded by DARPA and we had access to the Twitter data set early in the natural history of the availability of that data set, and it was focused on the large scale structure of propaganda and misinformation campaigns. And LinkedIn, we had planet scale descriptions of the structure of the global economy. And so primarily my work was homepage news feed relevant. So when you go to LinkedIn.com, you'd see updates from one of our machine learning models. But additionally, I was a research liaison as part of the economic graph challenge and had this Nature Communications paper where we demonstrated that 500 million jobs transitions can be hierarchically clustered as a network of labor flows and could predict next quarter S&P 500 market gap changes. And at Workday, I was director of financials machine learning. You start to see how organizations are organisms. And I think of the way that like an accountant or the market encodes information in databases similar to how social insects, for example, organize their work and make collective decisions about where to allocate resources or time and attention. And that especially with the work on Twitter, we would see network structures relating to polarization emerge organically out of the interactions of many individual components. And so like much of my professional work has been focused on this idea that our lives are governed by systems that we're unable to see from our locally constrained perspective. And when humans interact with technology, they create digital trace data that allows us to observe the structure of those systems as though through a microscope or a telescope. And particularly as regards finance, I think the markets are the ultimate manifestation and record of that collective decision making process that humans engage in. Alessio [00:03:21]: Just to start going off script right away, how do you think about some of these interactions creating the polarization and how that reflects in the language models today because they're trained on this data? Like do you think the models pick up on these things on their own as well? Mike [00:03:34]: Absolutely. Yeah. I think they are a compression of the world as it existed at the point in time when they were pre-trained. And so I think absolutely. And you see this in Word2Vec too. I mean, just the semantics of how we think about gender as it relates to professions are encoded in the structure of these models and like language models, I think are much more sort of complete representation of human sort of beliefs. Alessio [00:04:01]: So we left you at Databricks last time you were building Dolly. Tell us a bit more about Brightwave. This is the first time you're really talking about it publicly. Mike [00:04:09]: Yeah. Yeah. And it's a pleasure. So Brightwave is a $6 million seed round, led by Decibel, that we love working with, and including participation from Point72, one of the largest hedge funds in the world and Moonfire Ventures. And if you think of the job of an active asset manager, the work to be done is to understand something about the market that nobody else has seen in order to identify a mispriced asset. And it's our view that that is not a task that is well suited to human intellect or attention span. And so much as I was gesturing towards the ability of these models to perceive more than a human is able to, we think that there's a historically unique opportunity to expand individual's ability to reason about the structure of the economy and the markets. It's not clear that you get superhuman reasoning capabilities from human level demonstrations of skill. And by that I mean the pre-training corpus, but then additionally the fine tuning corpuses. I think you largely mimic the demonstrations that are present at model training time. But from a working memory standpoint, these models outclass humans in their ability to reason about these systems. Alessio [00:05:13]: And you started Brightwave with Brandon. What's the story? You two worked together at Workday, but he also has a really relevant background. Mike [00:05:20]: Yes. So Brandon Kotara is my co-founder, the CTO, and he's a very special human. So he has a deep background in finance. He was the former CTO of a federally regulated derivatives exchange, but his first deep learning patent was filed in 2018. And so he spans worlds. He has experience building mission critical infrastructure in highly regulated environments for finance use cases, but also was very early to the deep learning party and understand. He led at Workday, was the tech lead for semantic search over hundreds of millions of resumes and job listings. And so just has been working with information retrieval and neural information retrieval methods for a very long time. And so was an exceptional person, and I'm glad to count him among the people that we're doing this with. Alessio [00:06:07]: Yeah. And a great fisherman. Mike [00:06:09]: Yeah. Very talented. Alessio [00:06:11]: That's always important. Mike [00:06:12]: Very enthusiastic. Alessio [00:06:13]: And then you have a bunch of amazing engineers, then you have folks like JP who used to work at Goldman Sachs. Yeah. How should people think about team building in this more vertical domain? Obviously you come from a deep ML background, but you also need some of the industry side. What's the right balance? Mike [00:06:28]: I think one of the things that's interesting about building verticalized solutions in AI in 2024 is that historically, you need the AI capability, you need to understand both how the models behave and then how to get them to interact with other kinds of machine learning subsystems that together perform the work of a system that can reason on behalf of a human. There are also material systems engineering problems in there. So I saw, I forget who this is attributed to, but a tweet that made reference to all of the traditional software companies are trying to hire AI talent and all the AI companies are trying to hire systems engineers, and that is 100% the case. Getting these systems to behave in a predictable and repeatable and observable way is equally challenging to a lot of the methodological challenges. But then you bring in, whether it's law or medicine or public policy or in our case finance, I think a lot of the most valuable, like Grammarly is a good example of a company that has generative work product that is valuable by most humans. Whereas in finance, the character of the insight, the depth of insight and the non-consensusness of the insight really requires fairly deep domain expertise. And even operating an exchange, I mean, when we went to raise it around, a lot of people said, why don't you start a hedge fund? And it's like, there are many, many separate skills that are unrelated to AI in that problem. And so we've brought into the fold domain experts in finance who can help us evaluate the character and sort of steer the system. Alessio [00:07:59]: So that's the team. What does the system actually do? What's the Brightwave product? Mike [00:08:03]: Yeah. I mean, it does many, many things, but it acts as a partner in thought to finance professionals. So you can ask Brightwave a question like, how is NVIDIA's position in the GPU market impacted by rare earth metal shortages? And it will identify as thematic contributors to an investment decision or developing your thesis that in response to export controls on A100 cards, China has put in place licensors on the transfer of germanium and gallium, which are not rare earth metals, but they're semiconductor production inputs and has expanded its control of African and South American mining operations. And so we see, if you think about, we have a $20 billion crossover hedge fund. Their equities team uses this tool to go deep on a thesis. So I was describing this like multiple steps into the value chain or supply chain for companies. We see wealth management professionals using Brightwave to get up to speed extremely quickly as they step into nine conversations tomorrow with clients who are assessing like, do you know something that I don't? Can I trust you to be a steward of my financial wellbeing? We see investor relations teams using Brightwave. You just think about the universe of coverage that a person working in finance needs to be aware of, the ability to rip through filings and transcripts and have a very comprehensive view of the market. It's extremely rate limited by how quickly a person is able to read and not just read, but like solve the blank page problem of knowing what to say about a factor of finding. Alessio [00:09:34]: So you mentioned the $20 billion hedge fund. What's like the range of customers that you work with as far as AUM goes? Mike [00:09:41]: I mean, we have customers across the spectrum. So from $500 million owner operated RIAs to organizations with tens and tens of billions of dollars in asset center management. Alessio [00:09:52]: What else can you share about customers that you're working with? Mike [00:09:55]: Yeah. So we have seen traction that far exceeded our expectations from the market. You sit somebody down with a system that can take any question and generate tight, actionable financial analysis on that subject and the product kind of sells itself. So we see many, many different funds, firms, and strategies that are making use of Brightwave. So you've got 10 person owner operated registered investment advisor, the classical wealth manager, you know, $500 million in AUM. We have crossover hedge funds that have tens and tens of billions of dollars in assets center management, very different use case. So that's more investment research, whereas the wealth managers can use this to step into client interactions, just exceptionally well prepared. We see investor relations teams. We see corporate strategy types that are needing to understand very quickly new markets, new themes, and just the ability to very quickly develop a view on any investment theme or sort of strategic consideration is broadly applicable to many, many different kinds of personas. Alessio [00:10:56]: Yeah. I can attest to the product selling itself, given that I'm a user. Let's jump into some of the technical challenges and work behind it, because there are a lot of things. As I mentioned, you were on the podcast about a year ago. Yep. You had released Dolly from Databricks, which was one of the first open source LLMs. Yep. Dolly had a whopping 1,024 tokens of context size. And today, you know, I think a thousand tokens, a model would be unusable. Mike [00:11:23]: You lose that much out. Alessio [00:11:24]: Yeah, exactly. How did you think about the evolution of context sizes as you built the company and where we are today? What are things that people get wrong? Any commentary there? Mike [00:11:34]: Sure. We very much take a systems of systems approach. When I started the company, I think I had more faith in the ability of large context windows to generally solve problems relating to synthesis. And actually, if you think about the attention mechanism and the way that it computes similarities between tokens at a distance, I, on some level, believed that as you would scale that up, you would have the ability to simultaneously perceive and draw conclusions across vast, disparate bodies of content. And I think that does not empirically seem to be the case. So when, for example, you, and this is something anybody can try, take a very long document, like needle in a haystack. I think, sure, we can do information retrieval on specific fact-finding activities pretty easily. I kind of think about it like summarizing, if you write a book report on an entire book versus a synopsis of each individual chapter, there is a characteristic output length for these models. Let's say it's about 1,200 tokens. It is very difficult to get any of the commercial LLMs or LLAMA to write 5,000 tokens. And you think about it as, what is the conditional probability that I generate an end token? It just gets higher the more tokens are in the context window prior to that sort of next inference step. And so if I have 1,000 words in which to say something, the level of specificity and the level of depth when I am assessing a very large body of content is going to necessarily be less than if I am saying something specific about a sub-passage. I mean, if you think about drawing a parallel to consumer internet companies like LinkedIn or Facebook, there are many different subsystems with it. So let's take the Facebook example. Facebook almost certainly has, I mean, you can see this in your profile, your inferred interests. What are the things that it believes that you care about? Those assessments almost certainly feed into the feed relevance algorithms that would judge what you are, you know, am I going to show you snowboarding content? I'm going to show you aviation content. It's the outputs of one machine learning system feeding into another machine learning system. And I think with modern rag and sort of agent-based reasoning, it is really about creating subsystems that do specific tasks well. And I think the problem of deciding how to decompose large documents into more kind of atomic reasoning units is still very important. Now, it's an open question whether that is a model that is addressable by pre-training or instruction tuning. Like, can you have synthesis-oriented demonstrations at training time? And now this problem is more robustly solved because synthesis is quite different from complete the next word in the great Gatsby. I think empirically is not the case that you can just throw all of the SCC filings in a million token context window and get deep insight that is useful out the other end. Alessio [00:14:36]: Yeah. And I think that's the main difference about what you're doing. It's not about summarizing. It's about coming up with different ideas and kind of like thought threads to pull on. Mike [00:14:47]: Yeah. You know, if I think that GLP-1s are going to blow up the diet industry, identifying and putting in context a negative result from a human clinical trial, or for example, that adherence rates to Ozempic after a year are just 35%, what are the implications of this? So there's an information retrieval component. And then there's a not just presenting me with a summary of like, here's here are the facts, but like, what does this entail? And how does this fit into my worldview, my fund strategy? Broadly, I think that, you know, I mean, this idea, I think, is very eloquently puts it, which is, and this is not my insight, but that language models, and help me know who said this. You may be familiar, but language models are not tools for creating new knowledge. They're tools for helping me create new knowledge. Like they themselves do not do that. I think that that's presently the right way to think about it. Alessio [00:15:36]: Yeah. I've read a tweet about Needle in the Haystack actually being harmful to some of this work because now the model is like too focused on recalling everything versus saying, oh, that doesn't matter. Like ignoring some of the things, if you think about a S1 filing, like 85% is like boilerplate. It's like, you know, previous performance doesn't guarantee future performance. Like the company might not be able to turn a profit in the future, blah, blah, blah. All these things, they always come up again. Mike [00:16:02]: COVID and currency fluctuations. Alessio [00:16:03]: Yeah, yeah, yeah. Yada, yada, yada. We have a large workforce and all of that. Have you had to do any work at the model level to kind of like make it okay to forget these things? Or like have you found that making it a smaller problem than putting them back together kind of solves for that? Mike [00:16:19]: Absolutely. And I think this is where having domain expertise around the structure of these documents. So if you look at the different chunking strategies that you can employ to understand like what is the intent of this clause or phrase, and then really be selective at retrieval time in order to get the information that is most relevant to a user query based on the semantics of that unique document. And I think it's certainly not just a sliding window over that corpus. Alessio [00:16:45]: And then the flip side of it is obviously factuality. You don't want to forget things that were there. How do you tackle that? Mike [00:16:52]: Yeah, I mean, of course, it's a very deep problem. And I think I'll be a little circumspect about the specific kinds of methods we use. This sort of multiple passes over the material and saying, how convicted are you that what you're saying is in fact true? And you can take generations from multiple different models and compare and contrast and say, do these both reach the same conclusion? You can treat it like a voting problem. We train our own models to assess. You can think of this like entailment. Is this supported by the underlying primary sources? And I think that you have methodological approaches to this problem, but then you also have product affordances. There was a great blog post on Bard from the Bard team. It was sort of a design-led product innovation that allows you to ask the model to double-check the work. So if you have a surprising finding, we can let the user discretionarily spend more compute to double-check the work. And I think that you want to build product experiences that are fault tolerant. And the difference between hallucination and creativity is fuzzy. Do you ever get language models with Next Token Prediction as the loss function that are guaranteed to not contain factual misstatements? That is not clear. Now, maybe being able to invoke Code Interpreter, like code generation and then execution in a secure way, helps to solve some of these problems, especially for quantitative reasoning. That may be the case, but for right now, I think you need to have product affordances that allow you to live with the reality that these things are fallible. Alessio [00:18:26]: We did our RLHF 201 episode, just talking about different methods and whatnot. How do you think about something like this, where it's maybe unclear in the short term, even if the product is right? It might give an insight that might be right, but it might not prove until later. So it's kind of hard for the users to say, that's wrong, because actually it might be like, you think it's wrong. Like an investment, that's kind of what it comes down to. Some people are wrong. Some people are right. How do you think about some of the product features that you need and something like this to bring user feedback into the mix and maybe how you approach it today and how you think about it long term? Mike [00:19:01]: Yeah, well, I mean, I think that your point about the model may make a statement which is not actually verifiable. It's like, this may be the case. I think that is where the reason we think of this as a partner in thought, is that humans are always going to have access to information that has not been digitized. And so in finance, you see that, especially with regards to expert call networks, the unstated investment theses that a portfolio manager may have, like, we just don't do biotech. Or we think that Eli Lilly is actually very exposed because of how unpleasant it is to take examples. Right. Those are things that are beliefs about the world, but that may not be like falsifiable right now. And so I think you can, again, take pages from the consumer web playbook and think about personalization. So it is getting a person to articulate everything that they believe is not a realistic task. Netflix doesn't ask you to describe what kinds of movies you like and they give you the option to vote, but nobody does this. And so what I think you do is you observe people's revealed preferences. So one of the capabilities that our system exposes is, given everything that Brightwave has read and assessed, and like the sort of synthesized financial analysis, what are the natural next questions that a person investigating this subject should ask? And you can think of this chain of thought and this deepening kind of investigative process and the direction in which the user steers the attention of this system reveals information about what do they care about, what do they believe, what kinds of things are important. And so at the individual level, but then also at the fund and firm level, you can develop like an implicit representation of your beliefs about the world in a way that you just you're never going to get somebody to write everything down. Alessio [00:20:49]: How does that tie into one of our other favorite topics, e-mails? We had David Luan from Adapt and he mentioned they don't care about benchmarks because their customers don't work on benchmarks, they work on business results. How do you think about that for you? And maybe as you build a new company, when is the time to like still focus on the benchmark versus when it's time to like move on to your own evaluation using maybe labelers or whatnot? Mike [00:21:14]: We use a fair bit of LLM supervision to evaluate multiple different subsystems. And I think that one of the reasons that we pay human annotators to evaluate the quality of the generative outputs, and I think that that is always the reference standard, but we frequently first turn to LLM supervision as a way to have, whether it's at fine-tuning time or even for subsystems that are not generative, what is the quality of the system? I think we will generate a small corpus of high-quality domain expert annotations and always compare that against how well is either LLM supervision or even just a heuristic. A simple thing you can do, this is a technique that we do not use, but as an example, do not generate any integers or any numbers that are not present in the underlying source data. If they're doing rag, you can just say you can't name numbers that are not, it's very sort of heavy-handed, but you can take the annotations of a human evaluator and then compare that. I mean, Snorkel kind of takes a similar perspective, like multiple different weak sort of supervision data sets can give you substantially more than any one of them does on their own. And so I think you want to compare the quality of any evaluation against human-generated sort of benchmark. But at the end of the day, especially for things that are nuanced, is this transcendent poetry, there's just no way to multiple choice your way out of that, you know? And so really where I think a lot of the flywheels for some of the large LLM companies are, it's methodological, obviously, but it's also just data generation. And you think about like, you know, for anybody who's done crowdsource work, and this I think applies to the high-skilled human annotators as well, like you look at the Google search quality evaluator guidelines, it's like a 90 or 120-page rubric describing like, what is a high-quality Google search result? And it's like very difficult to get on a human level people to reproducibly follow a rubric. And so what is your process for orchestrating that motion? Like how do you articulate what is high-quality insight? I think that's where a lot of the work actually happens, and that it's sort of the last resort. Ideally, you want to automate everything, but ultimately the most interesting problems right now are those that are not especially automatable. Alessio [00:23:43]: One thing you did at Databricks was the, well, not that you did specifically, but the team there was like the Dolly 15K dataset. You mentioned people misvalue the value of this data. Why has no other company done anything similar with like creating this employee-led dataset? You can imagine some of these Goldman Sachs, they got like thousands and thousands of people in there. Obviously they have different privacy and whatnot requirements. Do you think more companies should do it? Do you think there's like a misunderstanding of how valuable that is? Mike [00:24:15]: So I think Databricks is a very special company and led by people who are very sort of courageous, I guess is one word for it. Just like, let's just ship it. And I think it's unusual. And it's also because I think most companies will recognize, like if they go to the effort to produce something like that, they recognize that it is competitive advantage to have it and to be the only company that has it. And I think Databricks is in an unusual position in that they benefit from more people having access to these kinds of sources, but you also saw scale, I guess they haven't released it. Alessio [00:24:49]: Well, yeah. I'm sure they have it because they charge people a lot of money. Mike [00:24:51]: They created that alternative to GSM 8K, I believe was how that's said. I guess they too are not releasing that. Alessio [00:25:01]: It's interesting because I talked to a lot of enterprises and a lot of them are like, man, I spent so much money on Scale. And I'm like, why don't you just do it? And they're like, what? Mike [00:25:11]: So I think this again gets to the human process orchestration. It's one thing to do like a single monolithic push to create a training data set like that or an evaluation corpus. But I think it's another to have a repeatable process. And a lot of that realistically is pretty unsexy, like people management work. So that's probably a big part of it. Alessio [00:25:32]: So we have these four wars of AI framework, the data quality war, we kind of touched on a little bit now. About RAG, that's like the other battlefield, RAG and context sizes and kind of like all these different things. You work in a space that has a couple of different things. One, temporality of data is important because every quarter there's new data and like the new data usually overrides the previous one. So you cannot just like do semantic search and hope that you get the latest one. And then you have obviously very structured numbers thing that are very important to the token level. Like, you know, 50% gross margins and 30% gross margins are very different, but you know, this organization is not that different. Any thoughts on like how to build a system to handle all of that as much as you can share, of course? Mike [00:26:19]: Yeah, absolutely. So I think this again, rather than having open ended retrieval, open ended reasoning, our approach is to decompose the problem into multiple different subsystems that have specific goals. And so, I mean, temporality is a great example. When you think about time, I mean, just look at all of the libraries for managing calendars. Time is kind of at the intersection of language and math. And this is one of the places where, without taking specific technical measures to ensure that you get high quality narrative overlays of statistics that are changing over time and have a description of how a PE multiple is increasing or decreasing, and like a retrieval system that is aware of the time, sort of the time intent of the user query, right? So if I'm asking something about breaking news, that's going to be very different than if I'm looking for a thematic account of the past 18 months in Fed interest rate policy. You have to have retrieval systems that are, to your point, like if I just look for something that is a nearest neighbor without any of that temporal or other qualitative metadata overlay, you're just going to get a kind of a bag of facts. I think that that is explicitly not helpful, because the worst failure state for these systems is that they are wrong in a convincing way. And so I think, at least presently, you have to have subsystems that are aware of the semantics of the documents, or aware of the semantics of the intent behind the question, and then have multiple, we have multiple evaluation steps. Once you have the generated outputs, we assess it multiple different ways to know, is this a factual statement given the sort of content that's been retrieved? Alessio [00:28:10]: Yep. And what about, I think people think of financial services, they think of privacy, confidentiality. What's kind of like customer's interest in that, as far as like sharing documents and like, how much of a deal breaker is that if you don't have them? I don't know if you want to share any about that and how you think about architecting the product. Mike [00:28:29]: Yeah, so one of the things that gives our customers a high degree of confidence is the fact that Brandon operated a federally regulated derivatives exchange. That experience in these highly regulated environments, I mean, additionally, at Workday, I worked with the financials product, and without going into specifics, it's exceptionally sensitive data, and you have multiple tenants, and it's just important that you take the right approach to being a steward of that material. And so, from the start, we've built in a way that anticipates the need for controls on how that data is managed, and who has access to it, and how it is treated throughout the lifecycle. And so that, for our customer base, where frequently the most interesting and alpha-generating material is not publicly available, has given them a great degree of confidence in sharing. Some of this, the most sensitive and interesting material, with systems that are able to combine it with content that is either publicly or semi-publicly available, to create non-consensus insight into some of the most interesting and challenging problems in finance. Alessio [00:29:40]: Yeah, we always say it breaks our recommendation systems for LLMs. How do you think about that when you have private versus public data, where sometimes you have public data as one thing, but then the private is like, well, actually, we got this insight model, with this insight scoop that we're going to figure out. How do you think in the RAC system about a value of these different documents? I know a lot of it is secret sauce, but- No, no, it's fine. Mike [00:30:05]: I mean, I think that there is, so I will gesture towards this by way of saying context-aware prompting. So you can have prompts that are composable, and that have different command units that may or may not be present based on the semantics of the content that is being populated into the RAG context window. And so that's something we make great use of, which is, where is this being retrieved from? What does it represent? And what should be in the instruction set in order to treat and respect the underlying contents, not just as like, here's a bunch of text, you figure it out, but this is important in the following way, or this aspect of the SEC filings are just categorically uninteresting, or this is sell-side analysis from a favored source. And so it's that creating it, much like you have with the qualitative, the problem of organizing the work of humans, you have the problem of organizing the work of all of these different AI subsystems, and getting them to propagate what they know through the rest of the stack, so that if you have multiple seven, 10 sequence inference calls, that all of the relevant metadata is propagated through that system, and that you are aware of, where did this come from? How convicted am I that it is a source that should be trusted? I mean, you see this also just in analysis, right? So different, like Seeking Alpha is a good example of just a lot of people with opinions, and some of them are great, some of them are really mid, and how do you build a system that is aware of the user's preferences for different sources? I think this is all related to how, we talked about systems engineering, it's all related to how you actually build the systems. Alessio [00:31:51]: And then, just to kind of wrap on the rec side, how should people think about knowledge graphs and kind of like extraction from documents, versus just like semantic search over the documents? Mike [00:32:01]: Knowledge graph extraction is an area where we're making a pretty substantial investment, and so I think that it is underappreciated how powerful, there's the generative capabilities of language models, but there's also the ability to program them to function as arbitrary machine learning systems, basically for marginally zero cost. And so, the ability to extract structured information from huge, sort of unfathomably large bodies of content in a way that is single pass, so rather than having to reanalyze a document every time that you perform inference or respond to a user query, we believe quite firmly that you can also, in an additive way, perform single pass extraction over this body of text and then bring that into the RAG context window. And this really sort of levers off of my experience at LinkedIn, where you had this structured graph representation of the global economy, where you said, person A works at company B, we believe that there's an opportunity to create a knowledge graph that has resolution that greatly exceeds what any, whether it's Bloomberg or LinkedIn, currently has access to, where we're getting as granular as person X submitted congressional testimony that was critical of organization Y, and this is the language that is attached to that testimony, and then you have a structured data artifact that you can pivot through and reason over that is complementary to the generative capabilities that language models expose. And so it's the same technology being applied to multiple different ends. And this is manifest in the product surface, where it's a highly facetable, pivotable product, but it also enhances the reasoning capability of the system. Alessio [00:33:49]: Yeah, you know, when you mentioned you don't wanna re-query like the same thing over and over, a lot of people may say, well, I'll just fine tune this information in the model, you know? How do you think about that? That was one thing when we started working together, you were like, we're not building foundation models. A lot of other startups were like, oh, we're building the finance financial model, the finance foundation model, or whatever. When is the right time for people to do fine tuning versus RAG? Any heuristics that you can share that you use to think about it? Mike [00:34:19]: So we, in general, I do not, I'll just say like, I don't have a strong opinion about how much information you can imbue into a model that is not present in pre-training through large-scale fine tuning. The benefit of rag is the capability around grounded reasoning. So the, you know, forcing it to attend to a collection of facts that are known and available at inference time, and sort of like materially, like only using these facts. At least in my view, the role of fine tuning is really more around, I think of like language models kind of like a stem cell, and then under fine tuning, they differentiate into different kinds of specific cells, so kidney or an eye cell. And if you think about specifically, like, I don't think that unbounded agentic behaviors are useful, and that instead, a useful LLM system is more like a finite state machine where the behavior of the system is occupying one of many different behavioral regimes and making decisions about what state should I occupy next in order to satisfy the goal. As you think about the graph of those states that your system is moving through, once you develop conviction that one behavior is useful and repeatable and worthwhile to differentiate down into a specific kind of subsystem, that's where like fine tuning and like specifically generating the training data, like having human annotators produce a corpus that is useful enough to get a specific class of behaviors, that's kind of how we use fine tuning rather than trying to imbue net new information into these systems. Alessio [00:36:00]: Yeah, and people always try to turn LLMs into humans. It's like, oh, this is my reviewer, this is my editor. I know you're not in that camp. So any thoughts you have on how people should think about, yeah, how to refer to models? Mike [00:36:16]: I mean, we've talked a little bit about this, and it's notable that I think there's a lot of anthropomorphizing going on, and that it reflects the difficulty of evaluating the systems. Is it like, does the saying that you're the journal editor for Nature, does that help? Like you've got the editor, and then you've got the reviewer and you've got the, you're the private investigator. It's like, this is, I think, literally we wave our hands and we say, maybe if I tell you that I'm gonna tip you, that's gonna help. And it sort of seems to, and like maybe it's just like the more cycles, the more compute that is attached to the prompt and then the sort of like chain of thought at inference time, it's like, maybe that's all that we're really doing and that it's kind of like hidden compute. But our experience has been that you can get really, really high quality reasoning from roughly an agentic system without needing to be too cute about it. You can describe the task and within well-defined bounds, you don't need to treat the LLM like a person in order to get it to generate high quality outputs. Alessio [00:37:24]: And the other thing is like all these agent frameworks are assuming everything is an LLM. Mike [00:37:29]: Yeah, for sure. And I think this is one of the places where traditional machine learning has a real material role to play in producing a system that hangs together. And there are guaranteeable like statistical promises that classical machine learning systems to include traditional deep learning can make about what is the set of outputs and like what is the characteristic distribution of those outputs that LLMs cannot afford. And so like one of the things that we do is we, as a philosophy, try to choose the right tool for the job. And so sometimes that is a de novo model that has nothing to do with LLMs that does one thing exceptionally well. And whether that's retrieval or critique or multiclass classification, I think having many, many different tools in your toolbox is always valuable. Alessio [00:38:20]: This is great. So there's kind of the missing piece that maybe people are wondering about. You do a financial services company and you didn't do anything in Excel. What's the story behind why you're doing partner in thought versus, hey, this is like a AI enabled model that understands any stock and all that? Mike [00:38:37]: Yeah, and to be clear, Brightwave does a fair amount of quantitative reasoning. I think what is an explicit non-goal for the company is to create Excel spreadsheets. And I think when you look at products that work in that way, you can spend hours with an Excel spreadsheet and not notice a subtle bug. And that is a highly non-fault tolerant product experience where you encounter a misstatement in a financial model in terms of how a formula is composed and all of your assumptions are suddenly violated. And now it's effectively wasted effort. So as opposed to the partner in thought modality, which is yes and, like if the model says something that you don't agree with, you can say, take it under consideration. This is not interesting to me. I'm going to pivot to the next finding or claim. And it's more like a dialogue. The other piece of this is that the financial modeling is often very, when we talk to our users, it's very personal. So they have a specific view of how a company is structured. They have the one key driver of asset performance that they think is really, really important. It's kind of like the difference between writing an essay and having an essay, I guess. Like the purpose of homework is to actually develop what do I think about this? And so it's not clear to me that like push a button, have a financial model is solving the actual problem that the financial model affords. That said, we take great efforts to have exceptionally high quality quantitative reasoning. So we think about, and I won't get into too many specifics about this, but we deal with a fair number of documents that have tabular data that is really important to making informed decisions. And so the way that our RAG systems operate over and retrieve from tabular data sources is it's something that we place a great degree of emphasis on it's just, I think the medium of Excel spreadsheets is just, I think not the right play for this class of technologies as they exist in 2024. Alessio [00:40:40]: Yeah, what about 2034? Mike [00:40:42]: 2034? Alessio [00:40:43]: Are people still going to be making Excel models or like, yeah, I think to me, the most interesting thing is like, how are the models abstracting people away from some of these more syntax driven thing and making them focus on what matters to them? Mike [00:40:58]: Yeah, I wouldn't be able to tell you what the future, 10 years from now it looks like. I think anybody who could convince you of that is not necessarily somebody to be trusted. I do think that, so let's draw the parallel to accountants in the 70s. So VisiCalc, I believe came out in 1979. And historically the core, you know, you would have as an accountant, as a finance professional in the 70s, like I'm the one who runs the, I run the numbers. I do the arithmetic and that's like my main job. And we think that, I mean, you just look now that's not a job anybody wants. And the sophistication of the analysis that a person is able to perform as a function of having access to powerful tools like computational spreadsheets is just much greater. And so I think that with regards to language models, it is probably the case that there is a play in the workflow where it is commenting on your analysis within that, you know, spreadsheet based context, or it is taking information from those models and sucking this into a system that does qualitative reasoning on top of that. But I think the, it is an open question as to whether the actual production of those models is still a human task. But I think the sophistication of the analysis that is available to us and the completeness of that analysis necessarily increases over time. Alessio [00:42:24]: What about AI hedge funds? Obviously, I mean, we have quants today, right? But those are more kind of like momentum driven, kind of like signal driven and less about long thesis driven. Do you think that's a possibility? Mike [00:42:35]: It's, this is an interesting question. I would put it back to you and say like, how different is that from what hedge funds do now? I think there is, the more that I have learned about how teams at hedge funds actually behave, and you look at like systematics desks or semi-systematic trading groups, man, it's a lot like a big machine learning team. And it's, I sort of think it's interesting, right? So like, if you look at video games and traditional like Bay Area tech, there's not a ton of like talent mobility between those two communities. You have people that work in video games and people that work in like SaaS software. And it's not that like cognitively they would not be able to work together. It's just like a different set of skill sets, a different set of relationships. And it's kind of like network clusters that don't interact. I think there's probably a similar phenomenon happening with regards to machine learning within the active asset allocation community. And so like, it's actually not clear to me that we don't have AI hedge funds now. The question of whether you have an AI that is operating a trading desk, that seems a little, maybe, like I don't have line of sight to something like that existing yet. No, I mean, I'm always curious. Alessio [00:43:48]: I think about asset management on a few different ways, but venture capital is like extremely power law driven. It's really hard to do machine learning in power law businesses because, you know, the distribution of outcomes is like so small versus public equities. Most high-frequency trading is like very, you know, bell curve, normal distribution. It's like, even if you just get 50.5% at the right scale, you're gonna make a lot of money. And I think AI starts there, right? And today, most high-frequency trading is already AI driven. You know, Renaissance started a long time ago using these models. But I'm curious how it's gonna move closer and closer to like power law businesses, right? I would say some boutique hedge funds, their pitch is like, hey, we're differentiated because we only do kind of like these long-only strategies that are like thesis driven versus, you know, movement driven. And most venture capitalists will tell you, well, our fund is different because we have this unique thesis on this market. And I think like five years ago, I've read this blog post about why machine learning would never work in venture because the things that you're investing in today, they're just like no precedent that should tell you this will work. You know, most new companies, a model will tell you this is not gonna work, you know, versus the closer you get to the public companies, the more any innovation is like, okay, this is kind of like this thing that happened. And I feel like these models are quite good at generalizing and thinking, again, going back to the partnering thought, like thinking about second order. Mike [00:45:13]: Yeah, and that's maybe where concrete example, I think it certainly is the case that we tell retrospective, to your point about venture, we tell retrospective stories where it's like, well, here was the set of observable facts. This was knowable at the time, and these people made the right call and were able to cross correlate all of these different sources and said, this is the bet we're gonna make. I think that process of idea generation is absolutely automatable. And the question of like, do you ever get somebody who just sets the system running and it's making all of its own decisions like that, and it is truly like doing thematic investing or more of the like what a human analyst would be kind of on the hook for, as opposed to like HFT. But the ability of models to say, here is a fact pattern that is noteworthy, and we should pay more attention here. Because if you think about the matrix of like all possible relationships in the economy, it grows with the square of the number of facts you're evaluating, like polynomial with the number of facts you're evaluating. And so if I want to make bets on AI, I think it's like, what are ways to profit from the rise of AI? It is very straightforward to take a model and say, parse through all of these documents and find second order derivative bets and say, oh, it turns out that energy is like very, very adjacent to investments in AI and may not be priced in the same way that GPUs are. And a derivative of energy, for example, is long duration energy storage. And so you need a bridge between renewables, which have fluctuating demands, and the compute requirements of these data centers. And I think, and I'm telling this story as like, having witnessed Brightwave do this work, you can take a premise and say like, what are second and third order bets that we can make on this topic? And it's going to come back with, here's a set of reasonable theses. And then I think a human's role in that world is to assess like, does this make sense given our fund strategy? Does this, is this coherent with the calls that I've had with the management teams? There's this broad body of knowledge that I think humans sort of are the ultimate like, synthesizers and deciders. And like, maybe I'm wrong. Maybe the world of the future looks like, and the AI that truly does everything, I think it is kind of a singularity vector where it's like really hard to reason about like, what that world looks like. And like, you asked me to speculate, but I'm actually kind of hesitant to do so because it's just the forecast, the hurricane path just diverges far too much to have a real conviction about what that looks like. Alessio [00:47:58]: Awesome, I know we've already taken up a lot of your time, but maybe one thing to touch on before wrapping is open source LLMs. Obviously you were at the forefront of it. We recorded our episode the day that Red Pajama was open source and we were like, oh man, this is mind blowing. This is going to be crazy. And now we're going to have an open source dense transformer model that is 400 billion parameters. I don't know if one year ago you could have told me that that was going to happen. So what do you think matters in open source? What do you think people should work on? What are like things that people should keep in mind to evaluate? Okay, is this model actually going to be good? Or is it just like cheating some benchmarks to look good? It's like, is there anything there? Like, yeah, this is the part of the podcast where people already dropped off if they wanted to. So they want to hear the hot things right now. Mike [00:48:46]: I mean, I do think that that's another reason to have your own private evaluation corpuses is so that you can objectively and out of sample measure the performance of these models. And again, sometimes that just looks like giving everybody on the team 250 annotations and saying, we're just going to grind through this. And you have to tell, does this meet? The other thing about doing the work yourself is that you get to articulate your loss function precisely. What is the thing that, what do I actually want the system to behave like? Do I prefer this system or this model or this other model? Yeah, and I think the work around overfitting on the test I think is like that 100% is happening. One notable, in contrast to a year ago, say, the incentives, the economic incentives for companies to train their own foundation models, I think are diminishing. So the window in which you are the dominant pre-train, and let's say that you spend five to $40 million for like a, call it kind of a commodity-ish pre-train, not 400 billion would be another sort of- Alessio [00:49:50]: It costs more than 40 million. Another leap. Mike [00:49:52]: But the kind of thing that, like a small multi-billion dollar mom and pop shop might be able to pull off. The benefit that you get from that is like, I think, diminishing over time. And so I think fewer companies are going to make that capital outlay. And I think that there's probably some material negatives to that. But the other piece is that we're seeing that, at least in the past two and a half, three months, there's a convergence towards like, well, these models all behave fairly similarly. And it's probably that the training data on which they are pre-trained is substantially overlapping. And so it's generalizing a model that generalizes to that training data. And so it's unclear to me that you have this sort of balkanization where there are many different models, each of which is good in its own unique way, versus something like Lama becomes like, listen, this is a fine standard to build off of. We'll see, it's just like the upfront cost is so high. And I think for the people that have the money, the benefit of doing the pre-train is now less. Where I think it gets really interesting is how do you differentiate these and all of these different behavioral regimes? And I think the cost of producing instruction tuning and fine tuning data that creates specific kinds of behaviors, I think that's probably where the next generation of really interesting work starts to happen. If you see that the same model architecture trained on much more training data can exhibit substantially improved performance, it's the value of modeling innovations. For fundamental machine learning and AI research, there is still so much to be done. But I think that much lower hanging fruit, I guess, is developing new kinds of training data corpuses that elicit new behaviors from these models in a specific way. And so that's where, when I think about the availability to like a year ago, you had to have access to fairly high performance GPUs that were hard to get in order to get the experience of multiple reps fine tuning these models. And what you're doing when you take a corpus and then fine tune the model and then see across many inference passes, what is the qualitative character of the output, you're developing your own internal mental model of how does the composition of the training corpus shape the behavior of the model in a qualitative way. A year ago, it was very expensive to get that experience. And now you can just recompose multiple different training corpuses and see like, well, what do I do if I insert this set of demonstrations or I ablate that set of demonstrations? And that I think is a very, very valuable skill and one of the ways that you can have models and products that other people don't have access to. And so I think as more people, as those sensibilities proliferate because more people have that experience, you're gonna see teams that release data corpuses that just imbue the models with new behaviors that are especially interesting and useful. And I think that may be where some of the next sets of kind of innovation differentiation come from. Alessio [00:53:03]: Yeah, yeah, when people ask me, I always tell them the half-life of a model, it's much shorter than a half-life of a dataset. Mike [00:53:08]: Yes, absolutely. Alessio [00:53:09]: I mean, the pile is still around and like core to most of these training runs versus all the models people trained a year ago. It's like, they're at the bottom of the LMC's litter board. Mike [00:53:20]: It's kind of crazy, like I don't, just the parallels to other kinds of computing technology where like the work involved in producing the artifact is so significant and the like shelf life is like a week. You know, I'm sure there's a precedent, but it is remarkable. Alessio [00:53:37]: Yeah, I remember when Dolly was the best open source model. Mike [00:53:42]: Dolly was never the best open source model, but it demonstrated something that was not obvious to many people at the time. Yeah, but we always were clear that it was never state-of-the-art. Alessio [00:53:53]: State-of-the-art or whatever that means, right? This is great, Mike. Anything that we forgot to cover that you want to add? Any call, I know you're, you know, thinking about growing the team. Mike [00:54:03]: We are hiring across the board, AI engineering, classical machine learning, systems engineering and distributed systems, front-end engineering, design. We have many open roles on the team. We hire exceptional people. We fit the job to the person as a philosophy and would love to work with more incredible humans. Awesome. Alessio [00:54:25]: Thank you so much for coming on, Mike. Mike [00:54:26]: Thanks, Alessio. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Mike Conover

Co-founder & CEO, Brightwave

BrightwaveDatabricks

In this episode, we dive into the realm of innovation with Databricks Chief Evangelist Ari Kaplan, exploring the cutting-edge technologies and strategies shaping the future of AI and data. Invest in AI Box: https://Republic.com/ai-box Get on the AI Box Waitlist: ⁠⁠https://AIBox.ai/⁠⁠ AI Facebook Community Learn more about AI in Music Learn more about AI Models See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Ari Kaplan

Chief Evangelist

Databricks

SaaStr 715: The 7 Biggest Mistakes CMOs Make (And How to Avoid Them) with the CMOs of Databricks, Zoom, and Okta In this CMO panel, Rick Schultz, CMO at Databricks, Janine Pelosi, Former CMO at Zoom, and Ryan Carlson, Former CMO at Okta, joined Carilu Dietrich, Advisor at Hypergrowth B2B Startups to talk about the seven biggest mistakes CMOs make. These CMOs have all taken their companies from the early stages through IPOs and have been apart of multi-billion dollar organizations. Here are the 7 big mistakes they made, so you can avoid them. -------------------------------------------------------------------------------------------- SaaStr hosts the largest SaaS community events on the planet. Join us in 2024 at: SaaStr Annual: Sept. 10-12 in the SF Bay Area. Join 12,500 SaaS professionals, CEOs, revenue leaders and investors for the world's LARGEST SaaS community event of the year. Podcast listeners can grab a discount on tickets here: https://www.saastrannual2024.com/buy-tickets?promo=fave20 SaaStr Europa: June 5-6 in London. We'll be hosting the 5th SaaStr Europa in London for two days of content and networking. Join 3,000 SaaS and Cloud leaders. Podcast listeners can grab a discount on Europa tickets here: https://www.saastreuropa2024.com/buy-tickets?promo=fave200 -------------- This episode is sponsored by: Northwest Registered Agent When starting your business, it's important to use a service that will actually help you. Northwest Registered Agent is that service. They'll form your company fast, give you the documents you need to open a business bank account, and even provide you with mail scanning and a business address to keep your personal privacy intact. Visit https://www.northwestregisteredagent.com/saastr to get a 60 percent discount on your next LLC.

Rick Schultz

CMO

Databricks

Janine Pelosi

Former CMO

Zoom

Ryan Carlson

Former CMO

Okta

David Meyer is the SVP Products at Databricks where he drives product strategy and execution. He previously ran Engineering and Product Management at OneLogin, where he grew the company to thousands of customers and market leadership. Before OneLogin, he cofounded UniversityNow, an accredited open university system, running Product and Engineering. Prior to that, David managed a $1 billion portfolio of business intelligence products at SAP and co-led cloud strategy. His first software journey was at Plumtree which went public before being acquired by BEA in 2005. In Today's Episode with David Meyer We Discuss: Entry into Product: How did David make his way into the world of product? Why did he not want to go into it? Why does David advise everyone "do not go into product management"? What does David know now that he wishes he had known when he entered product? 2. How to be a Great Product Leader: Why does David think most leaders suck at leading? Why is the most important thing to make your team feel seen? What can leaders do to ensure this? Why does David help his team members to find other roles outside of the company? 3. Building the Best Product Team: How does David hire for product today? What questions does he ask? What signals does he look for? What are David's biggest hiring mistakes? How did they change his approach? What are the biggest mistakes founders make when hiring for product? Why should you hire people who are not in product today? 4. David Meyer: The Art or Science of Product: Is product more art or science? If David were to put a number on it, what would it be? Is simple always better when it comes to product? Will AI remove the importance and focus on UI? Why are the most impressive companies business model innovations not product innovations?

David Meyer

SVP Products

Databricks

Today’s episode continues our coverage from a16z’s recent AI Revolution event. You’ll hear directly from a16z cofounder Ben Horowitz and Databricks cofounder and CEO, Ali Ghodsi as they answer questions around AI and the enterprise, plus their perspectives on open source, whether benchmarks are BS, and the scramble of universities to take part in the very wave they kicked off decades ago. If you’d like to access all the talks from AI Revolution in full, visit a16z.com/airevolution.   Resources: Find Ali on Twitter: https://twitter.com/alighodsi Find Ben on Twitter: https://twitter.com/bhorowitz Check out Databricks: https://twitter.com/databricks   Stay Updated:  Find a16z on Twitter: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Subscribe on your favorite podcast app: https://a16z.simplecast.com/ Follow our host: https://twitter.com/stephsmithio Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Stay Updated: Find a16z on YouTube: YouTube Find a16z on X Find a16z on LinkedIn Listen to the a16z Show on Spotify Listen to the a16z Show on Apple Podcasts Follow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Ali Ghodsi

Co-founder & CEO

Databricks

Startup MosaicML is on a mission to help the AI community enhance prediction accuracy, decrease costs, and save time by providing tools for easy training and deployment of large AI models. In this episode of NVIDIA's AI Podcast, host Noah Kravitz speaks with MosaicML CEO and co-founder Naveen Rao, about how the company aims to democratize access to large language models. MosaicML, a member of NVIDIA's Inception program, has identified two key barriers to widespread adoption: the difficulty of coordinating a large number of GPUs to train a model and the costs associated with this process. Making training of models accessible is key for many companies who need to control over model behavior, respect data privacy, and iterate fast to develop new products based on AI.

Naveen Rao

Co-Founder and CEO

Unconventional AIDatabricksMosaicMLIntel

In this episode, Nathan sits down with Jonathan Frankle, Chief Scientist, and Abhi Venigalla, Research Scientist of MosaicML. They chat about Mosaic’s custom LLMs, the customers seeking Mosaic out and what their journeys and use cases look like, and exciting developments in Mosaic’s research: including their new inference platform, as well as Mosaic’s MPT-7B-65k+ storywriter model. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics The Cognitive Revolution is a part of the Turpentine podcast network. To learn more: www.turpentine.co TIMESTAMPS: (00:00) Episode Preview (06:04) Mosaic’s business model (07:28) Who uses Mosaic’s custom LLMs? What does their data look like? (09:55)Mosaic’s use cases for custom LLMs (12:47) How much extraction and summarization was done by humans pre-LLMs? (15:28) Sponsor: Omneky (21:50) The journeys of Mosaic’s customers and would a Wendy’s LLM know about a Big Mac? (25:46) The curriculum model and fine-tuning (29:10) Language models in the life sciences (33:20) How raw can data be before it becomes a problem? (35:44) Using the output of bulk pre-training process vs additional after training (38:30) Redteaming as a service (39:40) Mosaic’s inference platform (41:53) Spending one cent on 20,000 tokens, how is that cent distributed? (46:00)) Selling compute on a dedicated capacity basis (47:30) Oracle and AWS (49:50) The storywriter model and 65,000 token window (54:35) The transition from finite parameters into infinite attention matrix LINKS: MosaicML: https://www.mosaicml.com/ MPT-7B Storywriter Model: https://huggingface.co/mosaicml/mpt-7b-storywriter TWITTER: @jefrankle (Jonathan) @abhi_venigalla (Abhi) @MosaicML (Mosaic) @CogRev_Podcast @labenz (Nathan) @eriktorenberg (Erik) SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries.From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at erik@turpentine.co. Music Credit: MusicLM

Jonathan Frankle

Chief AI Scientist, Databricks

DatabricksMosaicMLMassachusetts Institute of TechnologyMIT

Abhi Venigalla

Research Scientist

MosaicML

The holy grail of company building is finding product-market fit. But what most people don’t tell you is that once you’ve found it, product-market fit brings its own set of challenges, particularly when it comes to scaling rapidly and hiring the right executives at the right time. Drawing from their extensive experiences, a16z cofounder Ben Horowitz and Databricks cofounder and CEO Ali Ghodsi sit down to talk about the hardest things about executive hiring and firing, and what is at stake. They dive into the common reasons an exec fails, why sometimes micromanagement is a good idea, and the difference between someone who has written a playbook and someone who has only run one. If you’d like to dive even deeper, the Growth team at a16z has spent the last year compiling insights from dozens of leaders at late-stage startups to literally write the book on hiring executives to scale. You can find that book at: http://a16z.com/executive-hiring-playbook Topics Covered: 00:00 - When is it time to let someone go? 05:41 - When do you give someone a chance to grow into the role? 07:05-  Let go or level? 09:27 - How do you have the tough conversation?  16:39 - Exit packages 18:40 - Setting a new leader up for success 24:27 - The importance of the first quick win  26:24 - The impact a new leader has on culture 32:52 - Common ways new leaders fail 37:35 - Closing thoughts Resources: Read the Executive Hiring: The Key to Scaling in the Growth Stages: http://a16z.com/executive-hiring-playbook Learn more about Databricks: https://www.databricks.com Find Ben on Twitter: https://twitter.com/bhorowitz Find Ali on Twitter: https://twitter.com/alighodsi Stay Updated:  Find a16z on Twitter: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Subscribe on your favorite podcast app: https://a16z.simplecast.com/ Follow our host: https://twitter.com/stephsmithio Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. For more details please see a16z.com/disclosures Stay Updated: Find a16z on YouTube: YouTube Find a16z on X Find a16z on LinkedIn Listen to the a16z Show on Spotify Listen to the a16z Show on Apple Podcasts Follow our host: https://twitter.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Ali Ghodsi

Co-founder & CEO

Databricks

MosaicML is a platform for training and deploying large AI models at scale. Explore their docs, check out their blog, and keep an eye on their open roles. Jonathan Frankle is the Chief Scientist at MosaicML and an incoming Assistant Professor of Computer Science at Harvard. Abhinav Venigalla is the NLP Architect at MosaicML. Today’s Lifeboat badge winner is singmotor for rescuing How to remove columns with too many missing values in Python from the dustbin of history. See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Jonathan Frankle

Chief AI Scientist, Databricks

DatabricksMosaicMLMassachusetts Institute of TechnologyMIT

Abhinav Venigalla

NLP Architect

MosaicML

This Week in Startups is presented by: Vanta. Compliance and security shouldn't be a deal-breaker for startups to win new business. Vanta makes it easy for companies to get a SOC 2 report fast. TWiST listeners can get $1,000 off for a limited time at vanta.com/twist. Trovata. Starting up is hard. Trovata makes managing cash easy. Start automating your cash management at Trovata.io/TWIST. Use Code TWIST for 30% off one full year of premium features like AI forecasting. The Microsoft for Startups Founders Hub helps all founders build a better startup, at a lower cost, from day one. Startups get up to $150K in Azure credits, access to free OpenAI credits, free dev tools like GitHub, technical advisory, access to mentors and experts, and so much more. There is no funding requirement, and it only takes minutes to join. Sign up today at aka.ms/thisweekinstartups * Todays show: MosaicML Co-Founder and CEO Naveen Rao joins Jason to discuss the open-source vs closed AI debate, the profound impact of AI on society (41:06) AI’s rapid pace of change, and its implications for the future of employment and education (40:42). They wrap the show by breaking down the potential problems with centralized regulation (54:37). Follow Naveen: https://twitter.com/NaveenGRao Check Out MosaicML: https://mosaicml.com * Time stamps: (00:00) Naveen Rao joins Jason (2:54) MosaicML and its purpose (5:10) Obtaining datasets and incentivizing creators (8:30) Vanta - Get $1000 off your SOC 2 at https://vanta.com/twist (9:37) The process of using your data with MosaicML (11:55) Defining tokens and prompts (16:53) Fine-tuning the AI model and reinforcement learning (19:27) The competition with open-source models (24:26) The cost of running AI models (26:08) Trovata - Use code TWIST at https://trovata.io/twist for 30% off one year of premium features, like AI forecasting (27:35) How the GPU crunch has affected cloud models (32:13) Why demand will not cease (34:21) Specialized models vs. general models (39:12) Microsoft for Startups Founders Hub - Apply in 5 minutes for six figures in discounts at http://aka.ms/thisweekinstartups (40:42) The impact AI will have on employment (48:49) The impact AI will have on education (54:37) Thoughts on OpenAI becoming ClosedAI * Read LAUNCH Fund 4 Deal Memo & Apply for Funding Buy ANGEL Great recent interviews: Brian Chesky, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland, PrayingForExits, Jenny Lefcourt Check out Jason’s suite of newsletters: https://substack.com/@calacanis * Follow Jason: Twitter: https://twitter.com/jason Instagram: https://www.instagram.com/jason LinkedIn: https://www.linkedin.com/in/jasoncalacanis * Follow TWiST: Substack: https://twistartups.substack.com Twitter: https://twitter.com/TWiStartups YouTube: https://www.youtube.com/thisweekin * Subscribe to the Founder University Podcast: https://www.founder.university/podcast

Naveen Rao

Co-Founder and CEO

Unconventional AIDatabricksMosaicMLIntel

This week on The Changelog we’re taking you to the hallway track of The Linux Foundation’s Open Source Summit North America 2023 in Vancouver, Canada. Today’s anthology episode features: Beyang Liu (Co-founder and CTO at Sourcegraph), Denny Lee (Developer Advocate at Databricks), and Stella Biderman (Executive Director and Head of Research at EleutherAI). Special thanks to our friends at GitHub for sponsoring us to attend this conference as part of Maintainer Month. Join the discussion Changelog++ members get a bonus 3 minutes at the end of this episode and zero ads. Join today! Sponsors: DevCycle – Build better software with DevCycle. Feature flags, without the tech debt. DevCycle is a Feature Flag Management platform designed to help you build maintainable code at scale. Sentry – See the untested code causing errors - or whether it’s partially or fully covered - directly in your stack trace, so you can avoid similar errors from happening in the future. Use the code CHANGELOG and get the team plan free for three months. Rocky Linux – Enterprise Linux, the open source community way. Featuring: Beyang Liu – GitHub, X Denny Lee – GitHub, LinkedIn, Mastodon, X Stella Biderman – Website, GitHub, LinkedIn, X Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, X Jerod Santo – Website, GitHub, LinkedIn, Mastodon, X Show Notes: The common denominator for these conversations is open source AI. Beyang Liu and his team at Sourcegraph are focused on enabling more developers to understand code and their approach to a completely open source, model agnostic, coding assistant called Cody has significant interest from us. Denny Lee and the team at Databricks recently released Dolly 2.0, the first open source, instruction-following LLM, that has been fine-tuned on a human-generated instruction dataset and is licensed for research and commercial use. They want to be the platform of choice the future of AI development. Stella Biderman gave the keynote address on generative AI at the conference and works at the base layer doing open source research, model training, and AI ethics. Stella trained the EleutherAI pythia model family that Databricks’ used to create Dolly - 2.0. Cody from Sourcegraph - Read, write, and understand code 10x faster with AI. Cody answers code questions and writes code for you by reading your entire codebase and the code graph. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM EleutherAI - Empowering Open Source Artificial Intelligence Research Something missing or broken? PRs welcome!

Beyang Liu

Co-founder & CTO

Sourcegraph

Denny Lee

Developer Advocate

Databricks

Stella Biderman

Executive Director

EleutherAI

We are excited to be the first podcast in the world to release an in-depth interview on the new SOTA in commercially licensed open source models - MosiacML MPT-7B! The Latent Space crew will be at the NYC Lux AI Summit next week, and have two meetups in June. As usual, all events are on the Community page! We are also inviting beta testers for the upcoming AI for Engineers course. See you soon! One of GPT3’s biggest limitations is context length - you can only send it up to 4000 tokens (3k words, 6 pages) before it throws a hard error, requiring you to bring in LangChain and other retrieval techniques to process long documents and prompts. But MosaicML recently open sourced MPT-7B, the newest addition to their Foundation Series, with context length going up to 84,000 tokens (63k words, 126 pages): This transformer model, trained from scratch on 1 trillion tokens of text and code (compared to 300B for Pythia and OpenLLaMA, and 800B for StableLM), matches the quality of LLaMA-7B. It was trained on the MosaicML platform in 9.5 days on 440 GPUs with no human intervention, costing approximately $200,000. Unlike many open models, MPT-7B is licensed for commercial use and it’s optimized for fast training and inference through FlashAttention and FasterTransformer. They also released 3 finetuned models starting from the base MPT-7B: * MPT-7B-Instruct: finetuned on dolly_hhrlhf, a dataset built on top of dolly-5k (see our Dolly episode for more details). * MPT-7B-Chat: finetuned on the ShareGPT-Vicuna, HC3, Alpaca, Helpful and Harmless, and Evol-Instruct datasets. * MPT-7B-StoryWriter-65k+: it was finetuned with a context length of 65k tokens on a filtered fiction subset of the books3 dataset. While 65k is the advertised size, the team has gotten up to 84k tokens in response when running on a single node A100-80GB GPUs. ALiBi is the dark magic that makes this possible. Turns out The Great Gatsby is only about 68k tokens, so the team used the model to create new epilogues for it! On top of the model checkpoints, the team also open-sourced the entire codebase for pretraining, finetuning, and evaluating MPT via their new MosaicML LLM Foundry. The table we showed above was created using LLM Foundry in-context-learning eval framework itself! In this episode, we chatted with the leads of MPT-7B at Mosaic: Jonathan Frankle, Chief Scientist, and Abhinav Venigalla, Research Scientist who spearheaded the MPT-7B training run. We talked about some of the innovations they’ve brought into the training process to remove the need for 2am on-call PagerDutys, why the LLM dataset mix is such an important yet dark art, and why some of the traditional multiple-choice benchmarks might not be very helpful for the type of technology we are building. Show Notes * Introducing MPT-7B * Cerebras * Lottery Ticket Hypothesis * Hazy Research * ALiBi * Flash Attention * FasterTransformer * List of naughty words for C4 https://twitter.com/code_star/status/1661386844250963972 * What is Sparsity? * Hungry Hungry Hippos * BF16 FP p.s. yes, MPT-7B really is codenamed LLongboi! Timestamps * Introductions [00:00:00] * Intro to Mosaic [00:03:20] * Training and Creating the Models [00:05:45] * Data Choices and the Importance of Repetition [00:08:45] * The Central Question: What Mix of Data Sets Should You Use? [00:10:00] * Evaluation Challenges of LLMs [0:13:00] * Flash Attention [00:16:00] * Fine-tuning for Creativity [00:19:50] * Open Source Licenses and Ethical Considerations [00:23:00] * Training Stability Enhancement [00:25:15] * Data Readiness & Training Preparation [00:30:00] * Dynamic Real-time Model Evaluation [00:34:00] * Open Science for Affordable AI Research [00:36:00] * The Open Approach [00:40:15] * The Future of Mosaic [00:44:11] * Speed and Efficiency [00:48:01] * Trends and Transformers [00:54:00] * Lightning Round and Closing [1:00:55] Transcript Alessio: [00:00:00] Hey everyone. Welcome to the Latent Space podcast. This is Alessio partner and CTO-in-Residence at Decibel Partners. I'm joined by my co-host, Swyx, writer and editor of Latent Space. Swyx: Hey, and today we have Jonathan and Abhi from Mosaic ML. Welcome to our studio. Jonathan: Guys thank you so much for having us. Thanks so much. Swyx: How's it feel? Jonathan: Honestly, I've been doing a lot of podcasts during the pandemic, and it has not been the same. Swyx: No, not the same actually. So you have on your bio that you're primarily based in Boston, Jonathan: New York. New York, yeah. My Twitter bio was a probability distribution over locations. Swyx: Exactly, exactly. So I DMd you because I was obviously very interested in MPT-7B and DMd you, I was like, for the 0.2% of the time that you're in San Francisco, can you come please come to a podcast studio and you're like, I'm there next week. Jonathan: Yeah, it worked out perfectly.  Swyx: We're really lucky to have you, I'll read off a few intros that people should know about you and then you can fill in the blanks. So Jonathan, you did your BS and MS at Princeton in programming languages and then found your way into ML for your PhD at MiT where you made a real splash with the lottery ticket hypothesis in 2018, which people can check up on. I think you've done a few podcasts about it over the years, which has been highly influential, and we'll talk about sparse models at Mosaic. You have also had some side [00:01:30] quest. You taught programming for lawyers and you did some law and privacy stuff in, in DC and also did some cryptography stuff. Um, and you've been an assistant professor at Harvard before earning your PhD. Jonathan:  I've yet to start. Swyx: You, you yet to start. Okay. But you just got your PhD. Jonathan:. I technically just got my PhD. I was at Mosaic which delayed my defense by about two years. It was, I was at 99% done for two years. Got the job at Harvard, Mosaic started, and I had better things to do than write my dissertation for two years.  Swyx: You know, you know, this is very out of order. Jonathan: Like, oh, completely out of order, completely backwards. Go talk to my advisor about that. He's also an advisor at Mosaic and has been from the beginning. And, you know, go talk to him about finishing on time. Swyx: Great, great, great. And just to fill it out, Abhi, you did your BS and MS and MIT, you were a researcher at Cerebras, and you're now a research scientist at Mosaic. Just before we go into Mosaic stuff, I'm actually very curious about Cereus and, uh, just that, that space in general. Um, what are they doing that people should know about? Abhinav: Yeah, absolutely. Um, I think the biggest thing about CEREUS is that they're really building, you know, kind of the NextGen computing platform beyond, like GPUs. Um, they're trying to build a system that uses an entire wafer, you know, rather than cutting up a wafer into smaller chips and trying to train a model on that entire system, or actually more recently on many such wafers. Um, so it's, and it's really extraordinary. I think it's like the first time ever that kind of wafer scale computing has ever really worked. And so it's a really exciting time to be there, trying to figure out how we can map ML workloads to work, um, on a much, much bigger chip. Swyx: And do you use like [00:03:00] a different programming language or framework to do that? Or is that like.. Abhinav: Yeah, so I mean, things have changed a bit since I was there. I think, um, you can actually run just normal tensor flow and pie torch on there. Um, so they've built a kind of software stack that compiles it down. So it actually just kind of works naturally. But yeah. Jonathan : Compiled versions of Python is a hot topic at the moment with Mojo as well.  Swyx: And then Mosaic, you, you spearheaded the MPT-7B effort. INTRO TO MOSAIC [00:03:20] Abhinav: Uh, yeah. Yeah, so it's kind of like, it's been maybe six months, 12 months in the making. We kind of started working on LMs sort of back in the summer of last year. Um, and then we came with this blog post where we kind of profiled a lot of LMs and saw, hey, the cost of training is actually a lot lower than what people might think. Um, and then since then, you know, being inspired by kind of, you know, meta’s release, so the LLaMA models and lots of other open source work, we kind of started working towards, well, what if we were to release a really good kind of 7 billion parameter model? And that's what MPT is.  Alessio:You know, we mentioned some of the podcasts you had done, Jonathan, I think in one of them you mentioned Mosaic was not planning on building a  model and releasing and obviously you eventually did. So what are some of the things that got you there that maybe obviously LLaMA you mentioned was an inspiration. You now have both the training and like inference products that you offer. Was this more of a research challenge in a way, uh, that you wanted to do? Or how did the idea come to be? Jonathan: I think there were a couple of things. So we still don't have a first class model. We're not an open AI where, you know, our businesses come to use our one great model. Our business is built around customers creating their own models. But at the end of the day, if customers are gonna create their own models, we have to have the tools to help them do that, and to have the tools to help them do that and know that they work we have to create our own models to start.  We have to know that we can do something great if customers are gonna do something great. And one too many people may have challenged me on Twitter about the fact that, you know, mosaic claims all these amazing numbers, but, you know, I believe not to, you know, call out Ross Whiteman here, but, you know, I believe he said at some point, you know, show us the pudding. Um, and so Ross, you know, please let me know how the pudding tastes. But in all seriousness, like I think there is something, this is a demo in some sense. This is to say we did this in 9.5 days for a really reasonable cost, straight through 200, an intervention. 200 K. Yep. Um, you can do this too. Swyx: Uh, and just to reference the numbers that you're putting out, this is the, the last year you were making a lot of noise for trading GPT 3 under 450 K, which is your, your initial estimate. Um, and then it went down to a 100 K and stable diffusion 160 k going down to less than 50 K as well. Jonathan: So I will be careful about that 100 K number. That's certainly the challenge I've given Abhi to hit. Oh, I wouldn't make the promise that we’ve hit yet, but you know, it's certainly a target that we have. And I, you know, Abhi may kill me for saying this. I don't think it's crazy.  TRAINING AND CREATING THE MODELS [00:05:45]  Swyx: So we definitely want to get into like estimation math, right? Like what, what needs to happen for those big order magnitude changes to in, in infrastructure costs. But, uh, let's kind of stick to the MPT-7B story. Yeah. Tell us everything. Like you have, uh, three different models. One of them. State of the art essentially on context length. Let's talk about the process of training them, the, uh, the decisions that you made. Um, I can go into, you know, individual details, but I just wanna let you let you rip. Abhinav: Yeah, so I mean, I think, uh, we started off with the base model, which is kind of for all practical purposes, a recreation of LLaMA 7B. Um, so it's a 7 billion perimeter model trained on the trillion tokens. Um, and our goal was like, you know, we should do it efficiently. We should be able to do it like, kind of hands free so we don't have to babysit the runs as they're doing them. And it could be kind of a, a launching point for these fine tune models and those fine tune models, you know, on, on the one hand they're kind of really fun for the community, like the story writer model, which has like a 65,000 length context window and you can even kind of extrapolate beyond that. Um, but they're, they're also kind of just tr inspirations really. So you could kind of start with an MPT-7B base and then build your own custom, you know, downstream. If you want a long context code model, you could do that with our platform. If you wanted one that was for a particular language, you could do that too. But yeah, so we picked kind of the three variance chat and instruct and story writer just kind of like inspirations looking at what people were doing in the community today. Yeah.  Alessio: And what's the beginning of the math to come up with? You know, how many tokens you wanna turn it on? How many parameters do you want in a bottle? 7 billion and 30 billion seem to be kind of like two of the magic numbers going around right now.  Abhinav: Yeah, definitely. Definitely. Yeah, I think like there's sort of these scaling laws which kind of tell you how to best spend your training compute if that's all you cared about. So if you wanna spend $200,000 exactly in the most efficient way, there'd be a recipe for doing that. Um, and that we usually go by the Chinchilla laws. Now for these models, we actually didn't quite do that because we wanted to make sure that people could actually run these at home and that they [00:07:30] were good for inference. So we trained them kind of beyond those chinchilla points so that we're almost over-training them. I think there's like a joke going on online that they're like long boy and that that came up internally because we were training them for really, really long durations. So that 7B model, the chinchilla point might be 140 billion tokens. Instead, we trained a trillion, so almost seven times longer than you normally would. Swyx: So longboi was the code name. So is it, is it the trading method? Is it the scaling law that you're trying to coin or is it the code name for the 64 billion? Jonathan: Uh, 64. It was just an internal joke for the, for training on way more tokens than you would via chinchilla. Okay. Um, we can coin it long boy and it, it really stuck, but just to, you know, long boys filled with two ELs at the beginning. Yeah. Cause you know, we wanted the lLLaMA thing in there as well.  Jonathan: Yeah, yeah, yeah. Our darn CEO we have to rein him in that guy, you know, you can't, yeah. I'm gonna take away his Twitter password at some point. Um, but you know, he had to let that one out publicly. And then I believe there was a YouTube video where someone happened to see it mentioned before the model came out and called it the Long G boy or something like that. Like, so you know, now it's out there in the world. It's out there. It's like Sydnee can't put it back in Swyx: There's a beautiful picture which I think Naveen tweeted out, which, um, shows a long boy on a whiteboard. Jonathan: That was the origin of Long Boy. In fact, the legs of the lLLaMA were the two Ls and the long boy. DATA CHOICES AND THE IMPORTANCE OF REPETITION [00:08:45] Swyx: Well, talk to me about your data choices, right? Like this is your passion project. Like what can you tell us about it? Jonathan: Yeah, I think Abhi wanted to kill me by the end for trying to use all the GPUs on data and none of them on actually training the model.  Um, at the end of the day, We know that you need to train these models and [00:09:00] lots of data, but there are a bunch of things we don't know. Number one is what kinds of different data sources matter. The other is how much does repetition really matter? And really kind of repetition can be broken down into how much does quality versus quantity matter. Suppose I had the world's best 10 billion tokens of data. Would it be better to train on that a hundred times or better to train on a trillion tokens of low quality, fresh data? And obviously there's, there's a middle point in between. That's probably the sweet spot. But how do you even know what good quality data is? And. So, yeah, this is, nobody knows, and I think the more time I spent, we have a whole data team, so me and several other people, the more time that we spent on this, you know, I came away thinking, gosh, we know nothing. Gosh, if I were back in academia right now, I would definitely go and, you know, write a paper about this because I have no idea what's going on. Swyx: You would write a paper about it. I'm interested in such a paper. I haven't come across any that exists. Could you frame the central question of such a paper? THE CENTRAL QUESTION: WHAT MIX OF DATA SETS SHOULD YOU USE? [00:10:00] Jonathan: Yeah. The central question is what mix of data sets should you use? Okay. Actually I've, you know, you had mentioned my law school stuff. I went back to Georgetown Law where I used to teach, um, in the midst of creating this model, and I actually sat down with a class of law students and asked them, I gave them our exact data sets, our data mixes, um, like how many tokens we had, and I said, Create the best data set for your model. Knowing they knew nothing about large language models, they just know that data goes in and it's going to affect the behavior. Um, and I was like, create a mix and they basically covered all the different trade-offs. Um, you probably want a lot of English language [00:10:30] text to start with. You get that from the web, but do you want it to be multilingual? If so, you're gonna have a lot less English text. Maybe it'll be worse. Do you wanna have code in there? There are all these beliefs that code leads to models being better at logical reasoning, of which I've seen zero evidence. Rep. It's not, um, I mean, really made a great code model, but code models leading to better chain of thought reasoning on the part of language or code being in the training set leading to better chain of thought reasoning. People claim this all the time, but I've still never seen any real evidence beyond that. You know, one of the generations of the GPT three model started supposedly from Code Da Vinci. Yes. And so there's a belief that, you know, maybe that helped. But again, no evidence. You know, there's a belief that spending a lot of time on good sources like Wikipedia is good for the model. Again, no evidence. At the end of the day, we tried a bunch of different data mixes and the answer was that there are some that are better or worse than others. We did find that the pile, for example, was a really solid data mix, but you know, there were stronger data mixes by our evaluation metrics. And I'll get back to the evaluation question in a minute cuz that's a really important one. This data set called c4, which is what the original T five model was trained on, is weirdly good. And everybody, when I posted on this on Twitter, like Stella Beaterman from Luther mentioned this, I think someone else mentioned this as well. C4 does really well in the metrics and we have no idea why we de-duplicated it against our evaluation set. So it's not like it memorized the data, it is just one web scrape from 2019. If you actually look at the T five paper and see how it was pre-processed, it looks very silly. Mm-hmm. They removed anything that had the word JavaScript in it because they didn't want to get like no JavaScript [00:12:00] warnings. They removed anything with curly braces cuz they didn't wanna get JavaScript in it. They looked at this list of bad words, um, and removed anything that had those bad words. If you actually look at the list of bad words, words like gay are on that list. And so there's, you know, it is a very problematic, you know, list of words, but that was the cleaning that leads to a data set that seems to be unbeatable. So that to me says that we know nothing about data. We, in fact used a data set called mc four as well, which is they supposedly did the same pre-processing of C4 just on more web calls. The English portion is much worse than C4 for reasons that completely escape us. So in the midst of all that, Basically I set two criteria. One was I wanted to be at least as good as mc four English, like make sure that we're not making things actively worse. And mc four English is a nice step up over other stuff that's out there. And two was to go all in on diversity after that, making sure that we had some code, we had some scientific papers, we had Wikipedia, because people are gonna use this model for all sorts of different purposes. But I think the most important thing, and I'm guessing abhi had a million opinions on this, is you're only as good as your evaluation. And we don't know how to evaluate models for the kind of generation we ask them to do. So past a certain point, you have to kinda shrug and say, well, my evaluation's not even measuring what I care about. Mm-hmm. So let me just make reasonable choices.  EVALUATION CHALLENGES OF LLMs [0:13:00] Swyx: So you're saying MMLU, big bench, that kind of stuff is not. Convincing for you Jonathan: A lot of this stuff is you've got two kinds of tasks. Some of these are more of multiple choice style tasks where there is a right answer. Um, either you ask the model to spit out A, B, C, or D or you know, and if you're more [00:13:30] sophisticated, you look at the perplexity of each possible answer and pick the one that the model is most likely to generate. But we don't ask these models to do multiple choice questions. We ask them to do open-ended generation. There are also open-ended generation tasks like summarization. You compare using things like a blue score or a rouge score, which are known to be very bad ways of comparing text. At the end of the day, there are a lot of great summaries of a paper. There are a lot of great ways to do open form generation, and so humans are, to some extent, the gold standard. Humans are very expensive. It turns out we can't put them into our eval pipeline and just have the humans look at our model every, you know, 10 minutes? Not yet. Not yet. Maybe soon. Um, are you volunteering Abhi? Abhinav: I, I, I just know we have a great eval team who's, uh, who's helping us build new metrics. So if they're listening, Jonathan:  But it's, you know, evaluation of large language models is incredibly hard and I don't think any of these metrics really truly capture. What we expect from the models in practice. Swyx: Yeah. And we might draw wrong conclusions. There's been a debate recently about the emergence phenomenon, whether or not it's a mirage, right? I don't know if you guys have opinions about that process.  Abhinav: Yeah, I think I've seen like this paper and all and all, even just kind of plots from different people where like, well maybe it's just a artifact of power, like log scaling or metrics or, you know, we're meshing accuracy, which is this a very like harsh zero one thing. Yeah. Rather than kind of something more continuous. But yeah, similar to what Jonathan was saying about evals. Like there there's one issue of like you just like our diversity of eval metrics, like when we put these models up, even like the chat ones, the instruct ones, people are using 'em for such a variety of tasks. There's just almost no way we get ahead of time, like measuring individual dimensions. And then also particularly like, you know, at the 7B scale, [00:15:00] um, these models still are not super great yet at the really hard tasks, like some of the hardest tasks in MMLU and stuff. So sometimes they're barely scoring like the above kind of random chance, you know, like on really, really hard tasks. So potentially as we. You know, aim for higher and higher quality models. Some of these things will be more useful to us. But we kind of had to develop MPT 7B kind of flying a little bit blind on, on what we knew it was coming out and just going off of like, you know, a small set of common sensor reasoning tasks. And of course, you know, just comparing, you know, those metrics versus other open source models.  Alessio: I think fast training in inference was like one of the goals, right? So there's always the trade off between doing the hardest thing and like. Doing all the other things quickly. Abhinav: Yeah, absolutely. Yeah, I mean, I think like, you know, even at the 7B scale, you know, uh, people are trying to run these things on CPUs at home. You know, people are trying to port these to their phones, basically prioritizing the fact that the small scale would lead to our adoption. That was like a big, um, big thing going on.  Alessio: Yeah. and you mentioned, um, flash attention and faster transformer as like two of the core things. Can you maybe explain some of the benefits and maybe why other models don't use it? FLASH ATTENTION [00:16:00] Abhinav: Yeah, absolutely. So flash attention is this basically faster implementation of full attention. Um, it's like a mathematical equivalent developed by like actually some of our collaborators, uh, at Stanford. Uh, the hazy research. Hazy research, yeah, exactly. Jonathan: What is, what, what, what's the name hazy research mean? Abhinav: I actually have no idea. Swyx: I have no clue. All these labs have fun names. I always like the stories behind them. Abhinav: Yeah, absolutely. We really, really liked flash attention. We, I think, had to integrate into repo even as [00:16:30] as early as September of last year. And it really just helps, you know, with training speed and also inference speed and we kind of bake that into model architecture. And this is kind of unique amongst all the other hugging face models you see out there. So ours actually, you can toggle between normal torch attention, which will work anywhere and flash attention, which will work on GPUs right out of the box. And that way I think you get almost like a 2x speed up at training time and somewhere between like 50% to a hundred percent speed up at inference time as well. So again, this is just like, we really, really wanted people to use these and like, feel like an improvement and we, we have the team to, to help deliver that.  Swyx: Another part, um, of your choices was alibi position, encodings, which people are very interested in, maybe a lot of people just, uh, to sort of take in, in coatings as, as a given. But there's actually a lot of active research and honestly, it's a lot of, um, it's very opaque as well. Like people don't know how to evaluate encodings, including position encodings, but may, may, could you explain, um, alibi and, um, your choice? Abhinav: Yeah, for sure. The alibi and uh, kind of flash attention thing all kind of goes together in interesting ways. And even with training stability too. What alibi does really is that it eliminates the need to have positional embeddings in your model. Where previously, if you're a token position one, you have a particular embedding that you add, and you can't really go beyond your max position, which usually is like about 2000. With alibies, they get rid of that. Instead, just add a bias to the attention map itself. That's kind of like this slope. And if at inference time you wanna go much, much larger, they just kind of stretch that slope out to a longer, longer number of positions. And because the slope is kind of continuous and you can interpret it, it all works out now. Now one of [00:18:00] the, the funny things we found is like with flash attention, it saved so much memory and like improved performance so much that even as early as I kind of last year, like we were profiling models with, with very long context lines up to like, you know, the 65 k that you seen in release, we just never really got around to using it cuz we didn't really know what we might use it for. And also it's very hard to train stably. So we started experimenting with alibi integration, then we suddenly found that, oh wow, stability improves dramatically and now we can actually work together with alibi in a long context lens. That's how we got to like our story writer model where we can stably train these models out to very, very long context lenses and, and use them performantly. Jonathan: Yeah. Swyx: And it's also why you don't have a firm number. Most people now have a firm number on the context line. Now you're just like, eh, 65 to 85 Abhinav: Oh yeah, there's, there's a, there's a big age to be 64 K or 65 k. 65 k plus. Swyx: Just do powers of twos. So 64 isn't, you know.  Jonathan: Right, right. Yeah. Yeah. But we could, I mean, technically the context length is infinite. If you give me enough memory, um, you know, we can just keep going forever. We had a debate over what number to say is the longest that we could handle. We picked 84 cakes. It's the longest I expect people to see easily in practice. But, you know, we played around for even longer than that and I don't see why we couldn't go longer. Swyx: Yeah. Um, and so for those who haven't read the blog posts, you put the Great Gatsby in there and, uh, asked it to write an epilogue, which seemed pretty impressive. Jonathan: Yeah. There are a bunch of epilogues floating around internally at Mosaic. Yeah. That wasn't my favorite. I think we all have our own favorites. Yeah. But there are a bunch of really, really good ones. There was one where, you know, it's Gatsby's funeral and then Nick starts talking to Gatsby's Ghost, and Gatsby's father shows up and, you know, then he's [00:19:30] at the police station with Tom. It was very plot heavy, like this is what comes next. And a bunch of that were just very Fitzgerald-esque, like, you know, beautiful writing. Um, but it was cool to just see that Wow, the model seemed to actually be working with. You know, all this input. Yeah, yeah. Like it's, it's exciting. You can think of a lot of things you could do with that kind of context length. FINE-TUNING FOR CREATIVITY [00:19:50] Swyx: Is there a trick to fine tuning for a creative task rather than, um, factual task? Jonathan: I don't know what that is, but probably, yeah, I think, you know, the person, um, Alex who did this, he did fine tune the model explicitly on books. The goal was to try to get a model that was really a story writer. But, you know, beyond that, I'm not entirely sure. Actually, it's a great question. Well, no, I'll ask you back. How would you measure that?  Swyx: Uh, God, human feedback is the solve to all things. Um, I think there is a labeling question, right? Uh, in computer vision, we had a really, really good episode with Robo Flow on the segment. Anything model where you, you actually start human feedback on like very, I think it's something like 0.5% of the, the overall, uh, final, uh, uh, labels that you had. But then you sort augment them and then you, you fully automate them, um, which I think could be applied to text. It seems intuitive and probably people like snorkel have already raised ahead on this stuff, but I just haven't seen this applied in the language domain yet. Jonathan: It, I mean there are a lot of things that seem like they make a lot of sense in machine learning that never work and a lot of things that make zero sense that seem to work. So, you know, I've given up trying to even predict. Yeah, yeah. Until I see the data or try it, I just kind shg my shoulders and you know, you hope for the best. Bring data or else, right? Yeah, [00:21:00] exactly. Yeah, yeah, yeah. Alessio: The fine tuning of books. Books three is like one of the big data sets and there was the whole. Twitter thing about trade comments and like, you know, you know, I used to be a community moderator@agenius.com and we've run into a lot of things is, well, if you're explaining lyrics, do you have the right to redistribute the lyrics? I know you ended up changing the license on the model from a commercial use Permitted. Swyx: Yeah let's let them. I'm not sure they did.  Jonathan: So we flipped it for about a couple hours.  Swyx: Um, okay. Can we, can we introduce the story from the start Just for people who are under the loop.  Jonathan: Yeah. So I can tell the story very simply. So, you know, the book three data set does contain a lot of books. And it is, you know, as I discovered, um, it is a data set that provokes very strong feelings from a lot of folks. Um, that was one, one guy from one person in particular, in fact. Um, and that's about it. But it turns out one person who wants a lot of attention can, you know, get enough attention that we're talking about it now. And so we had a, we had a discussion internally after that conversation and we talked about flipping the license and, you know, very late at night I thought, you know, maybe it's a good thing to do. And decided, you know, actually probably better to just, you know, Stan Pat's license is still Apache too. And one of the conversations we had was kind of, we hadn't thought about this cuz we had our heads down, but the Hollywood writer Strike took place basically the moment we released the model. Mm-hmm. Um, we were releasing a model that could do AI generated creative content. And that is one of the big sticking points during the strike. Oh, the optics are not good. So the optics aren't good and that's not what we want to convey. This is really, this is a demo of the ability to do really long sequence lengths and. Boy, you know, [00:22:30] that's, that's not timing that we appreciated. And so we talked a lot internally that night about like, oh, we've had time to read the news. We've had time to take a breath. We don't really love this. Came to the conclusion that it's better to just leave it as it is now and learn the lesson for the future. But certainly that was one of my takeaways is this stuff, you know, there's a societal context around this that it's easy to forget when you're in the trenches just trying to get the model to train. And you know, in hindsight, you know, I might've gone with a different thing than a story writer. I might've gone with, you know, coder because we seem to have no problem putting programmers out of work with these models. Swyx: Oh yeah. Please, please, you know, take away this stuff from me. OPEN SOURCE LICENSES AND ETHICAL CONSIDERATIONS [00:23:00] Jonathan: Right. You know, so it's, I think, you know, really. The copyright concerns I leave to the lawyers. Um, that's really, if I learned one thing teaching at a law school, it was that I'm not a lawyer and all this stuff is a little complicated, especially open source licenses were not designed for this kind of world. They were designed for a world of forcing people to be more open, not forcing people to be more closed. And I think, you know, that was part of the impetus here, was to try to use licenses to make things more closed. Um, which is, I think, against the grain of the open source ethos. So that struck me as a little bit strange, but I think the most important part is, you know, we wanna be thoughtful and we wanna do the right thing. And in that case, you know, I hope with all that interesting licensing fund you saw, we're trying to be really thoughtful about this and it's hard. I learned a lot from that experience.  Swyx: There’s also, I think, an open question of fair use, right? Is training on words of fair use because you don't have a monopoly on words, but some certain arrangements of words you do. And who is to say how much is memorization by a model versus actually learning and internalizing and then. Sometimes happening to land at the right, the [00:24:00] same result. Jonathan: And if I've learned one lesson, I'm not gonna be the person to answer that question. Right, exactly. And so my position is, you know, we will try to make this stuff open and available. Yeah. And, you know, let the community make decisions about what they are or aren't comfortable using. Um, and at the end of the day, you know, it still strikes me as a little bit weird that someone is trying to use these open source licenses to, you know, to close the ecosystem and not to make things more open. That's very much against the ethos of why these licenses were created. Swyx: So the official mosaic position, I guess is like, before you use TC MPC 7B for anything commercial, check your own lawyers now trust our lawyers, not mosaic’s lawyers. Jonathan: Yeah, okay. Yeah. I'm, you know, our lawyers are not your lawyers. Exactly. And, you know, make the best decision for yourself. We've tried to be respectful of the content creators and, you know, at the end of the day, This is complicated. And this is something that is a new law. It's a new law. It's a new law that hasn't been established yet. Um, but it's a place where we're gonna continue to try to do the right thing. Um, and it's, I think, one of the commenters, you know, I really appreciated this said, you know, well, they're trying to do the right thing, but nobody knows what the right thing is to even do, you know, the, I guess the, the most right thing would've been to literally not release a model at all. But I don't think that would've been the best thing for the community either. Swyx: Cool.Well, thanks. Well handled. Uh, we had to cover it, just cause Jonathan:  Oh, yes, no worries. A big piece of news. It's been on my mind a lot. TRAINING STABILITY ENHANCEMENT [00:25:15] Swyx: Yeah. Yeah. Well, you've been very thoughtful about it. Okay. So a lot of these other ideas in terms of architecture, flash, attention, alibi, and the other data sets were contributions from the rest of the let's just call it open community of, of machine learning advancements. Uh, but Mosaic in [00:25:30] particular had some stability improvements to mitigate loss spikes, quote unquote, uh, which, uh, I, I took to mean, uh, your existing set of tools, uh, maybe we just co kind of covered that. I don't wanna sort of put words in your mouth, but when you say things like, uh, please enjoy my empty logbook. How much of an oversell is that? How much, you know, how much is that marketing versus how much is that reality? Abhinav: Oh yeah. That, that one's real. Yeah. It's like fully end-to-end. Um, and I think. Swyx: So maybe like what, what specific features of Mosaic malibu? Abhinav: Totally, totally. Yeah. I think I'll break it into two parts. One is like training stability, right? Knowing that your model's gonna basically get to the end of the training without loss spikes. Um, and I think, you know, at the 7B scale, you know, for some models like it ha it's not that big of a deal. As you train for longer and longer durations, we found that it's trickier and trickier to avoid these lost spikes. And so we actually spent a long time figuring out, you know, what can we do about our initialization, about our optimizers, about the architecture that basically prevents these lost spikes. And you know, even in our training run, if you zoom in, you'll see small intermittent spikes, but they recover within a few hundred steps. And so that's kind of the magical bit. Our line is one of defenses we recover from Las Vegas, like just naturally, right? Mm-hmm. Our line two defense was that we used determinism and basically really smart resumption strategies so that if something catastrophic happened, we can resume very quickly, like a few batches before. And apply some of these like, uh, interventions. So we had these kinds of preparations, like a plan B, but we didn't have to use them at all for MPT 7B training. So, that was kind of like a lucky break. And the third part of like basically getting all the way to the empty law book is having the right training infrastructure.[00:27:00] So this is basically what, like is, one of the big selling points of the platform is that when you try to train these models on hundreds of GPUs, not many people outside, you know, like deep industry research owners, but the GPUs fail like a lot. Um, I would say like almost once every thousand a 100 days. So for us on like a big 512 cluster every two days, basically the run will fail. Um, and this is either due to GPUs, like falling off the bus, like that's, that's a real error we see, or kind of networking failures or something like that. And so in those situations, what people have normally done is they'll have an on-call team that's just sitting round the clock, 24-7 on slack, once something goes wrong. And if then they'll basically like to try to inspect the cluster, take nodes out that are broken, restart it, and it's a huge pain. Like we ourselves did this for a few months. And as a result of that, because we're building such a platform, we basically step by step automated every single one of those processes. So now when a run fails, we have this automatic kind of watch talk that's watching. It'll basically stop the job. Test the nodes cord in anyone's that are broken and relaunch it. And because our software's all deterministic has fast resumption stuff, it just continues on gracefully. So within that log you can see sometimes I think maybe at like 2:00 AM or something, the run failed and within a few minutes it's back up and running and all of us are just sleeping peacefully. Jonathan: I do wanna say that was hard one. Mm-hmm. Um, certainly this is not how things were going, you know, many months ago, hardware failures we had on calls who were, you know, getting up at two in the morning to, you know, figure out which node had died for what reason, restart the job, have to cord the node. [00:28:30] Um, we were seeing catastrophic loss spikes really frequently, even at the 7B scale that we're just completely derailing runs. And so this was step by step just ratcheting our way there. As Abhi said, to the point where, Many models are training at the moment and I'm sitting here in the studio and not worrying one bit about whether the runs are gonna continue. Yeah.  Swyx: I'm, I'm not so much of a data center hardware kind of guy, but isn't there existing software to do this for CPUs and like, what's different about this domain? Does this question make sense at all? Jonathan: Yeah, so when I think about, like, I think back to all the Google fault tolerance papers I read, you know, as an undergrad or grad student mm-hmm. About, you know, building distributed systems. A lot of it is that, you know, Each CPU is doing, say, an individual unit of work. You've got a database that's distributed across your cluster. You wanna make sure that one CPU failing can't, or one machine failing can't, you know, delete data. So you, you replicate it. You know, you have protocols like Paxos where you're literally, you've got state machines that are replicated with, you know, with leaders and backups and things like that. And in this case, you were performing one giant computation where you cannot afford to lose any node. If you lose a node, you lose model state. If you lose a node, you can't continue. It may be that, that in the future we actually, you know, create new versions of a lot of our distributed training libraries that do have backups and where data is replicated so that if you lose a node, you can detect what node you've lost and just continue training without having to stop the run, you know? Pull from a checkpoint. Yeah. Restart again on different hardware. But for now, we're certainly in a world where if anything dies, that's the end of the run and you have to go back and recover from it. [00:30:00] DATA READINESS & TRAINING PREPARATION [00:30:00] Abhinav: Yeah. Like I think a big part, a big word there is like synchronous data pluralism, right? So like, we're basically saying that on every step, every GP is gonna do some work. They're gonna stay in sync with each other and average their, their gradients and continue. Now that there are algorithmic techniques to get around this, like you could say, oh, if a GP dies, just forget about it. All the data that's gonna see, we'll just forget about it. We're not gonna train on it. But, we don't like to do that currently because, um, it makes us give up determinism, stuff like that. Maybe in the future, as you go to extreme scales, we'll start looking at some of those methods. But at the current time it's like, we want determinism. We wanted to have a run that we could perfectly replicate if we needed to. And it was, the goal is figure out how to run it on a big cluster without humans having to babysit it. Babysit it.  Alessio: So as you mentioned, these models are kind of the starting point for a lot of your customers To start, you have a. Inference product. You have a training product. You previously had a composer product that is now kind of not rolled into, but you have like a super set of it, which is like the LLM foundry. How are you seeing that change, you know, like from the usual LOP stack and like how people train things before versus now they're starting from, you know, one of these MPT models and coming from there. Like worship teams think about as they come to you and start their journey. Jonathan: So I think there's a key distinction to make here, which is, you know, when you say starting from MPT models, you can mean two things. One is actually starting from one of our checkpoints, which I think very few of our customers are actually going to do, and one is starting from our configuration. You can look at our friends at Rep for that, where, you know, MPT was in progress when Refl [00:31:30] came to us and said, Hey, we need a 3 billion parameter model by next week on all of our data. We're like, well, here you go. This is what we're doing, and if it's good enough for us, um, hopefully it's good enough for you. And that's basically the message we wanna send to our customers. MPT is basically clearing a path all the way through where they know that they can come bring their data, they can use our training infrastructure, they can use all of our amazing orchestration and other tools that abhi just mentioned, for fault tolerance. They can use Composer, which is, you know, still at the heart of our stack. And then the l l M Foundry is really the specific model configuration. They can come in and they know that thing is gonna train well because we've already done it multiple times.  Swyx: Let's dig in a little bit more on what should people have ready before they come talk to you? So data architecture, eval that they're looking, etc. Abhinav: Yeah, I, I mean, I think we'll accept customers at any kind of stage in their pipeline. You know, like I'd say science, there's archetypes of people who have built products around like some of these API companies and reach a stage or maturity level where it's like we want our own custom models now, either for the purpose of reducing cost, right? Like our inference services. Quite a bit cheaper than using APIs or because they want some kind of customization that you can't really get from the other API providers. I'd say the most important things to have before training a big model. You know, you wanna have good eval metrics, you know, some kind of score that you can track as you're training your models and scaling up, they can tell you you're progressing. And it's really funny, like a lot of times customers will be really excited about training the models, right? It's really fun to like launch shelves on hundreds of gfs, just all around. It's super fun. But then they'll be like, but wait, what are we gonna measure? Not just the training loss, right? I mean, it's gotta be more than that.[00:33:00] So eval metrics is like a, it's a good pre-req also, you know, your data, you know, either coming with your own pre-training or fine-tune data and having like a strategy to clean it or we can help clean it too. I think we're, we're building a lot of tooling around that. And I think once you have those two kinds of inputs and sort of the budget that you want, we can pretty much walk you through the rest of it, right? Like that's kind of what we do. Recently we helped build CR FM's model for biomedical language a while back.  Jonathan: Um, we can. That's the center of research for foundation models.  Abhi: Exactly, exactly. Jonathan: Spelling it out for people. Of course. Abhinav: No, absolutely. Yeah, yeah. No, you've done more of these than I have. Um, I think, uh, basically it's sort of, we can help you figure out what model I should train to scale up so that when I go for my big run company, your here run, it's, uh, it's predictable. You can feel confident that it's gonna work, and you'll kind of know what quality you're gonna get out before you have to spend like a few hundred thousand dollars. DYNAMIC REAL-TIME MODEL EVALUATION [00:34:00] Alessio: The rap Reza from rap was on the podcast last week and, uh, they had human eval and then that, uh, I'm Jon Eval, which is like vibe based.  Jonathan: And I, I do think the vibe based eval cannot be, you know, underrated really at the, I mean, at the end of the day we, we did stop our models and do vibe checks and we did, as we monitor our models, one of our evals was we just had a bunch of prompts and we would watch the answers as the model trained and see if they changed cuz honestly, You know, I don't really believe in any of these eval metrics to capture what we care about. Mm-hmm. But when you ask it, uh, you know, I don't know. I think one of our prompts was to suggest games for a three-year-old and a seven-year-old. That would be fun to play. Like that was a lot more [00:34:30] valuable to me personally, to see how that answer evolved and changed over the course of training. So, you know, and human eval, just to clarify for folks, human human eval is an automated evaluation metric. There's no humans in it at all. There's no humans in it at all. It's really badly named. I got so confused the first time that someone brought that to me and I was like, no, we're not bringing humans in. It's like, no, it's, it's automated. They just called it a bad name and there's only a hundred cents on it or something. Abhinav: Yeah. Yeah. And, and it's for code specifically, right? Jonathan: Yeah. Yeah. It's very weird. It's a, it's a weird, confusing name that I hate, but you know, when other metrics are called hella swag, like, you know, you do it, just gotta roll with it at this point.  Swyx: You're doing live evals now. So one, one of the tweets that I saw from you was that it is, uh, important that you do it paralyzed. Uh, maybe you kind of wanna explain, uh, what, what you guys did. Abhinav: Yeah, for sure. So with LLM Foundry, there's many pieces to it. There's obviously the core training piece, but there's also, you know, tools for evaluation of models. And we've kind of had one of the, I think it's like the, the fastest like evaluation framework. Um, basically it's multi GPU compatible. It runs with Composer, it can support really, really big models. So basically our framework runs so fast that even Azure models are training. We can run these metrics live during the training. So like if you have a dashboard like weights and biases, you kind of watch all these evil metrics. We have, like, 15 or 20 of them honestly, that we track during the run and add negligible overhead. So we can actually watch as our models go and feel confident. Like, it's not like we wait until the very last day to, to test if the models good or not Jonathan: That's amazing. Yeah. I love that we've gotten this far into the conversation. We still haven't talked about efficiency and speed. Those are usually our two watch words at Mosaic, which is, you know, that's great. That says that we're [00:36:00] doing a lot of other cool stuff, but at the end of the day, um, you know, Cost comes first. If you can't afford it, it doesn't matter. And so, you know, getting things down cheap enough that, you know, we can monitor in real time, getting things down cheap enough that we can even do it in the first place. That's the basis for everything we do. OPEN SCIENCE FOR AFFORDABLE AI RESEARCH [00:36:00] Alessio: Do you think a lot of the questions that we have around, you know, what data sets we should use and things like that are just because training was so expensive before that, we just haven't run enough experiments to figure that out. And is that one of your goals is trying to make it cheaper so that we can actually get the answers? Jonathan: Yeah, that's a big part of my personal conviction for being here. I think I'm, I'm still in my heart, the second year grad student who was jealous of all his friends who had GPUs and he didn't, and I couldn't train any models except in my laptop. And that, I mean, the lottery ticket experiments began on my laptop that I had to beg for one K 80 so that I could run amist. And I'm still that person deep down in my heart. And I'm a believer that, you know, if we wanna do science and really understand these systems and understand how to make them work well, understand how they behave, understand what makes them safe and reliable. We need to make it cheap enough that we can actually do science, and science involves running dozens of experiments. When I finally, you know, cleaned out my g c s bucket from my PhD, I deleted a million model checkpoints. I'm not kidding. There were over a million model checkpoints. That is the kind of science we need, you know, that's just what it takes. In the same way that if you're in a biology lab, you don't just grow one cell and say like, eh, the drug seems to work on that cell. Like, there's a lot more science you have to do before you really know. Abhinav: Yeah. And I think one of the special things about Mosaic's kind of [00:37:30] position as well is that we have such, so many customers all trying to train models that basically we have the incentive to like to devote all these resources and time to do this science. Because when we learn which pieces actually work, which ones don't, we get to help many, many people, right? And so that kind of aggregation process I think is really important for us. I remember way back there was a paper about Google that basically would investigate batch sizes or something like that. And it was this paper that must have cost a few million dollars during all the experience. And it was just like, wow, what a, what a benefit to the whole community. Now, like now we all get to learn from that and we get, we get to save. We don't have to spend those millions of dollars anymore. So I think, um, kind of mosaical science, like the insights we get on, on data, on pre-screening architecture, on all these different things, um, that's why customers come to us. Swyx: Yeah, you guys did some really good stuff on PubMed, G B T as well. That's the first time I heard of you. Of you. And that's also published to the community. Abhinav: Yeah, that one was really fun. We were like, well, no one's really trained, like fully from scratch domain specific models before. Like, what if we just did a biomed one? Would it still work? And, uh, yeah, I'd be really excited. That did, um, we'll probably have some follow up soon, I think, later this summer. Jonathan: Yeah. Yes. Stay tuned on that. Um, but I, I will say just in general, it's a really important value for us to be open in some sense. We have no incentive not to be open. You know, we make our money off of helping people train better. There's no cost to us in sharing what we learn with the community. Cuz really at the end of the day, we make our money off of those custom models and great infrastructure and, and putting all the pieces together. That's honestly where the Mosaic name came from. Not off of like, oh, we've got, you know, this one cool secret trick [00:39:00] that we won't tell you, or, you know, closing up. I sometimes, you know, in the past couple weeks I've talked to my friends at places like Brain or, you know, what used to be Brain Now Google DeepMind. Oh, I R I P Brain. Yeah. R i p Brian. I spent a lot of time there and it was really a formative time for me. Um, so I miss it, but. You know, I kind of feel like we're one of the biggest open research labs left in industry, which is a very sad state of affairs because we're not very big. Um, but at least can you say how big the team is actually? Yeah. We were about 15 researchers, so we're, we're tiny compared to, you know, the huge army of researchers I remember at Brain or at fair, at Deep Mind back, you know, when I was there during their heydays. Um, you know, but everybody else is kind of, you know, closed up and isn't saying very much anymore. Yeah. And we're gonna keep talking and we're gonna keep sharing and, you know, we will try to be that vanguard to the best of our ability. We're very small and I, I can't promise we're gonna do what those labs used to do in terms of scale or quantity of research, but we will share what we learn and we will try to create resources for the community. Um, I, I dunno, I just, I believe in openness fundamentally. I'm an academic at heart and it's sad to me to watch that go away from a lot of the big labs.  THE OPEN APPROACH [00:40:15] Alessio: We just had a live pod about the, you know, open AI snow mode, uh, post that came out and it was one of the first time I really dove into Laura and some of the this new technologies, like how are you thinking about what it's gonna take for like the open approach to really work? Obviously today, GPT four is still, you know, part of like that state-of-the-art model for a [00:40:30] lot of tasks. Do you think some of the innovation and kind of returning methods that we have today are enough if enough people like you guys are like running these, these research groups that are open? Or do you think we still need a step function improvement there? Jonathan: I think one important point here is the idea of coexistence. I think when you look at, I don't know who won Linux or Windows, the answer is yes. Microsoft bought GitHub and has a Windows subsystem for Linux. Linux runs a huge number of our servers and Microsoft is still a wildly profitable company. Probably the most successful tech company right now. So who won open source or closed source? Yes. Um, and I think that's a similar world that we're gonna be in here where, you know, it's gonna be different things for different purposes. I would not run Linux on my laptop personally cuz I like connecting to wifi and printing things. But I wouldn't run Windows on one of my surfers. And so I do think what we're seeing with a lot of our customers is, do they choose opening IR mosaic? Yes. There's a purpose for each of these. You have to send your data off to somebody else with open eyes models. That's a risk. GPT four is amazing and I would never promise someone that if they come to Mosaic, they're gonna get a GPT four quality model. That's way beyond our means and not what we're trying to do anyway. But there's also a whole world for, you know, domain specific models, context specific models that are really specialized, proprietary, trained on your own data that can do things that you could never do with one of these big models. You can customize in crazy ways like G B T four is not gonna hit 65 K context length for a very long time, cuz they've already trained that [00:42:00] model and you know, they haven't even released the 32 K version yet. So we can, you know, we can do things differently, you know, by being flexible. So I think the answer to all this is yes. But we can't see the open source ecosystem disappear. And that's the scariest thing for me. I hear a lot of talk in academia about, you know, whatever happened to that academic research on this field called information retrieval? Well, in 1999 it disappeared. Why? Because Google came along and who cares about information retrieval research when you know you have a Google Scale, you know, Web Scale database. So you know, there's a balance here. We need to have both.  Swyx: I wanna applaud you, Elaine. We'll maybe edit it a little like crowd applause, uh, line. Cuz I, I think that, um, that is something that as a research community, as people interested in progress, we need to see these things instead of just, uh, seeing marketing papers from the advertising GPT 4. Jonathan: Yeah. I, I think I, you know, to get on my soapbox for 10 more seconds. Go ahead. When I talk to policymakers about, you know, the AI ecosystem, the usual fear that I bring up is, Innovation will slow because of lack of openness. I've been complaining about this for years and it's finally happened. Hmm. Why is Google sharing, you know, these papers? Why is Open AI sharing these papers? There are a lot of reasons. You know, I have my own beliefs, but it's not something we should take for granted that everybody's sharing the work that they do and it turns out well, I think we took it for granted for a while and now it's gone. I think it's gonna slow down the pace of progress. In a lot of cases, each of these labs has a bit of a monoculture and being able to pass ideas [00:43:30] back and forth was a lot of what kept, you know, scientific progress moving. So it's imperative not just, you know, for the open source community and for academia, but for the progress of technology. That we have a vibrant open source research community. THE FUTURE OF MOSAIC [00:44:11] Swyx: There’s a preview of the ecosystem and commentary that we're, we're gonna do. But I wanna close out some stuff on Mosaic. You launched a bunch of stuff this month. A lot of stuff, uh, actually was, I was listening to you on Gradient descent, uh, and other podcasts we know and love. Uh, and you said you also said you were not gonna do inference and, and, and last week you were like, here's Mosaic ML inference. Oops. So maybe just a, at a high level, what was Mosaic ml and like, what is it growing into? Like how do you conceptualize this?  Jonathan: Yeah, and I will say gradient, when graded dissent was recorded, we weren't doing inference and had no plans to do it. It took a little while for the podcast to get out. Um, in the meantime, basically, you know, one thing I've learned at a startup, and I'm sure abhi can comment on this as well, focus is the most important thing. We have done our best work when we've been focused on doing one thing really well and our worst work when we've tried to do lots of things. Yeah. So, We don't want to do inference, we don't want to have had to do inference. Um, and at the end of the day, our customers were begging us to do it because they wanted a good way to serve the models and they liked our ecosystem. And so in some sense, we got dragged into it kicking and screaming. We're very excited to have a product. We're going to put our best foot forward and make something really truly amazing. But there is, you know, that's something that we were reluctant to do. You know, our customers convinced us it would be good for our business. It's been wonderful for business and we are gonna put everything into this, but you know, back when grading dissent came out, I [00:45:00] was thinking like, or when we recorded it or focused, oh God, like focus is the most important thing. I've learned that the hard way multiple times that Mosaic, abhi can tell you like, you know, I've made a lot of mistakes on not focusing enough. Um, boy inference, that's a whole second thing, and a whole different animal from training. And at the end of the day, when we founded the company, our belief was that inference was relatively well served at that time. There were a lot of great inference companies out there. Um, training was not well served, especially efficient training. And we had something to add there. I think we've discovered that as the nature of the models have changed, the nature of what we had to add to inference changed a lot and there became an opportunity for us to contribute something. But that was not the plan. But now we do wanna be the place that people come when they wanna train these big, complex, difficult models and know that it's gonna go right the first time and they're gonna have something they can servee right away. Um, you know, really the rep example of, you know, with 10 days to go saying, Hey, can you please train that model? And, you know, three or four days later the model was trained and we were just having fun doing interesting, fine tuning work in it for the rest of the 10 days, you know. That also requires good inference.  Swyx: That’s true, that's true. Like, so running evals and, and fine tuning. I'm just putting my business hat on and you know, and Alessio as well, like, uh, I've actually had fights with potential co-founders about this on the primary business. Almost like being training, right? Like essentially a one-time cost. Jonathan: Who told you it was a one time cost? What, who, who told you that? Swyx: No, no, no, no. Correct me.  Jonathan: Yeah. Yeah. Let me correct you in two ways. Um, as our CEO Navine would say, if he were here, when you create version 1.0 of your software, do you then fire all the engineers? Of [00:46:30] course not. You never, like, MPT has a thousand different things we wanted to do that we never got to. So, you know, there will be future models. Abhinav: And, and the data that's been trained on is also changing over time too, right? If you wanna ask anything about, I guess like May of 2023, we'll have to retrain it further and so on. Right? And I think this is especially true for customers who run like the kind of things that need to be up to date on world knowledge. So I, I think like, you know, the other thing I would say too is that, The malls we have today are certainly not the best malls we'll ever produce. Right. They're gonna get smaller, they're gonna get faster, they're gonna get cheaper, they're gonna get lower latency, they're gonna get higher quality. Right? And so you always want the next gen version of MPT and the one after that and one after that. There's a reason that even the GPT series goes three, four, and we know there's gonna be a five. Right? Um, so I I I also don't see as a, as a one-time cost. Jonathan: Yeah. Yeah. And I, if you wanna cite a stat on this, there are very, very few stats floating around on training versus inference cost. Mm-hmm. One is this blog post from I think David Patterson at Google, um, on the energy usage of ML at Google. And they break down and say three fifths of energy over the previous three years. I think this 2022 article was for inference, and two fifths were for training. And so actually that, you know, this is Google, which is serving models to billions of users. They're probably the most inference heavy place in the world. It's only a two fifth, three fifth breakdown, and that's energy training. Hardware is probably more expensive because it has fancier networking. That could be a 50 50 cost breakdown. And that's Google for a lot of other folks. It's gonna be weighed even more heavily, in favor of training. SPEED AND EFFICIENCY [00:48:01] Swyx: Amazing answer. Well, thanks. Uh, we can, we can touch on a little bit [00:48:00] on, uh, efficiency and speed because we, we, uh, didn't mention about that. So right now people spend between three to 10 days. You, you spend 10 days on, on mpc, seven rep spend three days. What's feasible? What's what Do you wanna get it down to? Abhinav: Oh, for, for these original models? Yeah. Yeah. So I think, um, this is probably one of the most exciting years, I think for training efficiency, just generally speaking, because we have the, the combination of a couple things, like one is like this next generation of hardware, like the H 100 s coming out from Nvidia, which on their own should be like, at least like a two x improvement or they 100 s on top of that, there's also a new floating point format f P eight, um, which could also deliver that alone. Does it? Yes. Yeah. Yeah. How, what, why? Oh, the f p thing? Yeah. Yeah. So basically what's happening is that, you know, when we do all of our math, like in the models matrix, multiplication, math, we do it in a particular precision. We started off in 32 bit precision a few years ago, and then in video came with 16 bit, and over the course of several years, we've all figured out how to do 16 bit training and that basically, you know, due to the harder requirements like. Increase the throughput by two x, reduce the cost by two x. That's about to happen again with FBA eight, like starting this year. And with Mosaic, you know, we've already started profiling L L M training with f p eight on H 100 s. We're seeing really, really good improvements there. And so you're gonna see a huge cost reduction this year just from this hardware fact alone. On top of that, you know, there's a lot of architectural applications. We're looking at ways to introduce some forms of sparsity, not necessarily like the, the, the super unstructured sparsity like lottery ticket. Um, which not that I'm sure I'm really happy to talk about. Um, but, but, um, are there ways of doing, like you [00:49:30] gating or like, kind of like m moe style architectures? So, you know, I think originally, you know, what was like 500 k. I think to try and train a Jeep, the equality model, if at the end of the year we could get that down to a hundred k, that would be fantastic. Swyx: That is this year's type of thing.  Jonathan: Not, not, like, that's not a pie in the sky thing. Okay. It is not, it's not a place we are now, but I think it is a, you know, I don't think more than a year in the future these days, cuz it's impossible. I think that is very much a 2023 thing. Yeah. Yeah. Okay. And hold me to that later this year. Swyx: G PT three for a hundred K, let's go. Um, and then also stable diffusion originally reported to be 600 K. Uh, you guys can get it done for under 50. Anything different about image models that we should image, to text? Jonathan: Um, I mean I think the, the most important part in all this is, you know, it took us a while to get 50 down by almost seven x. That was our original kind of proof of concept project for Mosaic. You know, just at the beginning to show like, you know, we can even do this and our investors should give us more money. But what I love about newer models that come out is they're always really slow. We haven't figured out how to optimize them yet. And so there's so much work to be done. So getting, you know, in that case, I guess from the cost you mentioned like a 12 x cost reduction in stable diffusion. Mm-hmm. Honestly it was a lot easier than getting a seven X for RESNET 50 an image net or a three X for Burt, cuz the architecture was much newer and there were a lot of inefficiencies to improve. Um, you know, I'm guessing that's gonna continue to be the case as we lean toward the bleeding edge and try to, you know, push the bleeding edge. I hope that, you know, in some sense you'll see smaller speed ups from us because the new models will come from us and they'll already be fast. Alessio: So that's making existing [00:51:00] things better with the, the long boy, the 60 5K context window, uh, you've doubled instead of the r. There was the R M T a couple weeks ago that had a possible 1 million. Uh, that's the unlimited former thing that came out last week, which is theoretically limitless context. What should people think about trade offs? Implications? You mentioned memories kind of start to become one of the bounds. Yeah. What's the right number? Like is it based on the customer's needs? Like how would you advise customers and startups who might be building their own models? Jonathan: It's all contextual. You know, there's a lot of buzz coming for long contexts lately with a lot of these papers. None of them are exact. In terms of the way that they're doing attention. And so there's, you know, to some extent there's an approximation or a trade off between doing some kind of inexact or approximate or hierarchical or, you know, non quadratic attention versus doing it explicitly correctly the quadratic way. I'm a big fan of approximation, so I'm eager to dig into these papers. If I've learned one thing from writing and reading papers, it's to believe nothing until I've implemented it myself. And we've certainly been let down many, many, many times at Mosaic by papers that look very promising until we implement them and realize, you know, here's how they cook the books on their data. Here's, you know, the one big caveat that didn't show up in the paper. So I look at a lot of this with skepticism until, you know, I believe nothing until I re-implement it. And in general, I'm rewarded for doing that because, you know, a lot of this stuff doesn't end up working quite as well in practice. This is promised in a paper, the [00:52:30] incentives just aren't there, which is part of the reason we went with just pure quadratic attention here. Like it's known to work. We didn't have to make an approximation. There's no asterisk or caveat. This was in some sense a sheer force of will by our amazing engineers. Alessio: So people want super long context because, you know, they wanna feed more documents and right now people do it with embeddings and feed them into the context window. How do you kind of see that changing? Are we gonna get to a point where like, you know, maybe it's 60 4k, maybe it's 120 k, where it's like, okay. You know, semantic search and embeddings are gonna work better than just running a million parameters, like a million token context window. Jonathan: Do, do you wanna say the famous thing about 64 K? Does somebody wanna say that, that statement, the, you know, the 64 K is all you'll ever need? The Bill Gates statement about Rams. Swyx: Andre Kaparthi actually made that comparison before that, uh, context is essentially Ram, Jonathan: if I get quoted here saying 60 4K is all you need, I will be wrong. We have no idea. People are gonna get ambitious. Yes. Um, GPT four has probably taken an image and turning it into a bunch of tokens and plugging it in. I'm guessing each image is worth a hell of a lot of tokens. Um, maybe that's not a thousand words. Not a thousand words, but, you know, probably a thousand words worth of tokens, if not even more so. Maybe that's the reason they did 32 k. Maybe, you know, who knows? Maybe we'll wanna put videos in these models. Like every time that we say, ah, that isn't that model big enough, somebody just gets more ambitious. Who knows?  TRENDS AND TRANSFORMERS [00:54:00] Swyx: Right? Um, you've famously made one. [00:54:00] Countertrend, uh, bet, which is, uh, you, you're actually betting that, uh, transformers will stick around for a long time.  Jonathan: How is that counter trend?  Swyx: Counter trend is in, you just said, a lot of things won't last. Right. A lot of things will get replaced, uh, really easily, but Jonathan: transformers will stick around. I mean, look at the history here. How long did the Convolutional neural network stick around for? Oh wait. They're still here and vision Transformers still haven't replaced them. Mm-hmm. How long did r and n stick around for? Decades. And, you know, they're still alive and kicking in a bunch of different places, so, you know. The fundamental architecture improvements are really hard to come by. I can't wait to collect from Sasha on that bet. Abhinav: I, I think a lot of your bet hinges on what counts as attention, right. Swyx: Wait, what do you mean? Well, how, how can that change? Oh, because it'll be approximated. Abhinav: Well, I suppose if, if we ever replace like the Qk multiplication, something that looks sort of like it, I, I wonder who, who, who comes out on top here. Jonathan: Yeah. I mean at the end of the day is a feed forward network, you know, that's fully connected, just a transformer with very simple attention. Mm-hmm. Um, so Sasha better be very generous to me cause it's possible that could change, but at the end of the day, we're still doing Transformers the way, you know, Vaswani had all intended back six years ago now, so, I don't know, things. Six years is a pretty long time. What's another four years at this point? Alessio: Yeah. What do you think will replace it if you lose Ben? What do you think? You would've lost  it time? Jonathan: If I knew that I'd be working on it. Abhinav:  I think it's gonna be just like MLPs, you know, that's the only, that's the only way we can go, I think at this point, because Thelp, I, I dunno. Oh, just basically down to, to um, to linear layers.[00:55:30] Oh, mostly the percepts. Exactly. Got, yeah. Yeah. Yeah. Cuz the architecture's been stripped, simplified so much at this point. I think, uh, there's very little left other than like some linear layers, some like residual connections and, and of course the attention, um, dot product. Jonathan: But you're assuming things will get simpler, maybe things will get more complicated. Swyx: Yeah, there's some buzz about like, the hippo models. Hungry, hungry hippos. Jonathan: I, I mean there's always buzz about something, um, you know, that's not to dismiss this work or any other work, but there's always buzz about something. I tend to wait a little bit to see if things stand the test of time for like two weeks. Um, at this point, it used to be, you know, a year, but now it's down to two weeks. Oh. But you know, I'm. I don't know. I don't like to follow the hype. I like to see what sticks around, what people actually manage to build off of.  Swyx: I have a follow up question actually on that. Uh, what's a, what's an egregiously overrated paper that once you actually looked into it fell apart completely? Jonathan: I'm not going down that path. Okay. I, you know, I even, even though I think there are papers that, you know, did not hold up under scrutiny, I don't think any of this was out of malice. And so I don't wanna go down that path.  Alessio: Yeah. I know you already talked about your focus on open research. Are you mostly gonna focus on open models or are there also, are you working on configurations that are more just for your customers and private, like, what percentage of your time are you focusing on, on open work? Jonathan: It's a little fuzzy. I mean, I think at the end of the day you have to ask what is the point of our business? Our business is not just to train a bunch of open models and give them to the world. That would, our VCs probably wouldn't be very happy if that were the case. The open [00:57:00] models serve our business because they're demos. A demo does not mean we give away everything. Um, a demo does not mean every single thing we do is shared with the world, but. We do have a business imperative to share with the world, which I kind of like. That was part of the design of the company, was making sure we had an imperative to do science and an imperative to share. But we are still a company and we do have to make money, but it would be a disaster for our business if we didn't share. And that's by design from the start. So, you know, there's certainly going to be some work that we do that is for our customers only, but by and large for anything that we wanna advertise to customers, there has to be something that is meaningful and useful that's out there in the world. Otherwise we can't convince people that we have it.  Abhinav: Yeah, I think like this, our recent inference product also makes the decision easier for us, right? So even since these open malls like we've developed so far, um, you can actually like, you know, uh, query them on our inference api, like our starter tier, and we basically charge like a, a per token fee. Very, very similar to the other API fighters. So there are pathways by which, you know, like even the open mall we provide for free still end up like helping our business out, right? You can customize them, deploy them on our, on our platform, and that way we, we still make money off of them. Alessio: Do you wanna jump into the landing ground? Anything else that you guys wanna cover that we didn't get to? Jonathan: This has been great. These are great questions.  Swyx: Do you want to dish on why Sparsity is not a focus for Mosaic? Jonathan: Um, I can just say that, you know, sparsity is not a focus for Mosaic and I am definitely over lottery tickets when I give my mosaic talk. The first slide is a, you know, a circle with a slash through it over a lottery ticket. [00:58:30] Um, and anyone who mentions lottery tickets, I ask to leave the room. Um, cuz you know there's other work out there. But Abhi, please feel free to dish on sparsity. Abhinav: Yeah, I, I think it really comes down to the fact that we don't have hardware yet that can accelerate it. Right? Or at least it's been mostly true for a long period of time. So the kinds of sparsity that the lottery check was working on was like if you put random zeros in the, in the weights, you know, and basically we found basically the fast year is that yes, you can turn most of the weights to zeros and the model still does kind of work, but there's no hardware out there that can take a matrix with a bunch of zeros and one without and make it go fast. Now, the one caveat for this, and this is gonna sound like a bit of advertisement, is, is Cereus actually, and they've been, since the beginning, they've built that architecture for Sparsity and they've actually published some research papers just earlier this year showing that yes, they really can train with Sparsity and get, this is, uh, sparse. U P T. Exactly. Yeah, exactly right. So, the final missing piece is really like, okay, we have the science to show you can train with sparse models, you know, from initialization even, or, or close initialization. Um, the last piece is just, is there a piece of hardware that actually speeds it up and gives you a cost savings? In which case, like the, the field is wide open.  Jonathan: The other big challenge here is that if you want to make sparsity go fast in general right now on standard hardware, you do need it to be structured in various ways. And any incremental amount of structure that you force on the sparsity dramatically reduces the quality of the resulting model that you get up to the point where if you remove just, you know, entire neurons from the model, you're just making the layers smaller and that really hurts the quality of the model. So these models, steel is all you need. These models love unstructured [01:00:00] sparsity. Um, and yeah, if there were a chip and a software package that made it really, really easy to accelerate it, I bet we would be doing it at Mosaic right now.  Alessio: This is like Sarah Hooker's point with the hardware lottery post, talking about lotteries. Absolutely. Where you know, if you don't have the right hardware, some models, architectures just can't emerge quickly enough. Abhinav: This there, there's like an invariance to think of, which is that today's popular models always run fast on today's hardware. Like this, this has to be true. Mm-hmm. Right? Like there's no such thing as a popular model that runs slow cuz no one would've developed it. Yeah. Um, so it's kind of like with the new architectures, right? If there's new hardware that can do sparsity, you have to co-evolve like a new architecture that works with it. And then those two pair together really well. Transformers and GPUs are like a match made in heaven.  Jonathan: How would say transformers and GPUs are a match made in heaven. Yeah. And we're lucky that they work on GPUs, but the folks at Google D designed them for TPUs cuz TPUs and R and Ns were not a match made in heaven. LIGHTNING ROUND AND CLOSING [1:00:55] Alessio: All right, we have three questions. One is on acceleration, one on exploration, and then just a takeaway for the audience. And you can, you know, either of you can start and the other can finish. So the first one is, what has already happened in AI That thought would take much longer than it has? Abhinav: Do you have an answer, Jon?  Jonathan: Yeah, I have answer everything. Um, you know, I, I remember when GPT two came out and I looked at that and went, eh, you know, that doesn't seem very exciting. And gosh, it's already 1.5 billion parameters. You know, they can't possibly keep getting better as they make it bigger. And then GPT three came out and I was like, eh, it's slightly better at [01:01:30] generating text. Yeah, who cares? And you know, I've been wrong again and again and again. That. Next token prediction, making things big can produce useful models. To be fair, pretty much all of us were wrong about that. So I can't take that precisely on myself. Otherwise, Google, Facebook and Microsoft Research would all have had killer large language models way before opening I ever got the chance to do it. Um, opening I made a very strange bet and it happened to work out very well. But yeah, diffusion models, like they're pretty stupid at the end of the day and they produce beautiful images, it’s astounding. Abhinav: Yeah, I think my, my answer is gonna be like the, the chatbots at scale, like idea, like basically I thought it would be quite a while before, you know, like hundreds of millions of people will be talking to AI models for a large portion of the data, but now there's many startups and companies not, not just open with chat pt, but, but you know, like character and others where, um, it, it's really astounding, like how many people are actually developing like emotional connections to these, to these AI models. And I don't think I was. Would've predicted that like September, October of last year. But you know, the inflection point of the last six months has been really surprising. Swyx: I haven't actually tried any of these models, but I, I don't know. It seems like a very educational thing. It's like, oh, talk to Genius can, but like that's a very educational use case. Right? Right. Like what, what do you think they're using for, I guess, emotional support? Abhinav: Well, yes. I mean, I think some of them are sort of like, yeah, like either for emotional support or honestly just friends and stuff. Right. I mean, I think like, you know, loneliness mental health is a really a big problem everywhere. And so the most interesting I think I've found is that if you go to the subreddits, you know, for those communities and you see like how they [01:03:00] talk about and think about their like AI friends and like these characters, it's, it's, it's like out of a science fiction book, like I would never expect this to be like reality. Swyx: Yeah. What do you think are the most interesting unsolved questions in ai? Abhinav: I'm really interested in seeing how far down we can go in terms of precision and, and stuff like that. Particularly similar to the BF16 FP thing.  Swyx: Okay. Um, there's also like just quantizing until like it's two bits. Abhinav: Yeah, exactly. Like, or even like down to analog or something like that. Because our brains obviously are not running on digital logic and stuff and so, you know, how many orders of magnitude do we have remaining in kind of like just these um, things and I wonder if some of these problems just get easier with scale. Like there have been sort of hints in some papers that, you know, it becomes easier to quantize or easier to prune as it gets bigger and bigger. So maybe as we, almost as a natural consequence of a scaling up over the next few years, will we just naturally become easier and easier to just start going to like four bits or two that are even binary leg weights. Jonathan: I want to know how small we can go in a different way. I just want to know how efficient we can make it to get models that are this good. That was my research question for my entire PhD lottery tickets were one way to get at that. That's now kind of the research question I'm chasing at Mosaic in a sense. I, you know, open ai has shown us that there is one path to getting these incredible capabilities that is scale. I hope that's not the only path. I hope there are lots of ways of getting there. There's better modeling, there are better algorithms. I hate the neuroscience metaphors, but in some sense, our existence and our brains are, you know, evidence that there is at least one other way to get to these kinds of incredible capabilities that doesn't require, you know, [01:04:30] a trillion parameters and megawatts and megawatts and gazillions of dollars. So, you know, I do wonder how small we can go? Is there another path to get to these capabilities without having to do it this way? If it's there, I hope we find it at Mosaic. Swyx: Yeah my, my favorite fact is something on the order of the human brain runs on 30 watts of energy, and so we are, we're doing like dozens of orders of magnitude off on that one. Abhinav: I, I don't think you can get like one gpu, one different. Yeah. Alessio: If there’s one message you want everyone. To remember when thinking about this thing. There's a lot of, you know, fear mongering. There's a lot of messaging being spread around, like, what should people think about in ai? What should be top of mind for them? Jonathan: I'll go for it. Which is, you know, stay balanced. They're the people who really feed into the hype or who, you know, eat up the hype. They're the people who are, you know, big pessimists or react very strongly against the hype, or to some extent are in denial. Stay balanced, embrace the fact that we've built extraordinarily useful tools. Um, but we haven't built a g I and you know, personally, I don't think we're anywhere close to that. You know, so stay balanced and follow the science. I think that's really, that's what we try to do around Mosaic. We try to focus on what's useful to people, what will, you know, hopefully make the world a better place. We try our best on that, but especially, you know, how we can follow the science and use data to be our guide, not just, you know, talk a lot, you know, try to talk through our work instead. Abhinav: And I would also say just kinda like research done in the open. I think like, you know, there's no computing with the, the open community, [01:06:00] right? Just in volume, the number of like, kind of eyeballs you basically have, like looking at your models at the, even at the problems with the models, at ways we improve them. Um, I just think, you know, yeah, research done in the open. It will, it will be the way forward, both to keep our models safe and to bely, like examine the consequences of these AI models like in the world. Alessio: Awesome. Thank you so much guys for coming on. Swyx: and thanks for keeping AI open.  Abhinav: Thank you for having us.  Jonathan: Yeah. Thank you so much for having us. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Jonathan Frankle

Chief AI Scientist, Databricks

DatabricksMosaicMLMassachusetts Institute of TechnologyMIT

Abhinav Venigalla

NLP Architect

MosaicML

The race is on for the first fully GPT3/4-equivalent, truly open source Foundation Model! LLaMA’s release proved that a great model could be released and run on consumer-grade hardware (see llama.cpp), but its research license prohibits businesses from running it and all it’s variants (Alpaca, Vicuna, Koala, etc) for their own use at work. So there is great interest and desire for *truly* open source LLMs that are feasible for commercial use (with far better customization, finetuning, and privacy than the closed source LLM APIs). The previous leading contenders were Eleuther’s GPT-J and Neo on the small end (FLAN-T5 (137B), PaLM (540B), and BigScience’s BLOOM (176B) on the high end. But Databricks is to my knowledge the first to release not just a cleanly licensed, high quality LLM that can run on affordable devices, but also a simple Databricks notebook that can be customized to be finetuned for your data/desired style - for $30 in 30 minutes on one machine! Mike Conover tells the story of how a small team of Applied AI engineers got convinced Ali Ghodsi and 5,000 of their coworkers to join in the adventure of building the first open source, instruction-following LLM, fine-tuned on a human-generated instruction dataset licensed for research and commercial use. He also indulges our questions on other recent open source LLM projects, CerebasGPT and RedPajama, though we recorded this a week before Stability’s StableLM release. Stick around to the end for some easter eggs featuring AI Drake! Recorded in-person at the beautiful StudioPod studios in San Francisco. Full transcript is below the fold. Show Notes * Mike Conover LinkedIn and Twitter * Dolly 1.0 * Dolly 2.0 * CICERO and Diplomacy * Dolly and Deepspeed * LLMops: * https://nat.dev/ * PromptLayer * HumanLoop * Spreadsheets?? * Quadratic * Alessio’s Email GPT Drafter * Open Models * Open Assistant * Cerebras GPT * RedPajama * Reflexion, Recursive Criticism and Improvement * Lightning Round * AI Product: Google Maps * AI People: EleutherAI, Huggingface’s Stas Bekman * AI Prediction: Open LLaMA reproduction, AI Twins of People (AI Drake), Valuing Perplexity * Request for Startups: LLMOps/Benchmarks, Trail Mapping Timestamps * [00:00:21] Introducing Mike Conover * [00:03:10] Dolly 1.0 * [00:04:18] Making Dolly * [00:06:12] Dolly 2.0 * [00:09:28] Gamifying Instruction Tuning * [00:11:36] Summarization - Thumbnails for Language * [00:15:11] CICERO and Geopolitical AI Agents * [00:17:09] Datasets vs Intentional Design * [00:21:44] Biological Basis of AI * [00:23:27] Training Your Own LLMs * [00:28:21] You May Not Need a Large Model * [00:29:59] Good LLM Use cases * [00:31:33] Dolly Cost $30 on Databricks * [00:36:06] Databricks Open Source * [00:37:31] LLMOps and Prompt Tooling * [00:42:26] "I'm a Sheets Maxi" * [00:44:19] AI and Workplace Productivity * [00:47:02] OpenAssistant * [00:47:41] CerebrasGPT * [00:51:35] RedPajama * [00:54:07] Why Dolly > OpenAI GPT * [00:56:19] Open Source Licensing for AI Models * [00:57:09] Why Open Source Models? * [00:58:05] Moving Models * [01:00:34] Learning in a Simulation * [01:01:28] Why Model Reflexion and Self Criticism Works * [01:03:51] Lightning Round Transcripts [00:00:00] Hey everyone. Welcome to the Latent Space Podcast. This is Alessio Partner and CT and Residence and Decibel Partners. I'm Joan Bama, cohost swyx Brighter and Editor of Space. Welcome, Mike. [00:00:21] Introducing Mike Conover [00:00:21] Hey, pleasure to be here. Yeah, so [00:00:23] we tend to try to introduce you so that you don't have to introduce yourself. Yep. [00:00:27] But then we also ask you to fill in the blanks. So you are currently a, uh, staff software engineer at Databricks. Uh, but you got your PhD at Indiana on the University of Bloomington in Complex Systems analysis where you did some, uh, analysis of clusters on, on Twitter, which I found pretty interesting. [00:00:43] Yeah. Uh, I highly recommend people checking that out if you're interested in getting information from indirect sources or I, I don't know how you describe it. Yes. Yeah. And then you went to LinkedIn working on. Homepage News, relevance, and then SkipFlag, which is a smart enterprise knowledge graph, which was then acquired, uh, by Workday, where you became director of machine learning engineering and now your Databricks. [00:01:06] So that's the quick bio and we can kind of go over Yeah. Step by step. But, uh, what's not on your LinkedIn that people [00:01:12] should know about you? So, because I worked at LinkedIn, that's actually how new hires introduce themselves at LinkedIn is this question. So I, okay. I have a pat answer to it. Uhhuh. Um, I love getting off trail in the backcountry. [00:01:25] Okay. And I, you know, I think that the sort of like radical responsibility associated to that is clarifies the mind. And I think that the, the things that I really like about machine learning engineering and sort of the topology of high-dimensional spaces kind of manifest when you think about a topographic mat as a contour plot. [00:01:44] You know, it's a two-dimensional projection of a three-dimensional space and it's very much like looking at information visualizations and you're trying to relate your. Localized perception of the environment around you and the contours of, uh, ridges that you see, or basins that you might go into and you're like, there's that little creek down there. [00:02:04] And relate that to the projection that you see on the map. I think it's physically demanding. It's intellectually challenging. It's natural. Beauty is a big part of it, and you're generally spending time with friends, and so I just, I love that. I love that these are camping trips. Uh, multi-day. Yeah. Yeah. [00:02:21] Camping. I, I hunt too, you know, I, um, shoot archery, um, big game back country hunting, but yeah. You know, sometimes it's just, let's take a walk in the woods and see where it goes. [00:02:32] Oh yeah. You ever think about going on one of those, um, journeys in the, uh, the Australian Outbacks? Like where people find themselves? [00:02:40] I'm [00:02:40] a mountain. I'm a mountain guy. I like to You're mountain guy. I like to fly fish. I like to, you like to hill climb? Yeah. Like the outback seems beautiful. I think eight of the 10 most deadly snakes live in Australia. Like I'm, uh, yeah, you're good. You're good. Yeah. Yeah. [00:02:52] Yeah. Any lessons from like, Real hill climbing [00:02:55] versus machine learning, hill climbing. [00:02:56] Great Dude. It's a lot like gradient descent. Yeah, for sure, man. Um, yeah, I that I have remarked on that to myself before for sure. Yeah, I don't, I'm not sure. This is like least resistance, please. [00:03:10] Dolly 1.0 [00:03:10] That's awesome. So Dolly, you know, it's kind of come up in the last three weeks you went from a brand new project at Databricks to one of the hottest open source things out there. [00:03:19] So March 24th you had Dolly 1.0. It was a 6 billion parameters model based on GPT-J 6 billion and you saw alpaca training set to train it. First question is, why did you start with GPT-J instead of LLaMA, which was what everybody else was kind of starting from [00:03:34] at the time. Yeah, well, I mean, so, you know, we had talked about this a little before the show, but LLaMA's hard to get. [00:03:40] We had requested the model weights and had just not heard back. And you know, I think our experience with the, um, The original email alias for Dolly, before it was available on hugging face, you get hundreds of people asking for it, and I think it's like, it's easy to just not be able to handle the inbound. [00:03:56] Mm-hmm. And so like, I mean, there was a practical consideration, which is that, you know, we did not have the LLaMA weights, but additionally I think it's like much more interesting if anybody can build it. Right. And so I think that was our, um, and I had worked with the GPT-J model in the past and, and knew it to be high quality from a grammatical ness standpoint. [00:04:15] And so I think it was a reasonable choice. Mm-hmm. Yeah. [00:04:18] Making Dolly [00:04:18] Yeah. Maybe we should, we can also go into the impetus of why you started work on Dolly. Uh, you had been at Databricks for about a year. Mm-hmm. Was there, was this like a top-down directive? Was this your idea? We'll see, uh, [00:04:31] what happened? I've been working in N L P and language understanding for a fair while now. [00:04:36] I mean certainly since Skip flag back in 20 16, 20 17, we can introduce Skip flag is that's, if that's, sorry. You know, we don't have to focus too much on it, but like, this is a, an area how information moves through networks of people is a longstanding interest of mine. And we built a hack day project and I just slacked it to our c e o and I was, you know, this was when ChatGPT came out and it was an integration into the developer experience. [00:05:02] And I was like, as a user, this should exist. I want this. Mm-hmm. We should build this. It doesn't have to be us. And I mean, to our, uh, our leadership team is like 10 years into this journey, probably more than that at Databricks. And they are still. So hungry. It's wild. It's just wild to see these, these people in action, you know, this like this far into the marathon. [00:05:23] And, um, he's like, great, build it. Do make it. So, you know, and I, we had have, uh, full-time responsibilities and infrastructure forecasting and infrastructure optimization. And so we did, you know, and, um, we just started building and, you know, so we'd been working on this class of technologies for, um, several months. [00:05:46] And we had a stack that in part how we were able to kind of pivot on the balls of our feet. Uh, we repurposed a lot of existing code that we had built up, you know, in the past several quarters, um, to, to create Dolly and, and just to [00:05:58] be clear, like is this an internal stack or is this, uh, externally available as data? [00:06:02] Much of what we open sourced what, you know, like that that is a, that is the, the, it's, I mean, no, it's not the exhaustive stack by any account, but it's, it's some of the core components. Okay. Yeah. [00:06:12] Dolly 2.0 [00:06:12] It only took 19 days to go from 1.0 to 2.0. Yeah. So 2.0 is 12 billion. So twist the number of parameters. You base this on the model family from Elu. [00:06:23] I instead, and I think the, the biggest change is like instead of using the alpaca turning set, which is change generated, so it has its own limitations, you created a brand new, uh, training data set created by the Databricks employees. So I would love to talk about how you actually made that happen. You know, did you just go around and say, Hey guys, I just need to like today, spend your day coming up with the instruction set? [00:06:47] Or like, did people volunteer to be a part of this? [00:06:50] Yeah, I mean, so again, like a lot of credit to our founding team, they see it, I think as much as anybody you'll talk to who is a new founder or somebody trying to work in this space, like our executives have the fire and will see a, a bright neon meta future that, uh, Databricks will confidently lead. [00:07:12] The world into. And so Ali just sent emails twice a day. Do it, do it. You know, we put together, you know, we, we use the InstructGPT sort of task families, you know, gen content generation, brainstorming close qa, open qa, paraphrasing, things like this, and basically put together these Google forms. [00:07:34] You know, just like, how can we build this as quickly as possible? We see this need, you know, the alpaca trick is amazing that it works. It's amazing that we're highly non-obvious that, you know, for GPT-J or even lLLaMA, you know, hundreds of billions of tokens into the train, this whisper of new data, you know, sort of moves it in, moves the parameter, uh, tensors into a new part of the state space. [00:08:02] I think, you know, my background is roughly in statistical physics related areas, and I think kind of like a phase transition. Mm-hmm. Like ice and water. It's like they're. Very, very little separates the two, but they could not be more different. And so Ali just kept haranging, like a huge email list of people. [00:08:21] Um, thousands and thousands of people. And, um, it worked. The other thing is, you know, to our employees credit, people see the moment and they wanna be part of something. And I think there's just passion and enthusiasm for. Doing this. So it was easier than you would expect [00:08:37] The answer is, so you put some answers in the blog post. [00:08:40] Yeah. And they're pretty comprehensive. Cuz one of the questions was like, how do I build a campfire? Yeah. And then the response was four paragraphs [00:08:46] of actual Truly, and I think Yeah, true. Yeah. And I think part of it is that because of the rapid adoption of these technologies like that, you have hundreds of millions of people, you know, who knows what the numbers are. [00:08:58] But on ChatGPT. People have become educated in terms of, and opinionated about what they expect from these tools. And so I think, you know, a lot of the answers are like, written in the style of what you would want from one of these assistants. And I think just to kind of like riff on how this question of like how the composition, cuz this is really re relevant to our enterprise customers, how the composition of the dataset qualitatively shapes the resulting behaviors of the fine-tuned models that are exposed to that stimulus. [00:09:28] Gamifying Instruction Tuning [00:09:28] You know, you look at a dataset like flan, which is a really, really large dataset that is, I think thousand plus tasks. Um, that's, you know, kind of this. Gold standard instruction data set, and a lot of it's synthesized the responses and we'll talk about evaluation, but the responses are very brief. You know, it's like emit the word positive or negative in relation to the, you know, as a judgment of the sentiment of this utterance. [00:09:52] And so it's, it's very multitask and I think like having thousands of different task types perform sort of irregular, you can't overfit to one specific behavior and so you have to compress and like do many things reasonably well. And so that I think you, you have to kind of wind up in interpolating between different types of behaviors that way. [00:10:12] But there's also like the question of like, when do you predict the end of sequence token? And if your completions, particularly for instruction tuning are short. Our empirical observation is that the fine tune model emits shorter results. And so having how to build a campfire. And like a narrative thoughtful human-like description. [00:10:36] I think it requires that demonstration to get that behavior from the model. And you had a, you had a leaderboard, um, who did [00:10:43] what, uh, any fun shenanigans that came out of, uh, the gamification? [00:10:46] Well, so the thing is like, you know, I think you can just ask people like be helpful. Uh, you know, like, like some people always take it too far and then Sure. [00:10:55] Yeah. Well, so you definitely see a long tail distribution. I think I was looking at the open assistant paper last night, and I think, I mean, don't quote me on this, but something like 12 people accounted for 10% of the total responses, which is super, that's just human systems have that long tail distribution terms of activity thing. [00:11:12] Yeah, yeah, exactly. So it's not surprising. And we see that to a some degree in our data set as well, but, um, not in the way that you would if you opened it up to the, like internet at large. So I, I think people are incentivized coworkers. Yeah. Do the right thing and you know, it's, you know, and also it's our company. [00:11:29] Like we. Want it to actually be useful, not just a performance of usefulness. And I think people got that. [00:11:36] Summarization - Thumbnails for Language [00:11:36] Is there a task [00:11:37] that you found like particularly hard to get data on? Like good data summarization? [00:11:41] Oh, because it's like a, it's both like long, uh, it's long and requires thought, you know, you have to synthesize and as opposed to name all the people in places in this passage from Wikipedia that's like, I can kind of do that while I'm watching television, but like writing an essay. [00:11:59] Yeah, it's a compare is hard. Yeah, there's probably more structure and like in terms of um, like an information theoretic standpoint, how much new signal each record introduces into the model. I expect that summarization is actually. A very demanding task and would not soon become overfit. We're developing our, our, I don't have like definitive answers to how that works because we're still, it's an open research project for the, for the business. [00:12:27] Yeah. Well, I, you know, just categorically, I think sum summarization is becoming more important, the more generative ai. For freights because we kind of need to expand and we see the contract again, in terms of what, uh, what we consume in terms of, uh, [00:12:41] information. Truly. I mean, like, to kind of riff on that, I think the, there's just so much material at your business. [00:12:48] You think about like, uh, PRDs, like, or, you know, product requirement stocks, you know, reasonable people. You kind of want like a zoom lens on language and you want the ability to see the high level structure of something and then be able to get details on demand like you would pan or like, you know, zoom into an information visualization. [00:13:09] I was talking with. Um, The head of AI at Notion about this and who, you know, you guys probably know and as a really remarkable person, and this idea of like, what does a thumbnail for language look like? Because like your visual cortex is structured such that like it's highly evolutionarily conserved to be able to glance at something and perceive its essence. [00:13:28] And that makes seeing a field of thumbnails. Like you guys I think are gonna speak with, um, Lexi folks here shortly. And you can see us like the field of images in response to a query and get a sense for like, oh, these are all like moody cyber punk scenes. Mm-hmm. What is that for language? And maybe it's like, maybe it doesn't exist. [00:13:52] Maybe it's the case. Stop me if I'm getting too far afield here. But you think about clothes as a technology that has shaped our physiology. Right. Like, and our, our phen, our phenotypic expression, we used to be covered in hair. We evolved this technology fire would also be in this class, and our bodies changed in response to it on the very long time scale of human history. [00:14:15] Mm-hmm. It may be the case that AI in the way that the visual cortex has been evolutionarily conserved to be able to rapidly perceive things, shapes how we process information. I don't know. What to do about language right now. It looks like reading a lot of samples from different models and seeing how they perform as we move through the loss curve. [00:14:34] That makes [00:14:34] sense. I mean, if you think about images in text, you don't really have like peripheral vision. You know, when you're like seeing something, you focus on the main thing and then you kind of like start to expand to see the rest. Yes. Like text is kind of like a, the density is like the same across the tax. [00:14:49] Like nothing jumps out when you see a wall of tax versus when you see an NI image. Just like something usually jumps out first. Yes. So I don't have the answer either. Was gonna say, I'm really curious word [00:14:58] clouds, which, but that, that's the thing is like, that's such a joke, right? Wait for me. Yeah, it's like punchline. [00:15:06] You must have [00:15:06] done, you know, your, your Twitter [00:15:08] work. I've cut a few word clouds in my day. [00:15:11] CICERO and Geopolitical AI Agents [00:15:11] Um, you know, I also think like this question of like, what are you most excited about in ai? Like what do you see as the sort of like grandest potential? And one of the things that I reflect on is, is the. Possibility of having agents that are able to, to negotiate intractable geopolitical problems. [00:15:31] So like if you look at like, the Cicero paper from, from Meta, can you recap for those who are making Yeah. So I mean it's, you know, I don't wanna like represent somebody else's work as like you're just talking Yeah, exactly. But like, um, my understanding is that diplomacy is a, um, turn-based negotiating game, like risk where you are all making the decision in simultaneously and you're trying to convince people that you're going to do or not do something. [00:15:56] And, uh, this paper was co-authored with one of the top diplomacy players and Meta built a system that was very, very capable at this negotiating game. I. Can envision nation states operating ais that find game theoretically optimal and sort of non exploitable steady states basically. Mm-hmm. That, you know, if you think about a lot of the large scale geopolitical disputes where it's just like human mediators are unable to find a compromise, ais may be able to satisfying conditions that you're like, yeah, actually I don't, that works for me. [00:16:36] Mm-hmm. And to your point about like how the phobia and attention generally, but like how the actual visual cortex works, the idea that like a great writer says something in a way and it hits unique structures in your brain and you have that chemical cascade, which is understanding, we may be able to design systems that compress very long documents on a per person basis so as to maximize information transfer, and maybe that's what the thumbnail looks like. [00:17:03] Mm-hmm. [00:17:04] Yeah, maybe it's emojis all the way down. I dunno. [00:17:08] Yeah. [00:17:09] Datasets vs Intentional Design [00:17:09] Obviously the dataset is like one of the, the big things in Dolly. Yeah. But you talked about some of these technologies being like discover, not designed, like maybe talk a bit about the process that took it to Dolly and like the experimentation [00:17:21] there. [00:17:22] So it's not my, my friend, my dear friend, Jacob Burk kind of had this insight, which is that AI is you, you design a jet turbine, like for sure you make a plan. Mm-hmm. And you, you know, have some working model of aerodynamics and you execute on the jet turbine. I think that with ai, generally we see. You know, this instruction following behavior that we saw in Dolly was not present in the, the base model. [00:17:53] It, you know, effectively will, it's a, you know, very powerful base model, but it will just complete the prefix as though it's random page on the internet. We had Databricks, but also the community with Alpaca discovered that you can perturb them just, just so, and get quite different behavior. That was not really a design. [00:18:13] I mean, it's designed in the sense that you had an intent and then you saw it happen. But we do not like choose the parameters they are arrived upon. And the question that I have is, what other capabilities are latent in these models, right? GPT-J was two years old. Can it do anything else? That's surprising? [00:18:36] Probably so, and I think you look at, you know, particularly, and this is why the Pithia Suite is so cool, is that, and you know, a ton of credit to, for. Having this vision, and I think it will probably take some time for the research community to, to understand what to do with these artifacts that they've created. [00:18:54] But it's effectively like this matrix of model checkpoints and sizes where you say, I'm gonna take from I think 110 million all the way up to 12 billion, which is what Dolly two is based on. And then at every checkpoint through the training run under, I think it's 2 million. Yeah. Tokens. Yeah. Well, so the, I think the Pithia suite is just trained on the pile, so it's like three, 400 million, which is probably undertrained. [00:19:18] And did you guys see this red? I think it's red Pajama released this morning. They've reproduced the lLLaMA training data set. So, so it's 1.2 trillion tokens and it's, um, I mean, you know, a separate topic, but we looked pretty hard at what it would take to reproduce the LLaMA data set. And it's like, Non-trivial. [00:19:35] I mean, bringing Common Crawl online and then d near de-duping it and you know, filtering it for quality. So the, the Common Crawl data set in LLLaMA is they fit a model to predict whether a page in common crawl is likely to be a reference on Wikipedia. And so that's like a way to like, I don't want lists of phone numbers, for example, or like ads. [00:19:58] All of that is a lot of work. And so anyway, with Pit, I think we can start to ask questions like through this, this matrix with size and like checkpoint depth. We have these different model parameters. How do behaviors emerge through that training process? And at different scales, you know, maybe it will be less of a discovery process. [00:20:22] Maybe we will get more intentional about, like, I want to elicit the fol, I want summarization, I want closed form, question answering. Those are the only things that matter to me. How much data do I need to. Generate or buy, how many parameters do I need to solve that compression problem? And maybe it will become much more deterministic, but right now it feels a lot like we're just trying things and seeing if it works, which is quite different from a lot of engineering disciplines. [00:20:51] I'm curious, does that reflect your experiences? Like Yeah, I [00:20:54] think like we had a whole episode on, um, kind of like scaling loss and everything with Varun from Exafunction. And I feel like the, when the Chinch paper came out, a lot of teams look at their work and they were like, we're just kind of throwing darts. [00:21:07] Exactly. That's now one, [00:21:10] 1.2 to, uh, 1.7 tokens, uh, you know, per, uh, per parameter. And, uh, now we're redoing everything with [00:21:16] 20 tokens. It's exciting, but also as like, you know, I'm, I'm a, an engineer and a hacker, like I'm not a scientist, but I, you know, used to pretend to be a scientist. Not, you know, not really pretend, but like I respect the, I respect the craft and like, It's also very exciting to have something you really don't understand that well, because that's an opportunity to create knowledge. [00:21:41] So that's part of why it's such an exciting time in the field. There's some work [00:21:44] Biological Basis of AI [00:21:44] on with, um, understanding the development of AI progress, uh, using biological basis. Mm-hmm. So in, in some sense, we're a speed running evolution Yeah. With training. Yeah. So in a sense that of just natural discovery of things and, and just kind of throwing epox at it Yeah. [00:22:02] Is, makes intuitive sense to me. But, uh, I do think that it is unintuitive to estimate how different artificial life might evolve differently [00:22:12] from biological life. Yeah. I, so like Richard Dawkins had, um, this sort of toy model called bio morphs. Which, uh, no, I haven't heard of it. Yeah, it's, I think it was dates to the eighties. [00:22:25] So it's a pretty old school demonstration of capabilities. But the idea is that you have, imagine they look, they're little insects that look like vector art. And the parameters of how they are rendered are governed by, you know, it's parametric, right? So some of them have long antennas and some of them have wide bodies and some of them have 10 legs, some of them have four legs. [00:22:46] And the underlying method is, is genetic algorithms where you take subsets of the parameters and kind of recombine them. And you're presented as a user with a three by three grid, and you click based on what you find subjectively beautiful. And so the fitness function, then they're re combined and you render a new set of nine by nine, some of which are mutated. [00:23:05] And so the fitness function is your perception of aesthetic beauty. That is the pressure from the environment. And I think like with things like RLHF where you're having this preference learning task, that is a little different from next token prediction in terms of like what is synthetic life and how are our preferences reflected there? [00:23:23] I think it's a very sort of interesting, yeah, interesting area. Okay. So a [00:23:27] Training Your Own LLMs [00:23:27] lot of people are very inspired by work with Dolly. Obviously Databricks, uh, is doing it. Partially out of the kindness of your hearts, but also to advertise Databricks capabilities. Uh, how should businesses who want to do the similar things for their own data sets and companies, uh, how, how should they think about [00:23:43] going about this? [00:23:44] I really would actually say that it's probably less about advertising our capabilities. I mean, that, you know, we're exercising our capabilities, but I, I really think that to the extent that we can help define some of the moves that reasonable teams would make in creating technologies like this, it, it helps everybody understand more clearly what needs to be done to make it useful and not just interesting. [00:24:08] And so, one, you know, one of the canonical examples that we had in the original Dolly was write a love letter, ed Growlin Poe. Yep. Which is super cool and like very moody. You know, I, I dunno if you guys remember the particulars of it, but it was like, I. The person, the imagined person writing this letter was like, I, I basically couldn't, like, I couldn't stand you, but I can't stop thinking about you, you know, which is a very like, gothic, uh, kinda, uh, mood in, in a letter like that not relevant to the enterprise context. [00:24:39] Right. So, you know, like it's neat that it does it, but if I don't have to buy training data that gets it to write moody, gothic letters to Edgar and Poe, and if I can be choosy about how I invest my token budget, that is useful to many businesses. And so, you know, one of the things that. We're trying to understand more clearly is I, we talked a little bit about like different tasks require that you compress in a way that generalizes, you know, if you think about it, the, the parameters as compressing language and also world knowledge. [00:25:15] The question is like, for a given model size, how many demonstrations of summarization, for example, are required in order to get a really useful, grounded QA bot? And so I think in building these kinds of solutions and sort of seeing how the. Categories of behaviors in the instruction tuning or sort of fine tuning data sets are related to those behaviors, I think will develop a playbook for startups in the enterprise that makes it, um, so that you can move with an economy of motion. [00:25:44] And this is related to evaluations as well. So one of the things that we had talked about sort of before we started recording was the using the EleutherAI evaluation benchmarks, and I think helm and the, you know, there's a bunch of other batteries that you can push your models through. But the metrics that we looked at first when we built the first version of Dolly, and this is on our hanging face page, you can go see this yourself. [00:26:08] The GPT-J model. And the fine-tuned dolly model have almost identical benchmark scores, but the qualitative character of the model just couldn't be any more different. And so I think that it requires better ways to measure the desired behavior, and especially in these enterprise contexts where it's like, is this a good summary and how can I determine that without asking a person? [00:26:37] And maybe it's kind of like you train reward bottles where you, you know, you have sort of a learned preferences and then you show, you know, you take kind of an active learning approach where you show the ones that it's most uncertain about to crowd workers and it's kind of like human in the loop. [00:26:52] Would this be p p o ish? [00:26:54] I mean, potential. That's, so this, that's not an area of expertise in mine yet. You know, this is something that we're also trying to, uh, more deeply understand kind of what the applicability of that stack is to, like, I'm just trying to ship. Mm-hmm. You know, my understanding is that that's somewhat challenging to bring online and also requires a fair number of labels. [00:27:14] And so it's like from an active learning standpoint, uh, my thinking would be more like, You have a reward model that you've trained and you said like, this is based on human judgments from my employees or some crowd workers, what I want from a summarization or a close, close form question answering. And then you basically, you choose new examples to show to humans that are close to the decision boundary and that are like maximally confusing. [00:27:38] It's like, I'm just really not sure rather than things that are far from the decision boundary. And it's, it's kind of like, I actually think there's gonna be, in terms of value creation in the next, let's say 18 to 36 months, there's still room for like old tricks. You know, like not everything has to be generative AI for it to be very valuable and very useful. [00:27:56] And maybe, maybe these models and, and zero shot prompting just eats everything. But it's probably the case that like an ensemble of techniques will be valuable and that you don't have to, you know, establish like room temperature fusion to like, you know, create value in the world, at least for, you know, another year and a half. [00:28:20] You know, like [00:28:21] You May Not Need a Large Model [00:28:21] just, just to spell it out for people trying to, uh, go deep on stuff. Um, maybe leave breadcrumbs. Um, sure. When you say techniques, you don't just mean prompting. [00:28:29] Oh, I mean even like named entity recognition, like Yeah, there's just like classic NLP stuff, you know, like supervised learning. I mean, multi-class classifi. [00:28:37] I have customer support tickets. I want to know whether this is going to be flagged as. P zero. Like that's just, it's not a complicated problem to solve, but it's still very valuable in these models that can deeply understand the essence of something and not necessarily generate language. But understand, I expect that you will see like s because, so for example, inference right now is time consuming. [00:29:04] Mm-hmm. Just, you know, it's like, unless you are really rigorous, and I think it, one of the things I'm excited about at Databricks is that we're, our inference stack is very, very fast. Like orders of magnitude faster than you would get if you took the naive approach. And that leads to very qualitative, like a very different way that you interact with these models. [00:29:22] You can explore more and understand their behavior more when it doesn't take 30 or 40 seconds to generate a sample and it's instead 1800 milliseconds. You know, that's something that's very exciting. But if you need to spend your compute budget, Efficiently and you have tens of thousands of possible things that you could summarize, but you can really only, you know, in a day do so many. [00:29:45] Having some stack ranking of them with a classical machine learning model is just valuable. And I, I expect that you'll see like an ecosystem of tools and that it's not all going to be necessarily agents talking to agents. I could be proven wrong on that. Like, I, I don't know. We'll see. Hey, [00:29:59] Good LLM Use cases [00:29:59] going back to the evolutionary point, I feel like people think that the generative AI piece is like the one with the most like, uh, possible branches of the tree still to explore. [00:30:09] So they're all focusing on that. But like you said, we're probably gonna stop at some point and be like, oh. That thing we were doing is just as good. Let's pair them together and like use that instead of just like trying to make this model do everything. [00:30:22] Yeah. And there, yeah, there are things like categorically that only generative models can accomplish. [00:30:28] And I do think, I mean, one of the reasons that at Databricks we see so much value for companies is that you can, with zero shot prompting, you can say, given this customer support ticket, for example, give me a summary of the key issues represented in it. And then simply by changing that prefix, say, write a thoughtfully composed reply that addresses these issues in the tone and voice of our company. [00:30:53] And imagine you have a model that's been fine tuned on the tone voice that's in your, in your, uh, from your support team. Both of those problems historically would've taken like a reasonable machine learning team, six to eight weeks to build. And frankly, the right, the response, I'm not sure you can do it without generative techniques. [00:31:13] And now your director of sales can do that. You know, and it's like, the thing that might make me look foolish in retrospect is that. Orders of magnitudes cheaper to do it with prompting. And maybe it's like, well, sure the inference costs are non-trivial, but it's just we've saved all of that in time. I don't know. [00:31:33] Dolly Cost $30 on Databricks [00:31:33] We'll see. I'm [00:31:34] always interested in, uh, more economics of, um, of these things. Uh, and one of the headline figures that you guys put out for Dolly was the $30 training cost. Yes. How did you get that number? Was it. Much lower than you expected and just let's just go as deep [00:31:50] as you want. Well, you just think about, so you know, we trained the original dolly on a 100 s and so one of the cool things about this is we're doing this all on Databricks clusters, right? [00:32:00] So this like, this works out of the box on Databricks and like turns out, you know, I think you would probably need slightly different configurations if you were going to do your own full pre-training run on, you know, trillions of tokens. You have to think about things like network interconnect and like placement groups in the data center in a more like opinionated way than you might for spark clusters. [00:32:23] But for multi-node distributed fine tuning, the Databricks stack is great out of the box. That was wonderful to find. [00:32:32] You've been building the perfect fine tuning architecture the whole [00:32:34] time. Yeah. You know, may, maybe it's not perfect yet, but like, It's pretty good. And I think, so for the original Dolly, it was just a single node, and so you can bring up an eight node, a 100 machine, and I'm, you know, I thinking of just the off the rack pricing from the cloud providers, it's about 30 bucks. [00:32:55] I think the actual number's probably less than $30. For How long are you for? It was less than an hour to train the thing. It's 50, I mean it's 50 thou alpacas, 50,000 records. Right. [00:33:04] And you've open sourced the, the notebook, which people can check out what [00:33:07] gonna show notes. There's. The risk that I am making this up is zero. [00:33:11] Yeah. No, no, no. I'm not, I'm [00:33:12] not saying the I know you're not. I'm just saying I'm, I'm, I'm leaving break rooms for people to say, Hey, it, it's 30 [00:33:17] bucks, takes an hour. Go do it. It's, it's crazy. And, and that's like the, I mean, you think about, I yeah, I, I, I know for a fact that you're not suggesting that, but it's just like, what's nuts is that you can just try it. [00:33:28] You know, you can, if you have 30 bucks, you can stand this thing up and, um, on a single machine, execute this training run. And I think I talked about like this idea that it's kind of like a phase transition. What's surprising about it, if you were to say, Hey, given a corpus of millions of instruction pairs, you can for. [00:33:50] $10,000, which is still an order of magnitude less than it cost to train the thing, get this qualitatively different behavior. I'd be like, yeah, that that sounds about right. And it's like, yeah, if you have an afternoon, like you can do this. That was not certainly, it was not obvious to me that that was true. [00:34:08] I think especially like, you know, like with libraries, like deep speed that, you know, so deep speed is a, is a library that gives you many different options for dealing with models that don't fit in memory and helping increase the effective batch size by, you know, for example, putting the entire model on a GP on several different GPUs and then having device local batches that are then the gradients are, are accumulated, are sort of aggregated for those, those from those different devices to get an effective batch or sharding the actual different model submodules across GPUs. [00:34:43] And this is all available in the notebook and the, the model that we train does not fit on a single device. And so you have to shard the model across the GPUs to run the training, you know, an incredible time that like this technology is just like free and open source and it's like the Microsoft team and the, you know, the hugging face team have made it so easy. [00:35:04] To accomplish things that even just two years ago really required a PhD. And so it's like level of effort, capital expenditure, substantially less than I would've expected. Yeah. [00:35:17] And you, you sort of co-evolve this cuz you also happen to work on the infrastructure optimization [00:35:21] team. Yeah, I mean that's kind of, um, like, you know, this is really kind of a separate project at Databricks, which is like making sure that we have a great customer experience and that we have the resources that are required for all of our customers. [00:35:37] You can push a button, get a computer, uh, get a Spark cluster. And I think when you look to a world where everybody is using GPUs on Databricks, making sure that we are running as efficiently as possible so that we can make Databricks a place that is extremely cost effective to train and operate these models. [00:35:55] I think you have to solve both problems simultaneously. And I think the company that does that effectively is, um, is gonna create a lot of value for the market. [00:36:06] Databricks Open Source [00:36:06] Yeah. You mentioned Spark, obviously Databricks, you know, Started, like the founders of Databricks created a spark. Yeah. At Berkeley. Then, you know, from an open source project, you start thinking about the enterprise use cases. [00:36:18] You end up building a whole platform. Yeah. You still had a lot of great open source projects like uh, ML Flow, Delta Lakes. Yeah. Um, yeah. Things like that. How are you thinking about that was kind of the ML ops phase. Yeah. Right. As you think about the l lm ops, like needs, you know, like obviously. We can think of some of these models as the spark, so to speak, of this new generation. [00:36:39] Like what are some of the things that you see needed in infrastructure and that maybe you're thinking about building? [00:36:44] Yeah, I mean, um, so kind of first to address this, this matter of open source. I think, you know, Databricks has done a lot of things that, and has released into the public domain a lot of technologies where a reasonable person could have said, you should. [00:37:00] Treat that as IP that you and no one else has. And I think time and again, the story has been more, is better and we all succeed together. And when you create a new class, people rush in to fill it with ideas and use cases and that it's, it's really powerful. It's both good business and it's good for the community. [00:37:21] And Dolly I think is very much a natural extension of that urge, which just, I think reflects our founders tastes and beliefs about markets and, and technology [00:37:31] LLMOps and Prompt Tooling [00:37:31] when it comes to LM ops, which is not a phrase that rolls off the tongue. We'll, we're gonna need something better than that. We, this kinda gets back to like what is a thumbnail for text. [00:37:43] Mm-hmm. One of the things that my team winds up doing a fair amount of right now is like slacking back and forth examples of like generated samples. Okay. Because like these evaluation benchmarks do not capture the behaviors of interest. And so we often have like a reference battery of prompts. Let's say 50 to a hundred. [00:38:03] Write a love letter to Edgar and Poe. Yeah. Give me a list of ins. Like what are, what are one of our things is what are considerations? Like it should keep in mind when planning for a backcountry backpacking trip can you generate a list of reasonable suggestions for a backpacking trip. And you see, as you kind of move the model through the loss curve under instruction tuning that um, that behavior emerges and that like you kind of wind up qualitatively evaluating is the model doing what I want in respect to these prompts that I've seen many different models answer this model or this, this instruction tuning data set is generating shorter completions. [00:38:40] This one is generating the. Wackier completions, you know, this one is much likelier to produce lists all of these things. I don't know if you've seen Nat Devrel. Mm-hmm. I'm sure, of course you have that idea of the grid of like, I want to run inference in parallel on arbitrary prompts and compare and contrast, like tooling like that is going to make it, and especially with a fast inference layer, and this is where I think Databricks has a lot of opportunity to create value for people is being able to serve, interact, and measure the behavior of the model as it changes over time and subject it not only to quantitative. [00:39:19] Benchmarks, but also qualitative subjective benchmarks plus human in the loop feedback where imagine that I burn a model checkpoint and every thousand steps, I send it off to an annotation team and I get a hundred pieces of human feedback on the results. And it's like there's kind of like what is the right volume of human feedback to get to statistical significance? [00:39:43] But I think there is. An ensemble, you know, each of these is like a different perspective on the behavior of the model. A quantitative, qualitative, and then human, uh, feedback at scale. Somebody's going to build a product that does these things well in a delightful user form factor. And that is fast and um, addresses the specific needs of AI developers. [00:40:04] And I think that business will be very successful and I would like for it to be Databricks. Ah, okay. [00:40:10] Teasing what you might be [00:40:11] building. Interesting. You know, and this, not to make forward-looking statements, but it's just like, make sense as obvious as a person, you wanna do it? Mm-hmm. I need that. Yeah. [00:40:19] Yeah. I need that. Yeah. I happen to work at a company. [00:40:21] Yeah. So just to push on, uh, uh, this one a little bit, cuz I have spent some time looking into this. Sure. Have you come across prompt layer? That would be one of the leading tools. And then I think Human Loop has a little bit of it, but yes, it's not a course focus of theirs, is it? [00:40:34] Prompt layer? Yeah. I'll, okay. Send And happy to drop that reference cuz uh, he has reached out to me and I, I looked at his demo video and it, yeah, it kind of is, isn't that in the ballpark? And I think there are a lot of people, uh, zeroing in on it. But the reason I have not done anything in, in, in this area at all is because I could just do it in a spreadsheet. [00:40:51] Like all you need to do is Yeah. [00:40:53] Spreadsheet function that you can, but I mean like editing text and Google Sheets is a drag. Is it? Yeah. I, I mean mm-hmm. What's missing? You know? Oh, so a, like the text editing experience in it, like you're trying to wrap these cells. Okay. And so now you gotta like double click to get into the editing mode. [00:41:12] I think they struggle with large record sets. So like the spreadsheets slow down, you kind of want, this is not some, like a, this specific question of like, how does Google Sheets fail to meet the need is something that, you know, I don't have a talk track around Sure. But like linking it to an underlying data source where it's sort of like persisted. [00:41:34] Cuz now I'm, now I have a bunch of spreadsheets that I'm managing and it's like, those live on in Google Drive, which has kind of a garbage ui. Or is it on my local machine? Am I sending those around? Like, if, can I lock the records so that they can't be annotated later? How do I collect multiple evaluations from different people? [00:41:50] How do I compute summary statistics across those evaluations? Listen, I'm the first person to like, fire up sublime. Yeah. You know, like, keep it simple, right? Yeah. Just for sure. I feel like the, the way that I have talked with colleagues about it is it's like we are emailing around. Photocopies of signed printouts of PDFs and DocuSign doesn't exist yet, and nobody realizes that they're doing this like ridiculous dance. [00:42:16] And I get it. I too have used Google Sheets to solve this problem, and I believe that they're, there's maybe something better. I've Stockholm Syndrome. [00:42:26] "I'm a Sheets Maxi" [00:42:26] So there's a couple more that I would highlight, uh, which is Quadra. Uh, okay. Uh, full disclosure, an investment of mine, but basically Google Sheets implement, implemented a web assembly. [00:42:35] Yeah. And a, and a canvas. Okay. And it speaks Python and sql. Yeah. Yeah. And, uh, and Scala. Yeah. Uh, so I, I think, I think, yeah, there, there's some people working on interesting hearings [00:42:46] at those. And what you could do is like, like imagine that you have a Google Sheets type ui, the ability to select like a column or a range and subject all of those values to a prompt. [00:42:59] Yes. And like say like, I have template filling and I want, that's what I want. My problem [00:43:04] with most other SaaS attempts is people tend to build UIs that get in your way of just free range experimentation. Yes. And I'm a sheet's, uh, maxi. Like if I can do it in a sheet, I'll do [00:43:16] in a sheet, you know? Yeah. Well, and I mean, kind of to continue, like on the sheets, sort of mining that vein, you know, on the, sort of like how does AI impact the workplace and like human productivity? [00:43:29] I think like a, I really like the metaphor, which is comparing, uh, AI technologies to the development, the advent of spreadsheets in the eighties, and this idea that like you had a lot of professionals who were like well educated, like serious people doing serious accounting and finance work, who saw as their kind of core job function manually calculating. [00:43:53] Values in forecasts on paper as like, this is how I create value for the business. And spreadsheets came along and I think. There was a lot of concern that like, what am I gonna do? Yeah. With my days? And it turns out that like I think of it sometimes, like being in a warm bath and you don't notice how nice the water is until you wiggle your toes a little bit. [00:44:14] You kind of get used to your circumstances and you stop noticing the things that would stand out. [00:44:19] AI and Workplace Productivity [00:44:19] So on the subject of how artificial intelligence technologies will shape productivity in the workplace, you have, I think, a good metaphor in comparing this to spreadsheets and the Adventist spreadsheets In the eighties, I think you had a lot of really serious people who were taking, making an earnest effort to be as productive and effective as possible in their lives, who were not making it their business to waste time. [00:44:42] Saw spreadsheet technology come out and it's like, man, well what am I gonna do? I'm the person that calculates things. Like I write it all down and that's how I create value. And then like you start using this new tool and it's like, oh, it turns out that was the Ted most tedious and least rewarding part of my job. [00:44:58] And I'm just so, you know, like I have, like, I still have that human drive to create. You just kind of point it at like more pressing and important problems. And I think that, that we probably don't, especially, and even when it comes to writing, which feels like a very like quintessentially human and creative act, there's a lot of just formulaic writing that you have to do. [00:45:22] Oh yeah. And it's like, maybe I shouldn't be spending my time on all of that kind of boiler plate. And, you know, there's a question of like, should we be spending our time reading boilerplate? And if so, why is there so much boiler plate? But I, I think that humans are incredibly resourceful and incredibly perceptive as to how they can be effective. [00:45:43] And that, you know, the, I think it will free us up to do much more useful things with our time. I think right now [00:45:50] there's still a, a bit of a stigma around, you know, you're using the model mm-hmm. To generate some of the text. But I built a open source, like a email drafter. Yeah. So for all of my emails, I get a G PT four pre-draft response. [00:46:04] And a lot of them I just sent, but now I'm still pretending to be me. [00:46:07] Okay. So that's why I'm talking to you [00:46:09] When I talk to you, you need to fine tune it. Right. [00:46:12] But in the future, maybe it's just gonna be acceptable that it's like, Hey, we don't actually need to spend this time, you and I talking. Yes. It's like, let the agents like cash it out and then come back to us and say, this [00:46:22] is what you're gonna do next. [00:46:23] Articulate your preferences and then you, I think this like trustworthiness is a piece of this here where like hallucinations, T b D, whether it is like actually attractable problem or whether you need other affordances like grounded methods to, to sort of. Is a hallucination, just a form of creativity, like, we'll see. [00:46:42] But um, I do think eventually we'll get to a point where we can, we trust these things to act on our behalf. And that scenario of like calendaring, for example, or just like, you know, even working out contract details, it's like, Just let me tell you exactly what I want and you make sure that you faithfully represent my interests. [00:47:00] That'll be really powerful. [00:47:02] OpenAssistant [00:47:02] So we haven't run this by you, but uh, I think you have a lot of opinions about, you know, the projects that are out there, uh mm-hmm. And three that are, are on mine. For one, you've already mentioned Open Assistant two, cereus, G B T also came out roughly in the same timeframe. I'm not sure if you want to comment on it, I'd like to compare because they, they also had a similar starting point as as you guys, and then three Red Pajama, which, uh, was just out this morning. [00:47:24] Yeah. We might, as might as well get a soundbite from you on your thoughts. So yeah, if you want to pick one, what was the first one? Uh, open Assistant. [00:47:30] Yeah. So, I mean, open Assistant is awesome. I love what they've done. I will be eager to use their free and open data set, uh, to improve the quality of Dolly three. [00:47:41] CerebrasGPT [00:47:41] Yeah, but also just like we're seeing the, the training is, so Cerus is a good example of, you know, I think they were, my understanding, and I don't know that team or really, you know, I haven't looked too closely at the technology, but I have worked with the model is that it's a demonstration of their capabilities on this unique chip that they've designed where they don't have to federate the models out to multiple cards. [00:48:04] But I think if you look at some of the benchmarks, it is on par or maybe a little shy of some of the Ethe I models. And I think that one of the things that you may see here is that the market for foundation models and like the importance of having your own foundation model is actually not that great. [00:48:27] That like you have a few. Core trains that people, I think of these kind of like stem cells where, you know, a stem cell is a piece of is, is a cell that can become more like its surrounding context. It can become anything upon differentiation when it's exposed to eye tissue or kidney tissue. These foundation models sort of are archetypal and then under fine tuning become the specific agent that you have a desire for. [00:48:53] And so I think they're expensive to train. They take a long time to train. Even with thousands of GPUs, I think you're still looking at like a month to stand up some of these really big models, and that's assuming everything goes correctly. And so what Open Assistant is doing is. I think representative of the next stage, which is like open data sets, and that's what the Dolly release is also about, is, I kind of think of it like an upgrade in a video game. [00:49:21] I don't play a ton of video games, but I, you know, I, I used to, and I'm familiar with the concept of like, your character can now double jump. Mm-hmm. Right. Great. You know, it's like, here's a data set that gives it the ability to talk to you. Hmm. Here's a data set that gives it the ability to answer questions over passages from a vector index. [00:49:38] I think anybody who's listening, I think there's a tremendous opportunity to create a lot of value for people by going through this exercise of the unsexy work, of just writing it down and figuring out ways to do that at scale. Some of that looks like semi-synthetic methods, so something I would love to see from the Dolly data set. [00:49:58] Is paraphrasing of all the prompts. So basically you now have multiple different ways of saying the same thing and you have the completions which are correct answers to different variants of the question. I think that will act as like a regular, it's kind of like image augmentation. I was gonna say, you flip it. [00:50:13] Yeah. Yeah. I believe that that will work for language. Like one of the things you could do. Cause we, we saw that within 24 hours the dataset had been translated into Spanish and Japanese. The dolly dataset. Yeah, it was, I mean, you know, it's maybe, yeah. Yeah. Right. Yeah. So that's super cool. Um, and also something that is only possible with open data. [00:50:31] Well, it's only useful with open data, but I just last night was thinking like, I wonder if you could to paraphrase, cuz it's not obvious to me like what the best and state of the most state-of-the-art paraphrasing model is. You could use Google Translate potentially and take the prompt. Translate it to Spanish and then translate it back to English, you get a slightly different way of saying the same thing. [00:50:54] Ah, right. So I think the self instruct paper is really about like few shot prompting to get more prompts and then using large models to get completions and then using human annotators to judge or train a reward model. I think that bootstrapping loop on the back of these open data sets is going to create multimillion scale training corpuses. [00:51:14] And so I, what Open Assistant has done is a, it's a great model. I don't know if you've tried their interactive chat, but it's just really quite an impressive accomplishment. But that the gesture towards open data that you know, the Dolly dataset and the open assistant dataset represent, I think is probably gonna define the next six to nine months of. [00:51:35] RedPajama [00:51:35] Work in this space. Um, and then the red, a red pajama. Red pajama, I mean, yeah, it's like I said, you can do a close read of the LLaMA paper. There's the dataset section and I think they use seven distinct data sets, archive, and I think maybe Stack exchange and common crawl. [00:51:50] Okay. So they have common crawl. [00:51:52] Yep. C4, which is Common crawl, but filtered subset. Yeah. Uh, GitHub archive books. Wikipedia Stack Exchange. [00:51:59] Yes. So, you know, take Common Crawl, for example, when you read the lLLaMA paper. So a common crawl I think is three terabytes in the lLLaMA paper. It's not something you just download from, like it's, you have to produce this data set, or at least the CC net, um, implementation that they reference there. [00:52:18] And you have like a single paragraph in this research paper dedicated to how they produce Common Crawl and they do near de-duplication. They train a model to predict whether something is likely to be a link, a reference link on Wikipedia. And there's just a bunch of other stuff that. Not only from like a, where do you get the model to predict whether something is a link as a reference on Wikipedia when you train it and then like where's your cut point? [00:52:41] You know, now you have kinda this precision recall trade off and it's like those decisions have material impacts on the quality and the character of the model that you learn. But also just from a scale standpoint, like building Common Crawl locally requires like a non-trivial distributed systems left. [00:52:59] And so I think Red Pajama is, and I think it's Mila and Chris Ray's lab hazy research, I think, or at least he's attached and together and I think together is kind of leading. There's a bunch of great teams behind that and so I have no reason to think they didn't do. The hard, difficult work correctly. [00:53:21] Yeah. And now is this major piece of the lift if you're wanting to do a lLLaMA repro in public. And I think that's would very naturally be the next step. And I would be kind of surprised if a train was not currently underway. Everybody agrees. LLLaMA is very, very strong. Also, we agree that it is not open incentives for somebody to spend a couple million bucks and produce it and then be the team that opened this architecture is, are quite high. [00:53:50] Mm-hmm. So I, I think in the next, you know, you asked for like predictions. I think we're five months at most away from a open LLaMA clone that is as high quality as, as what meta is produced. I will be disappointed if that's not the case. [00:54:07] Why Dolly > OpenAI GPT [00:54:07] And I think like there's the big distinction between what is open and what is like, Open in a way that is commercially usable. [00:54:13] Yeah. After that, I know the Dolly two post, you mentioned that you had a lot of inbound with Dolly. Yeah. 1.0. But a lot of businesses could not use it. Yeah. Because of where the data training data came from. Yes. What are some of the use cases that people have? There is, uh, a lot of it kind of like talking to your data. [00:54:30] Are there like, uh, other things that are maybe people are not thinking about using it for? [00:54:34] Yeah, so I mean, we have a number of customers who have reached out with really concrete use cases around customer support ticket resolution. One of the things that a lot of business open AI's models are incredibly powerful, and Databricks wants to be a business where you can use the right tool for the job. [00:54:55] Like if you have information from the public web, let's say you have forum posts, right, that you need to synthesize and process, that's just not sensitive information. You should be able to use truly whatever model. That might be a fine-tuned model that is like laser focused on your problem. It might be a general instruction following model and, and sort of whatever kind of intelligence GPT4 is, it's, you know, it's quite powerful. [00:55:20] You should be able to use those tools. There are definitely use cases in the enterprise where it's like, I either just, I'm not interested in sharing this ip. You know, these are effectively our state secrets. Or from a regulatory and compliance standpoint. I just can't send this data to a third party sub-process or something. [00:55:38] Even as quotidian is like, I just really don't want to go through procurement on this. You know, like it's kind of around those, um, I have some reasons to keep this in house. A lot of use cases like that and that, you know, I'm not a lawyer and so I won't speculate on the sort of actual licensing considerations or the actual obligations, but it's just like people like to be able to move confidently and what we've done with Dolly is make it super clear. [00:56:09] This model and this data set are licensed for commercial use. You can build a business on the back of this. And that, I think is a big part of why the response has been so positive. [00:56:19] Open Source Licensing for AI Models [00:56:19] Hugging face has, uh, the rail license responsible, um mm-hmm. AI license, which isn't recognized as open source yet. So that was the whole problem with stable diffusion, that it's just unclear cuz this, this is completely new license that is, uh, unproven. [00:56:32] But I just find it interesting that the existing open source licensing regime is mostly around code. And right now, you know, the, the value has shifted from code to the waits. [00:56:43] Yes. I think we can go on a three hour rant about the open source initiative and like who decides what an open source license is. [00:56:51] But I think there's a, I think the approach of like, hey, We know what commercial uses. Like this is good for it. Yes, it's good. You're not gonna have to worry about us suing you. It's like, you know, the semantics of it. Clear is always better. Exactly. It's like we don't need to be approved by the osi. Yeah. [00:57:07] You're gonna be okay. Just [00:57:09] Why Open Source Models? [00:57:09] to kind of like continue, like why open source? Yeah. I think that like it is with many eyes, all bugs are shallow. I think the reality is that like we do not know what the challenges we face with AI systems will be. Mm-hmm. And that the likelihood that we can get it a representative and comprehensive solution to the challenges they present by putting it in public and creating research artifacts that people who deal with ethics bias, ai, safety, security, these really sort of thorny issues, that they can take a hard look at how the actual thing is built and how it works and study it comprehensively rather than, Hey, we've got a team for that. [00:57:50] You're gonna mm-hmm. Just, you're just, we're just gonna need you to trust our work. I think I wanna be in that the former future rather than sort of like, I, I hope that people have done this correctly. I hope that this is somebody is taking care of this. [00:58:05] Moving Models [00:58:05] When people [00:58:06] evaluate this, how do you think about moving between models? [00:58:10] You know, obviously we talked about how the data set kind of shapes how the model behaves. Hmm. There's obviously people that might start on open AI and now they wanna try dollies. Yeah. Like what are some of the infrastructure there that maybe needs to be built to allow people to move their prompts from model to model? [00:58:26] Like to figure out, uh, how that works. [00:58:28] That's really interesting. Um, because you see even like moving between GPT3.5 and GPT4 that the behavior, like some things that were not possible on three five are No, I mean, many, many things that were not possible on three five are not possible on four, but you kind of want like slightly different problem formula, like slightly different prompt formulations or. [00:58:51] It's kind of like you want regression tests for prompts, and you could see like an automated system, which is uh, helps design a prompt such that the output of this new model is isomorphic to the outputs of the previous model. And sort of like using a language model to iterate on the prompt. So it just kind of evolves it to like adapt to the new model. [00:59:13] I have two beautiful boys who are, they're just incredible humans and my friend Ben and I built them a, an interactive choose your own adventure storytelling book that uses ChatGPT to generate stories and then options within those stories, and then uses open AI's image generation model Dolly to illustrate. [00:59:36] Those options. And then the kids can kind of choose their way through these stories. And the thing that you really like when you start to really push these things for more than just like single turn prompt response and I'm, I'm, you know, it's fine if it's language and you really need it to be like an api. [00:59:52] Is that like 19 times out, 20 it's like an p i and then the 20th generation. It's like just a totally different format. And he just like, you really like try to in the system prompt really seriously. I just only want you to give me three options. Yeah. And letter A, B, C, you know, I think that from a regression test standpoint, how do you know, like if I run this prompt a hundred times does a hundred out of a hun, does it come back a hundred out of a hundred in the format and sort of character that I require? [01:00:21] That's not something a person can really do effectively, and so I think you do need sort of model meta models that judge the outputs and that manage those migrations. Mm-hmm. Yeah, so I had, that's an interesting. Product class. I hadn't thought about it too much. Yeah. [01:00:34] Learning in a Simulation [01:00:34] When you mentioned before the example of the, you know, back country trip, I was like, yeah, it would be so cool if you had a, like a simulation where like, okay, this is the list you had. [01:00:44] Now I have this game where like I'm putting a character with that inventory and see if they survive in the back country. Cause you can like, you know, the first time I went to Yellowstone to camp, I forgot to pack like a fly for my tent and obviously it rained. That's because, you know, you get punished [01:00:58] right away. [01:00:59] Yeah. That's the environment providing you with a gradient. Exactly. Update your model eight. You should be grateful to have such an excellent Yeah. Mini [01:01:06] these models like the, the evolutionary piece that is missing is like, these models cannot. Die. They cannot break a arm. They cannot, when they make suggestions, like they don't actually Yeah. [01:01:16] Have any repercussion on them. Um, so I'm really curious if in the future, you know, okay, you wanna make a poem, uh, you know, I love poem. Now we're gonna send this structural people. Yeah. And if you get rejected, your model's gonna [01:01:28] Why Model Reflexion and Self Criticism Works [01:01:28] die. So I think like one of the things that's cool about Lang Chain, for example, we all know they're doing awesome work and building useful tools, but these models can tell if they're wrong. [01:01:38] So you can, like, you can ask a model to generate an utterance. And that next token prediction loss function may not capture. You may hallucinate something, you may make something up, but then you can show that generation to the same model. And ask it to tell you if it's correct or not. And it can, it can recognize that it's not, and I think that is a directly a function of the attention weights and that you can attend to the entire. [01:02:03] Whereas like for next token prediction, all I can see is the prefix and I'm just trying to choose and choosing sarcastically. Right. You're f frequently, like it's a weighted sample from the distribution over that soft softmax output vector, which does not have any. Reference to factuality, but when you resubmit to the model and you give it like, here's the entire generated passage, judge it in its completeness. [01:02:25] Well now I can attend to all of the token simultaneously, and it's just a much, much easier problem to solve. And so I think that like, oh, that's a cool insight. Yeah. Yeah. I mean it's, yeah. It's just, this is reflection. Yeah. You, you can just see what you said and like the model may contain enough information to judge it. [01:02:41] And so it's kind of like subject your plan mm-hmm. To an environment and see how it performs. I think like you could probably ask the model, I mean, we can try this today. Here's my plan for a trip. Critique it. Mm-hmm. Right? Like, what are, what are the things that could go wrong with this inventory? And I think that there's one scenario, there's one trajectory for this class of technologies, which would be like self-reflexive models where it is not super linear. [01:03:10] You don't get anything more than what is already contained in the models, and you just kind of saturate and it's like, okay, you need human feedback. There's another scenario, which is the alpha go scenario where models can play themselves and in observing their behavior and interactions they. Get stronger and better and more capable. [01:03:31] That's a much more interesting scenario and this idea that like in considering the entire generated sample, I have more insight than just when I'm sampling the next token. Mm-hmm. Suggests that there may. Be that escape potential in terms of getting super, you know, unsaturated returns on quality. [01:03:51] Lightning Round [01:03:51] Yeah, this was great, Mike kind of we're where a time, maybe we can jump into landing ground next. [01:03:55] We'll read you the questions again. Okay. If you wanna think about it. So, okay. Favorite AI [01:04:00] product? This is a boring answer, but it's true. Google Maps. Ah. And it's, how is it AI A, they're recently doing stuff with Nerf so that you can using Yeah. Multiple different photos. You can explore the interior of a business. [01:04:15] They are also undoubtedly, I mean like, I don't know the team at Google doing this, but digesting the sum total of human knowledge about each entity in their graph to like process that language and make judgements about what is this business? And listen, it's not an AI product, but it is a machine learning product categorically, and it's also an amazing product. [01:04:37] You forget how much you use it. I was at the coffee shop around the corner. I used it to figure out where to come. It was literally 150 meter walk, you know, it's just like that reflexive, but it's also from a, an information visualization. So I love maps. Mm-hmm. I opened our conversation saying that I think a lot about maps, that it is adaptive at multiple scales and will corson and refine the, the information that's displayed requires many, many judgements to be made and sim simultaneously about what is relevant and it's personalized. [01:05:08] It will take your intent. Are you driving? Okay, well show me parking garages preferentially. So it's very adaptive in such subtle ways that we don't notice it. And I think that's like great product design is like good editing. You don't notice it when it's good. Mm-hmm. And so I think Google Maps is an incredible AI ml. [01:05:28] Product accomplishment. Google Maps. Yeah. It's a great pick. Great. Well, and they need the help. Yeah. [01:05:36] It is actually the best ad uh, real estate, right? Like, there should be a ton of people buying ads specifically on Google Maps. Yeah. So they just show up and I, I don't know how big that business is, but it's gotta be huge. [01:05:45] Yeah. And, and then my subsequent thing is like, there should be Google Maps optimization, where you would name your business like Best Barbershop and it would show up as Best Barbershop when you look at it. Yeah, [01:05:55] of course. Right? Yeah. It's like AAA lock picks. Yeah. Right at the front of the Yellow Pages. [01:06:01] Favorite [01:06:01] AI people and communities you wanna shout out? [01:06:03] You know, I don't think that I have necessarily anything super original to say on this front. Um, The best of my understanding, this is an all volunteer effort and it's, you know, incredible what they have been able to accomplish. And it's like kind of in the constellation of projects. [01:06:20] You know, the additionally, I think these are what you would say and answer in response to this question, I think like the hugging face group is, it's kind of like Google Maps in a way, in the sense that you like, forget how complicated the thing that it's doing is, and I think they have. You see like specific people, I was thinking of STAs STAs, who works on the, works on a lot of the deep speed stuff, just super conscientious and like engaged with the community and like that the entire team at Hugging face is incredible and you know, they, you know, have made a lot of what is happening possible in the industry at large. [01:06:53] And so, um, and I think, yeah, this is like the power of open source ultimately Transformers, library, diffusers, all of it. It's just great. It's a great, it's a delightful product experience. [01:07:03] I think a lot of people, like I had, I once had hugging Face explained to me as Free, get LFS hosting. And I think they've, uh, they've moved beyond that in, in [01:07:11] recent years. [01:07:11] Yeah. A bit. Yeah. It's, it's quite strong work. Yeah. [01:07:14] Yeah. A year from now, what will people be the most surprised by in ai? You already [01:07:19] hinted [01:07:19] at? Uh, yeah, but I think that's not, like, I think that won't be surprising, I think as we're on a ballistic trajectory to having like a, an open lLLaMA reproduction. So here's something I think that will happen that we are not, like socially, we don't have a lot of priors for how to deal with, so this ghost writer track just came out this Kanye West Weekend. [01:07:40] Mm-hmm. AI collaboration. He has thoughts, Drake? Yeah. His thoughts. It's not really, Dave has thoughts. It's not really like, I, I like a different breed of hiphop, but like, it's. For an example of the class, it's like that does sound like a thing I might hear on the radio. So there's a world in, so skip flag was this knowledge graph that's builds itself from your workplace communication. [01:08:02] Think about all of the times that you have expressed your position and intent around a given topic in workplace communication or on the internet at large. I think like character AI is going in this direction where you're going to be able to talk to high fidelity avatars that represent the beliefs and intents of people around you, and that it will be both useful and convincing. [01:08:27] I don't know that like society has good models for how to sort of adapt to that existing and that it will, I suspect just on the basis of like what people are doing. Happened rather quickly at first. [01:08:41] Listen, you can definitely tell it's really good. Mm-hmm. I'm really curious what the long-term results are gonna be, because once you listen it once or twice, you can tell that it's like, it's not really like a coherent song kind of written. [01:08:55] But to me that the funniest thing is that actually, so Drake and the Weekend that never made a song together again because they kinda had a, a follow up between then and, and the Weekend at One song where he said, if you made me then replace me. Because Drake basically hinting that like if he didn't put the weekend on his album, he would've never become popular. [01:09:13] Okay. So it's funny that now there's like this AI generated song from the weekend. It just kind of puts the, you know, if you made me then replace me line in in a different context. But I think this will be super interesting for the labels, you know, like a lot of them do on the Masters to a lot of this music they do on, yeah. [01:09:31] A lot of rides. So, At some point, it's much easier to generate music this way than to do it in person. But I still think you need the artist touch. [01:09:39] Just like what is it that is unique and what, you know. I think artists frequently, you know, I, I know in my own writing and sort of like creative process, you sometimes feel like you're just going through the motions. [01:09:50] And it's funny how we have ways of talking about a phrase rolls off the tongue. That's very much like a causal language model. Mm-hmm. Where like we talk about talk tracks. I have a whole spiel, you know, you talk to a startup founder and you're like, oh my God, how many times have you said like, very close, like very tight variance on this Three minutes sometimes. [01:10:10] That's good. Yeah. It's, it's fine. It's just, it's a thing that we do. And so touching on this idea that like some of what we consider creative acts may not actually be creative acts and sort of, is there a pr, is there a market pressure to favor things that are truly creative versus just like formulaic and like re like rehashing kind of the same essence? [01:10:29] I think like art. Transcends boundaries is often the most interesting art to engage with, where it, it truly does confront you with something you haven't considered before. I hope that that's the place where humans play. And that they're kind of like, oh, I just need some lo-fi study beats. It's like, just gimme an infinite stream. [01:10:49] I'm fine. Because I'm just like, [01:10:52] you've seen that chart of like pop uh, songs, declining interns of the key changes, key changes in [01:10:58] Octa ranges. Completely. Completely. And like, I mean, we used to have [01:11:02] Bohemian Rhapsody and, and [01:11:03] yeah, it's a great example of something that would not be priced appropriately. [01:11:08] This is why I, I think perplexity AI is just very well named because we want more perplexity in our lives. Yes, by the way, shout out for replica ai. I don't know if you've come across them, but Absolutely. They are working on the digital twin stuff. Okay. Ai, uh, request for startups. AI thing you would pay for if someone [01:11:21] built it. [01:11:22] Well, so the LM op stuff for sure. Just like make it easy to generate and evaluate samples using multimodal, multimodal, I mean multiple modalities, not images and texts, but rather like humans, quantitative benchmarks and qualitative Oh, samples that I, I am able to evaluate myself, but other AI startups. I think that we have your sister, your wife, your wife has family that works in the park system. [01:11:49] Mm-hmm. Because it is so everybody has access to effectively the same information about what's interesting in the outdoors. I think you get to a lot of trail heads and you have very, very tight parking lots and it's difficult to get to a lot of these beautiful places. And like, um, mere Woods is another example of like, you gotta reserve a parking spot in the woods that's a plumber. [01:12:12] But I think that the US in particular is so unique in that we have such an expansive public lands, and I think that there are a lot of really majestic and beautiful places in the world that are not written about. And so I think from a geospatial standpoint, you could imagine representing each tile on a map like a word deve. [01:12:39] Embedding where you look at the context in which a location exists and the things people have said about it, and you, you kind of distill the essence of a place and you can given a statement about how I wanna spend my day route traffic more evenly. On the surface of the earth so that we are not all competing for the same fixed pool of resources. [01:13:03] I don't know that that's something really that's monetizable in like a, you know, is this gonna be the next 10 billion business sort of way. But like there's so much public land and there's so many back roads and like the days where I have, you know, rumbling down a dirt road, my brother are just the best days of my life. [01:13:22] And, uh, I want more of those. I want systems that help us live as fully as possible as humans. Yeah, there's definitely [01:13:29] a lot of, you know, you got the. The most popular trails. Everybody wants to be there. Yeah. And then there's the less known ones. And I feel like a lot of people back to the text to back is like, they don't know what they're gonna find, you know? [01:13:41] Mm-hmm. There's not like YouTube reviews of all these trails. Totally. But like you can see it. So I think a way to, to better understand that would be, would be cool. [01:13:49] I mean, just to kind of like riff on this just a little more and we can wrap, like I do think there's a AI technology as a swarm management. [01:13:59] Tool, you know, being able to perceive sensor and camera inputs from multiple different agents in a system. And I think about like ultra low powered gliders as an example of like, I would like to be able to get, I mean, there, there are tools now where you can, uh, for 180 bucks get a satellite to take a da a picture today of like a five by five kilometer area. [01:14:21] I just wanna be able to run recon fleets on the back country and get like up to date trail conditions. I don't know that anybody's gonna make any real money doing this, but if it existed, I would use it. So maybe I should build it maybe. Yeah, exactly. Open source. It's part of Databricks longstanding commitment to open source for diversifying new markets. [01:14:44] Awesome. Mike, it was, it was great [01:14:45] to have you. Oh, this was a, yeah. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe

Mike Conover

Co-founder & CEO, Brightwave

BrightwaveDatabricks

If you have 30 dollars, a few hours, and one server, then you are ready to create a ChatGPT-like model that can do what’s known as instruction-following. Databricks’ latest launch, Dolly, foreshadows a potential move in the industry toward smaller and more accessible but extremely capable AIs. Plus, Dolly is open source, requires less computing power, and fewer data parameters than its counterparts. Matei Zaharia, Cofounder & Chief Technologist at Databricks, joins Sarah and Elad to talk about how big data sets actually need to be, why manual annotation is becoming less necessary to train some models, and how he went from a Berkeley PhD student with a little project called Spark to the founder of a company that is now critical data infrastructure that’s increasingly moving into AI. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: Hello Dolly: Democratizing the magic of ChatGPT with open models Dolly Source Code on Github Matei Zaharia - Chief Technologist & Cofounder - Databricks | LinkedIn Matei Zaharia - Google Scholar Databricks debuts ChatGPT-like Dolly, a clone any enterprise can own | VentureBeat Sign up for new podcasts every week. Email feedback to show@no-priors.com Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @Databricks | @Matei_Zaharia Show Notes:  [01:29] - Origin of Databricks [4:30] - Work at Stanford Lab [5:29] - Dolly and Role of Open Source [12:30] - Industry focus on high parameter count, understanding reasoning at small model scale [18:42] - Enterprise applications for Dolly & chat bots [25:06] - Making bets as an academic turned CTO [36:23] - The early stages of AI and future predictions

Matei Zaharia

Co-founder & CTO

Databricks

Jonathan Frankle, Chief Scientist at MosaicML and Assistant Professor of Computer Science at Harvard University, joins us on this episode. With comprehensive infrastructure and software tools, MosaicML aims to help businesses train complex machine-learning models using their own proprietary data. We discuss: - Details of Jonathan’s Ph.D. dissertation which explores his “Lottery Ticket Hypothesis.” - The role of neural network pruning and how it impacts the performance of ML models. - Why transformers will be the go-to way to train NLP models for the foreseeable future. - Why the process of speeding up neural net learning is both scientific and artisanal. - What MosaicML does, and how it approaches working with clients. - The challenges for developing AGI. - Details around ML training policy and ethics. - Why data brings the magic to customized ML models. - The many use cases for companies looking to build customized AI models. Jonathan Frankle - https://www.linkedin.com/in/jfrankle/ Resources: - https://mosaicml.com/ - The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Thanks for listening to the Gradient Dissent podcast, brought to you by Weights & Biases. If you enjoyed this episode, please leave a review to help get the word out about the show. And be sure to subscribe so you never miss another insightful conversation. #OCR #DeepLearning #AI #Modeling #ML

Jonathan Frankle

Chief AI Scientist, Databricks

DatabricksMosaicMLMassachusetts Institute of TechnologyMIT