Hamel.ai

Hamel Husain and Shreya Shankar teach the world’s most popular course on AI evals and have trained over 2,000 PMs and engineers (including many teams at OpenAI and Anthropic). In this conversation, they demystify the process of developing effective evals, walk through real examples, and share practical techniques that’ll help you improve your AI product. What you’ll learn: 1. WTF evals are 2. Why they’ve become the most important new skill for AI product builders 3. A step-by-step walkthrough of how to create an effective eval 4. A deep dive into error analysis, open coding, and axial coding 5. Code-based evals vs. LLM-as-judge 6. The most common pitfalls and how to avoid them 7. Practical tips for implementing evals with minimal time investment (30 minutes per week after initial setup) 8. Insight into the debate between “vibes” and systematic evals — Brought to you by: Fin—The #1 AI agent for customer service Dscout—The UX platform to capture insights at every stage: from ideation to production Mercury—The art of simplified finances — Where to find Shreya Shankar • X: https://x.com/sh_reya • LinkedIn: https://www.linkedin.com/in/shrshnk/ • Website: https://www.sh-reya.com/ • Maven course: https://bit.ly/4myp27m — Where to find Hamel Husain • X: https://x.com/HamelHusain • LinkedIn: https://www.linkedin.com/in/hamelhusain/ • Website: https://hamel.dev/ • Maven course: https://bit.ly/4myp27m — In this episode, we cover: (00:00) Introduction to Hamel and Shreya (04:57) What are evals? (09:56) Demo: Examining real traces from a property management AI assistant (16:51) Writing notes on errors (23:54) Why LLMs can’t replace humans in the initial error analysis (25:16) The concept of a “benevolent dictator” in the eval process (28:07) Theoretical saturation: when to stop (31:39) Using axial codes to help categorize and synthesize error notes (44:39) The results (46:06) Building an LLM-as-judge to evaluate specific failure modes (48:31) The difference between code-based evals and LLM-as-judge (52:10) Example: LLM-as-judge (54:45) Testing your LLM judge against human judgment (01:00:51) Why evals are the new PRDs for AI products (01:05:09) How many evals you actually need (01:07:41) What comes after evals (01:09:57) The great evals debate (1:15:15) Why dogfooding isn’t enough for most AI products (01:18:23) OpenAI’s Statsig acquisition (1:23:02) The Claude Code controversy and the importance of context (01:24:13) Common misconceptions around evals (1:22:28) Tips and tricks for implementing evals effectively (1:30:37) The time investment (1:33:38) Overview of their comprehensive evals course (1:37:57) Lightning round and final thoughts — LLM Log Open Codes Analysis Prompt: Please analyze the following CSV file. There is a metadata field which has an nested field called z_note that contains open codes for analysis of LLM logs that we are conducting. Please extract all of the different open codes. From the _note field, propose 5-6 categories that we can create axial codes from. — Referenced: • Building eval systems that improve your AI product: https://www.lennysnewsletter.com/p/building-eval-systems-that-improve • Mercor: https://mercor.com/ • Brendan Foody on LinkedIn: https://www.linkedin.com/in/brendan-foody-2995ab10b • Nurture Boss: https://nurtureboss.io/ • Braintrust: https://www.braintrust.dev/ • Andrew Ng on X: https://x.com/andrewyng • Carrying Out Error Analysis: https://www.youtube.com/watch?v=JoAxZsdw_3w • Julius AI: https://julius.ai/ • Brendan Foody on X—“evals are the new PRDs”: https://x.com/BrendanFoody/status/1939764763485171948 • Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://dl.acm.org/doi/abs/10.1145/3654777.3676450 • Lenny’s post on X about evals: https://x.com/lennysan/status/1909636749103599729 • Statsig: https://statsig.com/ • Claude Code: https://www.anthropic.com/claude-code • Cursor: https://cursor.com/ • Occam’s razor: https://en.wikipedia.org/wiki/Occam%27s_razor • Frozen: https://www.imdb.com/title/tt2294629/ • The Wire on HBO: https://en.wikipedia.org/wiki/The_Wire — Recommended books: • Pachinko: https://www.amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935 • Apple in China: The Capture of the World’s Greatest Company: https://www.amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373/ • Machine Learning: https://www.amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955 • Artificial Intelligence: A Modern Approach: https://www.amazon.com/Artificial-Intelligence-Modern-Approach-Global/dp/1292401133/ Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com. — Lenny may be an investor in the companies discussed. My biggest takeaways from this conversation: To hear more, visit www.lennysnewsletter.com

Hamel Husain

AI Researcher and Educator

Hamel.aiParlance LabsGitHub

Shreya Shankar

PhD Researcher

UC Berkeley

Today, we're joined by Hamel Husain, founder of Parlance Labs, to discuss the ins and outs of building real-world products using large language models (LLMs). We kick things off discussing novel applications of LLMs and how to think about modern AI user experiences. We then dig into the key challenge faced by LLM developers—how to iterate from a snazzy demo or proof-of-concept to a working LLM-based application. We discuss the pros, cons, and role of fine-tuning LLMs and dig into when to use this technique. We cover the fine-tuning process, common pitfalls in evaluation—such as relying too heavily on generic tools and missing the nuances of specific use cases, open-source LLM fine-tuning tools like Axolotl, the use of LoRA adapters, and more. Hamel also shares insights on model optimization and inference frameworks and how developers should approach these tools. Finally, we dig into how to use systematic evaluation techniques to guide the improvement of your LLM application, the importance of data generation and curation, and the parallels to traditional software engineering practices. The complete show notes for this episode can be found at https://twimlai.com/go/694.

Hamel Husain

AI Researcher and Educator

Hamel.aiParlance LabsGitHub

Today we’re joined by Hamel Husain, Staff Machine Learning Engineer at GitHub.  Over the last few years, Hamel has had the opportunity to work on some of the most popular open source projects in the ML world, including fast.ai, nbdev, fastpages, and fastcore, just to name a few. In our conversation with Hamel, we discuss his journey into Silicon Valley, and how he discovered that the ML tooling and infrastructure weren’t quite as advanced as he’d assumed, and how that led him to help build some of the foundational pieces of Airbnb’s Bighead Platform.  We also spend time exploring Hamel’s time working with Jeremy Howard and the team creating fast.ai, how nbdev came about, and how it plans to change the way practitioners interact with traditional jupyter notebooks. Finally, talk through a few more tools in the fast.ai ecosystem, fastpages, fastcore, how these tools interact with Github Actions, and the up and coming ML tools that Hamel is excited about.  The complete show notes for this episode can be found at twimlai.com/go/532.

Hamel Husain

AI Researcher and Educator

Hamel.aiParlance LabsGitHub

Hamel Husain is a Staff Machine Learning Engineer at Github. He has extensive experience building data analytics and predictive modeling solutions for a wide range of industries, including: hospitality, telecom, retail, restaurant, entertainment and finance. He has built large data science teams (50+) from the ground up and have extensive experience building solutions as an individual contributor. Follow Hamel on Twitter: https://twitter.com/HamelHusain And on his website: http://hamel.io/ Learn more about Github Actions: https://github.com/features/actions and the CodeSearchNet Challenge: https://github.blog/2019-09-26-introducing-the-codesearchnet-challenge/ Visit our podcasts homepage for transcripts and more episodes! www.wandb.com/podcast 🔊 Get our podcast on Apple, and Spotify! Apple Podcasts: https://bit.ly/2WdrUvI Spotify: https://bit.ly/2SqtadF We started Weights and Biases to build tools for Machine Learning practitioners because we care a lot about the impact that Machine Learning can have in the world and we love working in the trenches with the people building these models. One of the most fun things about these building tools has been the conversations with these ML practitioners and learning about the interesting things they’re working on. This process has been so fun that we wanted to open it up to the world in the form of our new podcast called Gradient Dissent. We hope you have as much fun listening to it as we had making it! 👩🏼‍🚀Weights and Biases: We’re always free for academics and open source projects. Email carey@wandb.com with any questions or feature suggestions. - Blog: https://www.wandb.com/articles - Gallery: See what you can create with W&B - https://app.wandb.ai/gallery - Continue the conversation on our slack community - http://bit.ly/wandb-forum 🎙Host: Lukas Biewald - https://twitter.com/l2k 👩🏼‍💻Producer: Lavanya Shukla - https://twitter.com/lavanyaai 📹Editor: Cayla Sharp - http://caylasharp.com/

Hamel Husain

AI Researcher and Educator

Hamel.aiParlance LabsGitHub