Constructing and Judging Modern Agentic Workflows
<p>How can you improve your LLM agent systems through specification enrichment? What are the advantages of having an LLM act as a judge within an agent system? This week on the show, Senior IEEE Member and Quality Engineer Suneet Malhotra joins us to discuss building and evaluating agentic architecture.</p> <p>Suneet Malhotra is an independent practitioner-researcher with 18 years of experience in Quality Engineering (QE) and test automation for consumer-scale platforms. He discusses building specification-enrichment loops, monitoring performance, and using Cohen’s kappa to measure agreement between LLM judgments.</p> <p>Suneet is currently publishing multiple papers that are under peer review on these topics. He also provides links to his work and GitHub projects if you want to experiment with these concepts and methods yourself.</p> <p><a href="https://survey.realpython.com/r/gDrDW4">Quick Survey: Get more out of the podcast show notes</a></p> <div class="alert alert-primary" role="alert"> <p><strong>Video Course Spotlight:</strong> <a href="https://realpython.com/courses/testing-mcp-servers-with-a-python-mcp-client/">Testing MCP Servers With a Python MCP Client</a></p> <p>Learn how to build a Python MCP client that tests MCP servers from your terminal. List their tools, prompts, and resources, then call each one.</p> </div> <p>Topics:</p> <ul> <li>00:00:00 – Introduction</li> <li>00:00:56 – Survey: RP Podcast show notes</li> <li>00:02:11 – How did you get into testing?</li> <li>00:05:04 – Has working for large public-facing corporations changed how you approach testing?</li> <li>00:07:06 – Writing a paper on LLM-as-Judge</li> <li>00:09:22 – Looking across the Software Development Lifecycle</li> <li>00:14:46 – Agentic AI: theater vs methodology</li> <li>00:17:23 – Specification enrichment</li> <li>00:27:52 – Video Course Spotlight</li> <li>00:29:18 – Saving the specifications</li> <li>00:31:27 – Using the LLM as a judge & Cohen’s kappa</li> <li>00:39:31 – How can people try out the project?</li> <li>00:43:35 – What are some of the failure modes you’ve seen?</li> <li>00:50:26 – What’s an inexpensive way to try these ideas out?</li> <li>00:54:04 – What are you excited about in the world of Python?</li> <li>00:56:14 – What do you want to learn next?</li> <li>00:57:14 – How can people follow your work online?</li> <li>00:57:32 – Thanks and goodbye</li> </ul> <p>Show Links:</p> <ul> <li><a href="https://suneetmalhotra.com/">Suneet Malhotra - AI-Driven Quality Engineering Leader</a></li> <li><a href="https://github.com/SuneetMalhotra/">SuneetMalhotra - GitHub</a></li> <li><a href="https://orcid.org/0009-0003-8707-9590">Suneet Malhotra - ORCID</a></li> <li><a href="https://zenodo.org/records/20618807">Cross-Layer Observability for LLM-Assisted Test Automation — Reference Implementation and Evaluation Data - Zenodo</a></li> <li><a href="https://zenodo.org/records/20692871">Specification Enrichment v28 (EISEJ submission) — Reference implementation and empirical evaluation - Zenodo</a></li> <li><a href="https://zenodo.org/records/20645248">Visual Oracle Bench — Two-Judge Synthetic-HTML Pilot for LLM-as-Judge Visual Regression Detection with Specificity Reporting - Zenodo</a></li> <li><a href="https://en.wikipedia.org/wiki/Cohen's_kappa">Cohen’s kappa - Wikipedia</a></li> <li><a href="https://www.linkedin.com/in/suneet-m/">Suneet Malhotra - LinkedIn</a></li> <li><a href="https://survey.realpython.com/r/gDrDW4">Survey: Get more out of every Real Python Podcast episode</a></li> </ul> <p>Level up your Python skills with our expert-led courses:</p> <ul> <li><a href="https://realpython.com/courses/improve-tests-mock-object-library/">Improving Your Tests With the Python Mock Object Library</a></li> <li><a href="https://realpython.com/courses/building-type-safe-llm-agents-with-pydantic-ai/">Building Type-Safe LLM Agents With Pydantic AI</a></li> <li><a href="https://realpython.com/courses/testing-mcp-servers-with-a-python-mcp-client/">Testing MCP Servers With a Python MCP Client</a></li> </ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>
Quality Engineer and Independent Practitioner-Researcher