Configuring a Versatile LLM Harness & Scraping the Web With Scrapy
<p>Which is more important, the model or the “harness” around an LLM? What are ways to assemble an efficient agentic developer workflow? This week on the show, Ayan Pahwa joins us to discuss harnessing, web scraping, and self-hosting Python applications.</p> <p>Ayan is a developer advocate at Zyte and an experienced project builder. We discuss a recent article he wrote about creating an extension for the web scraping tool Scrapy. He also digs into his self-hosting setup for Python applications and tools.</p> <p>Our discussion extends to the complexities of developing effective harnesses. Ayan shares his setup and how he navigated shifting from prompt engineering to context and loop engineering.</p> <p>This episode is sponsored by <a href="https://hydradb.com/?utm_source=real-python&utm_medium=real%20python&utm_campaign=dev-tool-listing&utm_content=podcast">HydraDB</a>.</p> <div class="alert alert-primary" role="alert"> <p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/introduction-to-web-scraping-with-python/">Introduction to Web Scraping With Python</a></p> <p>In this video course, you’ll learn all about web scraping in Python. You’ll see how to parse data from websites and interact with HTML forms using tools such as Beautiful Soup and MechanicalSoup.</p> </div> <p>Topics:</p> <ul> <li>00:00:00 – Introduction</li> <li>00:02:01 – Scrapy and building an extension</li> <li>00:08:46 – Zyte and the web scraping API</li> <li>00:11:19 – Sponsor: HydraDB</li> <li>00:12:22 – noalgotube project</li> <li>00:15:49 – Homelab & self hosting projects</li> <li>00:22:19 – What goes into a harness?</li> <li>00:32:33 – Where did you start exploring LLM tools?</li> <li>00:36:13 – Local models & edge computing</li> <li>00:39:31 – ExtractPod and discussing Apple’s AI</li> <li>00:43:35 – Video Course Spotlight</li> <li>00:44:54 – Managing token use and tools</li> <li>00:52:37 – What are you excited about in the world of Python?</li> <li>00:55:10 – What do you want to learn next?</li> <li>00:56:38 – What is the best way to follow your work online?</li> <li>00:56:58 – Thanks and goodbye</li> </ul> <p>Show Links:</p> <ul> <li><a href="https://www.zyte.com/blog/how-to-build-your-first-scrapy-extension/">How to build your first Scrapy extension</a></li> <li><a href="https://www.zyte.com/zyte-api/">Web Scraping API - All-in-one Web Scraper - Zyte API</a></li> <li><a href="https://realpython.com/web-scraping-with-scrapy-and-mongodb/">Web Scraping With Scrapy and MongoDB – Real Python</a></li> <li><a href="https://codensolder.com/posts/noalgotube-algorithm-free-youtube-rss-reader/">noalgotube: I Built My Own YouTube Feed Because the Algorithm Stopped Working for Me - CodeNSolder</a></li> <li><a href="https://github.com/iayanpahwa/noalgotube">noalgotube: A personal content aggregator for YouTube channels and blog RSS feeds</a></li> <li><a href="https://codensolder.com/posts/why-homelab-building-a-proper-self-hosted-setup-from-scratch/">Why homelab? Building a proper self-hosted setup from scratch - CodeNSolder</a></li> <li><a href="https://opnsense.org/">OPNsense: Open source, feature rich firewall and routing platform, offering cutting-edge network protection</a></li> <li><a href="https://www.proxmox.com/en/">Proxmox - Powerful open-source server solutions</a></li> <li><a href="https://pi-hole.net/">Pi-hole – Network-wide Ad Blocking</a></li> <li><a href="https://github.com/iib0011/omni-tools">omni-tools: Self-hosted collection of powerful web-based tools for everyday tasks</a></li> <li><a href="https://frigate.video/">Frigate NVR</a></li> <li><a href="https://www.zyte.com/blog/harness-engineering-part-1-what-is-an-agent-harness-and-why-it-matters/">Harness Engineering, part 1: What is an agent harness and why it matters</a></li> <li><a href="https://www.zyte.com/blog/my-agentic-coding-setup-claude-code-multi-agent-orchestration-and-how-i-actually-work/">My agentic coding setup: Claude Code, multi-agent orchestration, and how I actually work</a></li> <li><a href="https://www.youtube.com/watch?v=kZP_TumcFAY">ExtractPod EP07 - AI Harnesses, our model usage and a Scottish dinner staple. - YouTube</a></li> <li><a href="https://opencode.ai/">OpenCode - The open source AI coding agent</a></li> <li><a href="https://openrouter.ai/">OpenRouter</a></li> <li><a href="https://deepmind.google/models/gemma/gemma-4/">Gemma 4 — Google DeepMind</a></li> <li><a href="https://lmstudio.ai/">LM Studio Bionic - Agent for Open Models</a></li> <li><a href="https://z.ai/blog/glm-5.2">GLM-5.2: Built for Long-Horizon Tasks</a></li> <li><a href="https://github.com/JuliusBrussee/caveman">caveman: 🪨 why use many token when few token do trick — Claude Code skill</a></li> <li><a href="https://github.com/DietrichGebert/ponytail">ponytail: Makes your AI agent think like the laziest senior dev in the room</a></li> <li><a href="https://realpython.com/podcasts/rpp/301/">Episode #301: Running Python Locally in a Sandbox</a></li> <li><a href="https://aillio.com/r2">Aillio – Bullet R2 - Coffee Roaster</a></li> <li><a href="https://codensolder.com/">CodeNSolder</a></li> <li><a href="https://www.zyte.com/author/ayan-pahwa/">Ayan Pahwa - Zyte</a></li> <li><a href="https://hydradb.com/?utm_source=real-python&utm_medium=real%20python&utm_campaign=dev-tool-listing&utm_content=podcast">HydraDB</a></li> </ul> <p>Level up your Python skills with our expert-led courses:</p> <ul> <li><a href="https://realpython.com/courses/introduction-to-web-scraping-with-python/">Introduction to Web Scraping With Python</a></li> <li><a href="https://realpython.com/courses/getting-started-claude-code/">Getting Started With Claude Code</a></li> <li><a href="https://realpython.com/courses/testing-mcp-servers-with-a-python-mcp-client/">Testing MCP Servers With a Python MCP Client</a></li> </ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>