Redis

AI agents are increasingly capable of reasoning and performing autonomous work over long periods. However, as agents take on more complex, longer-horizon tasks, keeping them supplied with the right information becomes the core engineering challenge. The industry is moving away from pre-loading context upfront toward a model where agents dynamically navigate and retrieve the data they need, when they need it. Redis is approaching context management using a context engine, which is an architecture built around four pillars: on-demand context retrieval, data that is always current, fast retrieval, and a memory layer that improves over time. In practice this means building materialized views of data with a semantic layer on top, rather than giving agents direct access to production databases. A memory system sits alongside this, extracting and compacting information asynchronously as the agent works. Simba Khadder leads AI strategy at Redis, and he previously co-founded the feature store platform FeatureForm, which was acquired by Redis in 2025. In this episode, Simba joins Kevin Ball to discuss why context has become the defining challenge in agentic AI, how context engines differ from traditional RAG architectures, how materialized views underpin reliable agent data pipelines, how memory systems can improve through async extraction and compaction, and how engineering teams need to adapt their practices as AI-driven development accelerates. Full Disclosure: This episode is sponsored by Redis. Kevin Ball or KBall, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI inaction discussion group through Latent Space. Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com The post Unlocking the Data Layer for Agentic AI with Simba Khadder appeared first on Software Engineering Daily.

Simba Khadder

RedisStreamSQL

A key challenge with designing AI agents is that large language models are stateless and have limited context windows. This requires careful engineering to maintain continuity and reliability across sequential LLM interactions. To perform well, agents need fast systems for storing and retrieving short-term conversations, summaries, and long-term facts. Redis is an open‑source, in‑memory data store widely used for high‑performance caching, analytics, and message brokering. Recent advances have extended Redis’ capabilities to vector search and semantic caching, which has made it an increasingly popular part of the agentic application stack. Andrew Brookins is a Principal Applied AI Engineer at Redis. He joins the show with Sean Falconer to discuss the challenges of building AI agents, the role of memory in agents, hybrid search versus vector-only search, the concept of world models, and more. Full Disclosure: This episode is sponsored by Redis. Sean’s been an academic, startup founder, and Googler. He has published works covering a wide range of topics from AI to quantum computing. Currently, Sean is an AI Entrepreneur in Residence at Confluent where he works on AI strategy and thought leadership. You can connect with Sean on LinkedIn. Please click here to see the transcript of this episode. Sponsorship inquiries: sponsor@softwareengineeringdaily.com   The post Redis and AI Agent Memory with Andrew Brookins appeared first on Software Engineering Daily.

Andrew Brookins

Principal Applied AI Engineer

Redis

Antirez has returned to Redis! Yes, Salvatore Sanfilippo (aka Antirez), the creator of Redis has returned to Redis and he joined us to share the backstory on Redis, what’s going on with the tech and the company, the possible (likely) move back to open source via the AGPL license, the new possibilities of AI and vector embeddings in Redis, and some good ’ol LLM inference discussions. Join the discussion Changelog++ members get a bonus 5 minutes at the end of this episode and zero ads. Join today! Sponsors: Augment Code – Developer AI that uses deep understanding of your large codebase and how you build software to deliver personalized code suggestions and insights. Augment provides relevant, contextualized code right in your IDE or Slack. It transforms scattered knowledge into code or answers, eliminating time spent searching docs or interrupting teammates. Retool – The low-code platform for developers to build internal tools — Some of the best teams out there trust Retool…Brex, Coinbase, Plaid, Doordash, LegalGenius, Amazon, Allbirds, Peloton, and so many more – the developers at these teams trust Retool as the platform to build their internal tools. Try it free at retool.com/changelog Temporal – Build invincible applications. Manage failures, network outages, flaky endpoints, long-running processes and more, ensuring your workflows never fail. Register for Replay in London, March 3-5 to break free from the status quo. Fly.io – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun to get started in minutes. Featuring: Salvatore Sanfilippo &ndash; Website, GitHub, X Adam Stacoviak &ndash; Website, GitHub, LinkedIn, Mastodon, X Jerod Santo &ndash; Website, GitHub, LinkedIn, Mastodon, X Show Notes: Deep Seek on Ollama Claude Sonnet 3.7 is out! <antirez> - We are destroying software Something missing or broken? PRs welcome!

Salvatore Sanfilippo

Creator of Redis

RedisVMware

Redis, best known as a data cache or real-time data platform, is evolving into much more, Tim Hall, chief of product at the company told The New Stack in a recent TNS Makers podcast.   Redis is an in-memory database or memory-first database, which means the data lands there and people are using us for both caching and persistence. However, these days, the company has a number of flexible data models, but one of the brand promises of Redis is developers can store the data as they're working with it. So as opposed to a SQL database where you might have to turn your data structures into columns and tables, you can actually store the data structures that you're working with directly into Redis, Hall said.     Primary Database?   “About 40% of our customers today are using us as a primary database technology,” he said. “That may surprise some people if you're sort of a classic Redis user and you knew us from in-memory caching, you probably didn't realize we added a variety of mechanisms for persistence over the years.”   Meanwhile, to store the data, Redis does store it on disk, sort of behind the scenes while keeping a copy in memory. So if there's any sort of failure, Redis can recover the data off of disk and replay it into memory and get you back up and running. That's a mechanism that has been around about half a decade now.   Yet, Redis is playing what Hall called the ‘long game', particularly in terms of continuing to reach out to developers and showing them what the latest capabilities are.   “If you look at the top 10 databases on the planet, they've all moved into the multimodal category. And Redis is no different from that perspective” Hall said. “So if you look at Oracle it was traditionally a relational database, Mongo is traditionally JSON documents store only, and obviously Redis is a key-value store. We've all moved down the field now. Now, why would we do that? We're all looking to simplify the developer’s world, right?”   Yet, each vendor is really trying to leverage their core differentiation and expand out from there. And the good news for Redis is speed is its core differentiation.   “Why would you want a slow data platform? You don't, Hall said. “So the more that we can offer those extended capabilities for working with things like JSON, or we just launched a data structure called t-digest, that people can use along and we've had support for Bloom filter, which is a probabilistic data structure like all of these things, we kind of expand our footprint, we're saying if you need speed, and reducing latency, and having high interactivity is your goal Redis should be your starting point. If you want some esoteric edge case functionality where you need to manipulate JSON in some very strange way, you probably should go with Mongo. I probably won't support that for a long time. But if you're just working with the basic data structures, you need to be able to query, you need to be able to update your JSON document. Those straightforward use cases we support very, very well, and we support them at speed and scale.”   Customer View   As a Redis customer, Alain Russell, CEO at Blackpepper, a digital e-commerce agency in Auckland, New Zealand, said his firm has undergone the same transition.   “We started off as a Redis as a cache, that helped us speed up traditional data that was slower than we wanted it,” he said. “And then we went down a cloud path a couple of years ago. Part of that migration included us becoming, you know, what's deemed as ‘cloud native.’ And we started using all of these different data stores and data structures and dealing with all of them is actually complicated. You know, and from a developer perspective, it can be a bit painful.”   So, Blackpepper started looking for how to make things simpler, but also keep their platform very fast and they looked at the Redis Stack. “And honestly, it filled all of our needs in one platform. And we're kind of in this path at the moment, we were using the basics of it. And we're very early on in our journey, right? We're still learning how things work and how to use it properly. But we also have a big list of things that we're using other data stores for traditional data, and working out, okay, this will be something that we will migrate to, you know, because we use persistent heavily now, in Redis.”   Twenty-year-old Blackpepper works with predominantly traditional retailers and helps them in their omni-channel journey.   Commercial vs. Open Source   Hall said there are three modes of access to the Redis technology: the Redis open source project, the Redis Stack – which the company recommends that developers start with today -- and then there's Redis Enterprise Edition, which is available as software or in the cloud.   “It's the most popular NoSQL database on the planet six years running,” Hall said. “And people love it because of its simplicity.”   Meanwhile, it takes effort to maintain both the commercial product and the open source effort. Allen, who has worked at Hortonworks, InfluxData, said “Not every open source company is the same in terms of how you make decisions about what lands in your commercial offering and what lands in open source and where the contributions come from and who's involved.”   For instance, “if there was something that somebody wanted to contribute that was going to go against our commercial interest, we probably not would not merge that,” Hall said.   Redis was run by project founder Salvatore Sanfilippo, for many, many years, and he was the sole arbiter of what landed and what did not land in Redis itself. Then, over the last couple of years, Redis created a core steering committee. It's made up of one individual from AWS, one individual from Alibaba, and three Redis employees who look after the contributions that are coming in from the Redis open source community members who want to contribute those things.   “And then we reconcile what we want from a commercial interest perspective, either upstream, or things that, frankly, may have been commoditized and that we want to push downstream into the open source offering, Hall said. “And so the thing that you're asking about is sort of my core existential challenge all the time, that is figuring out where we're going from a commercial perspective. What do we want to land there first? And how can we create a conveyor belt of commercial opportunity that keeps us in business as a software company, creating differentiation against potential competitors show up? And then over time, making sure that those things that do become commoditized, or maybe are not as differentiating anymore, I want to release those to the open source community. But this upstream/downstream kind of challenge is something that we're constantly working through.”   Blackpepper was an open source Redis user initially, but they started a journey where they used Memcached to speed up data. Then they migrated to Redis when they moved to the AWS cloud, Russell said.   Listen to the Podcast   The Redis TNS Makers podcast goes on to look at the use of AI/ML in the platform, the acquisition of RESP.app, the importance of JSON and RediSearch, and where Redis is headed in the future.

Tim Hall

Chief of Product

Redis

About Yiftach Yiftach is an experienced technologist, having held leadership engineering and product roles in diverse fields from application acceleration, cloud computing and software-as-a-service (SaaS), to broadband networks and metro networks. He was the founder, president and CTO of Crescendo Networks (acquired by F5, NASDAQ:FFIV), the vice president of software development at Native Networks (acquired by Alcatel, NASDAQ: ALU) and part of the founding team at ECI Telecom broadband division, where he served as vice president of software engineering. Yiftach holds a Bachelor of Science in Mathematics and Computer Science and has completed studies for Master of Science in Computer Science at Tel-Aviv University. Links: Redis, Inc.: https://redis.com/ Redis open source project: https://redis.io LinkedIn: https://www.linkedin.com/in/yiftachshoolman/ Twitter: https://twitter.com/yiftachsh

Yiftach Shoolman

Co-founder and CTO

Redis

Summary Machine learning is a process driven by iteration and experimentation which requires fast and easy access to relevant features of the data being processed. In order to reduce friction in the process of developing and delivering models there has been a recent trend toward building a dedicated feature. In this episode Simba Khadder discusses his work at StreamSQL building a feature store to make creation, discovery, and monitoring of features fast and easy to manage. He describes the architecture of the system, the benefits of streaming data for machine learning, and how a feature store provides a useful interface between data engineers and machine learning engineers to reduce communication overhead. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management What are the pieces of advice that you wish you had received early in your career of data engineering? If you hand a book to a new data engineer, what wisdom would you add to it? I’m working with O’Reilly on a project to collect the 97 things that every data engineer should know, and I need your help. Go to dataengineeringpodcast.com/97things to add your voice and share your hard-earned expertise. When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $60 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Your host is Tobias Macey and today I’m interviewing Simba Khadder about his views on the importance of ML feature stores, and his experience implementing one at StreamSQL Interview Introduction How did you get involved in the areas of machine learning and data management? What is StreamSQL and what motivated you to start the business? Can you describe what a machine learning feature is? What is the difference between generating features for training a model and generating features for serving? How is feature management typically handled today? What is a feature store and how is it different from the status quo? What is the overall lifecycle of identifying useful features, defining and generating them, using them for training, and then serving them in production? How does the usage of a feature store impact the workflow of ML engineers/data scientists and data engineers? What are the general requirements of a feature store? What additional capabilities or tangential services are necessary for providing a pleasant UX for a feature store? How is discovery and documentation of features handled? What is the current landscape of feature stores and how does StreamSQL compare? How is the StreamSQL feature store implemented? How is the supporting infrastructure architected and how has it evolved since you first began working on it? Why is streaming data such a focal point of feature stores? How do you generate features for training? How do you approach monitoring of features and what does remediation look like for a feature that is no longer valid? How do you handle versioning and deploying features? What’s the process for integrating data sources into StreamSQL for processing into features? How are the features materialized? What are the most challenging or complex aspects of working on or with a feature store? When is StreamSQL the wrong choice for a feature store? What are the most interesting, challenging, or unexpected lessons that you have learned in the process of building StreamSQL? What do you have planned for the future of the product? Contact Info LinkedIn @simba_khadder on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links StreamSQL Feature Stores for ML Distributed Systems Google Cloud Datastore Triton Uber Michelangelo AirBnB Zipline Lyft Dryft Apache Flink Podcast Episode Apache Kafka Spark Streaming Apache Cassandra Redis Apache Pulsar Podcast Episode StreamNative Episode TDD == Test Driven Development Lyft presentation – Bootstrapping Flink Go-Jek Feast Hopsworks The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

Simba Khadder

RedisStreamSQL

Wynn caught up with Salvatore Sanfilippo to talk about Redis, the super hot key value store. Join the discussion Changelog++ members support our work, get closer to the metal, and make the ads disappear. Join today! Featuring: Salvatore Sanfilippo &ndash; Website, GitHub, X Wynn Netherland &ndash; GitHub, X Show Notes: VMware signs the paychecks for Salvatore and Pieter Noordhuis Redis is an open source, advanced key-value store and data structure server wherein keys can contain strings, hashes, lists, sets and sorted sets Redis internals consist of ANSI C with an evented model Non-blocking replication has always been a Redis design goal Replication in Redis is async Salvatore’s Redis toolbox includes the Redis Ruby gem and Sinatra Chances are you can find a Redis library in your favorite language The C client is the only officially supported wrapper Salvatore thinks the NoSQL moniker isn’t perfect, focusing too much on performance, but it frames a discussion Redis Pub/Sub is perfect for real-time apps GitHub’s adoption of Redis in Resque helped fuel the growth of the project Redis users tend to use it as a database, as a messaging bus, or as a cache Salvatore thinks hosted solutions like Redis-to-Go need to add more value like more frequent backups and seamless upgrades. Blizzard uses an 8-node Redis install in serving avatars for WoW Justin Campbell asks will VMWare feature Redis in any upcoming projects? Ezra Zygmuntowicz and GitHub were among the first “few brave users” After a few months Salvatore noticed a dip in adoption , but he trusted his gut and stuck with it Something missing or broken? PRs welcome!

Salvatore Sanfilippo

Creator of Redis

RedisVMware