Data is the new oil, and your database is the only way to extract it
Ryan sits down with Shireesh Thota, CVP of Azure Databases at Microsoft, to discuss the evolution of databases at Microsoft; Azure’s comprehensive portfolio that includes SQL Server, CosmosDB, and Postgres; and the challenges that come with database architecture, from the importance of cost governance and multi-cloud strategies to the future of databases when it comes to AI.
Episode notes:
You can read all about the latest Azure database announcements from Microsoft Ignite—including updates for SQL Server, Postgres, DocumentDB, and Fabric—on their Azure blog.
Connect with Shireesh on LinkedIn.
Today’s shoutout goes to user Guffa for winning a Populist badge on their answer to Virtual method tables.
TRANSCRIPT
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Is Your Data Strategy Ready for the Agentic AI Era?
Enterprise AI is still in its infancy, with less than 1% of enterprise data currently used to fuel AI, according to Raj Verma, CEO of SingleStore. While consumer AI is slightly more advanced, most organizations are only beginning to understand the scale of infrastructure needed for true AI adoption. Verma predicts AI will evolve in three phases: first, the easy tasks will be automated; next, complex tasks will become easier; and finally, the seemingly impossible will become achievable—likely within three years.
However, to reach that point, enterprises must align their data strategies with their AI ambitions. Many have rushed into AI fearing obsolescence, but without preparing their data infrastructure, they're at risk of failure. Current legacy systems are not designed for the massive concurrency demands of agentic AI, potentially leading to underperformance. Verma emphasizes the need to move beyond siloed or "swim lane" databases toward unified, high-performance data platforms tailored for the scale and complexity of the AI era.
Learn more from The New Stack about the latest evolution in AI infrastructure:
How To Use AI To Design Intelligent, Adaptable Infrastructure
How to Support Developers in Building AI Workloads
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
What’s Next in Building Better Generative AI Applications?
Since the release of OpenAI's ChatGPT-3 in late 2022, various industries have been actively exploring its applications. Madhukar Kumar, CMO of SingleStore, discussed his experiments with large language models (LLMs) in this podcast episode with TNS host Heather Joslyn. He mentioned a specific LLM called Gorilla, which is trained on APIs and can generate APIs based on specific tasks. Kumar also talked about SingleStore Now, an AI conference, where they plan to teach attendees how to build generative AI applications from scratch, focusing on enterprise applications.
Kumar highlighted a limitation with current LLMs - they are "frozen in time" and cannot provide real-time information. To address this, a method called "retrieval augmented generation" (RAG) has emerged. SingleStore is using RAG to keep LLMs updated. In this approach, a user query is first matched with up-to-date enterprise data to provide context, and then the LLM is tasked with generating answers based on this context. This method aims to prevent the generation of factually incorrect responses and relies on storing data as vectors for efficient real-time processing, which SingleStore enables.
This strategy ensures that LLMs can provide current and contextually accurate information, making AI applications more reliable and responsive for enterprises.
Learn more from The New Stack about LLMs and SingleStore:
Top 5 Large Language Models and How to Use Them Effectively
Using ChatGPT for Questions Specific to Your Company Data
6 Reasons Private LLMs Are Key for Enterprises
When Data is Your Brand and Your Job with Joe Karlsson
Joe Karlsson, Data Engineer at Tinybird, joins Corey on Screaming in the Cloud to discuss what it’s like working in the world of data right now and how he manages the overlap between his social media presence and career. Corey and Joe chat about the rise of AI and whether or not we’re truly seeing advancements in that realm or just trendy marketing plays, and Joe shares why he feels data is getting a lot more attention these days and what it’s like to work in data at this time. Joe also shares insights into how his mental health has been impacted by having a career and social media presence that overlaps, and what steps he’s taken to mitigate the negative impact.
About Joe
Joe Karlsson (He/They) is a Software Engineer turned Developer Advocate at Tinybird. He empowers developers to think creatively when building data intensive applications through demos, blogs, videos, or whatever else developers need.
Joe's career has taken him from building out database best practices and demos for MongoDB, architecting and building one of the largest eCommerce websites in North America at Best Buy, and teaching at one of the most highly-rated software development boot camps on Earth. Joe is also a TEDx Speaker, film buff, and avid TikToker and Tweeter.
Links Referenced:
Tinybird: https://www.tinybird.co/
Personal website: https://joekarlsson.com
Tinybird: https://www.tinybird.co/
Personal website: https://joekarlsson.com
A Multipurpose Database For Transactions And Analytics To Simplify Your Data Architecture With Singlestore
Summary
A large fraction of data engineering work involves moving data from one storage location to another in order to support different access and query patterns. Singlestore aims to cut down on the number of database engines that you need to run so that you can reduce the amount of copying that is required. By supporting fast, in-memory row-based queries and columnar on-disk representation, it lets your transactional and analytical workloads run in the same database. In this episode SVP of engineering Shireesh Thota describes the impact on your overall system architecture that Singlestore can have and the benefits of using a cloud-native database engine for your next application.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Atlan is a collaborative workspace for data-driven teams, like Github for engineering or Figma for design teams. By acting as a virtual hub for data assets ranging from tables and dashboards to SQL snippets & code, Atlan enables teams to create a single source of truth for all their data assets, and collaborate across the modern data stack through deep integrations with tools like Snowflake, Slack, Looker and more. Go to dataengineeringpodcast.com/atlan today and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $3000 on an annual subscription
So now your modern data stack is set up. How is everyone going to find the data they need, and understand it? Select Star is a data discovery platform that automatically analyzes & documents your data. For every table in Select Star, you can find out where the data originated, which dashboards are built on top of it, who’s using it in the company, and how they’re using it, all the way down to the SQL queries. Best of all, it’s simple to set up, and easy for both engineering and operations teams to use. With Select Star’s data catalog, a single source of truth for your data is built in minutes, even across thousands of datasets. Try it out for free and double the length of your free trial today at dataengineeringpodcast.com/selectstar. You’ll also get a swag package when you continue on a paid plan.
Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend.io, 95% in fact reported being at or over capacity. With 72% of data experts reporting demands on their team going up faster than they can hire, it’s no surprise they are increasingly turning to automation. In fact, while only 3.5% report having current investments in automation, 85% of data teams plan on investing in automation in the next 12 months. 85%!!! That’s where our friends at Ascend.io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation, orchestration, and observability. Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug-in architecture, as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark, and can be deployed in AWS, Azure, or GCP. Go to dataengineeringpodcast.com/ascend and sign up for a free trial. If you’re a data engineering podcast listener, you get credits worth $5,000 when you become a customer.
Your host is Tobias Macey and today I’m interviewing Shireesh Thota about Singlestore (formerly MemSQL), the industry’s first modern relational database for multi-cloud, hybrid and on-premises workloads
Interview
Introduction
How did you get involved in the area of data management?
Can you describe what SingleStore is and the story behind it?
The database market has gotten very crouded, with different areas of specialization and nuance being the differentiating factors. What are the core sets of workloads that SingleStore is aimed at addressing?
What are some of the capabilities that it offers to reduce the need to incorporate multiple data stores for application and analytical architectures?
What are some of the most valuable lessons that you learned in your time at MicroSoft that are applicable to SingleStore’s product focus and direction?
Nikita Shamgunov joined the show in October of 2018 to talk about what was then MemSQL. What are the notable changes in the engine and business that have occurred in the intervening time?
What are the macroscopic trends in data management and application development that are having the most impact on product direction?
For engineering teams that are already invested in, or considering adoption of, the "modern data stack" paradigm, where does SingleStore fit in that architecture?
What are the services or tools that might be replaced by an installation of SingleStore?
What are the efficiencies or new capabilities that an engineering team might expect by adopting SingleStore?
What are some of the features that are underappreciated/overlooked which you would like to call attention to?
What are the most interesting, innovative, or unexpected ways that you have seen SingleStore used?
What are the most interesting, unexpected, or challenging lessons that you have learned while working on SingleStore?
When is SingleStore the wrong choice?
What do you have planned for the future of SingleStore?
Contact Info
LinkedIn
@ShireeshThota on Twitter
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Links
MemSQL Interview With Nikita Shamgunov
Singlestore
MS SQL Server
Azure Cosmos DB
CitusDB
Podcast Episode
Debezium
Podcast Episode
PostgreSQL
Podcast Episode
MySQL
HTAP == Hybrid Transactional-Analytical Processing
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
Living with Kubernetes After the 'Honeymoon' Ends
Nearly seven years after Google released Kubernetes, the open source container orchestrator, into an unsuspecting world, 5.6 million developers worldwide use it.
But that number, from the latest Cloud Native Computing Foundation (CNCF) annual survey, masks a lot of frustration. Kubernetes (K8s) can make life easier for the organization that adopts it — after it makes it a lot harder. And as it scales, it can create an unending cadence of triumph and challenge.
In other words: It’s complicated.
At KubeCon + CloudNativeCon EU in Valencia, Spain last week, a trio of experts — Saad Malik, chief technology officer and co-founder of Spectro Cloud; Bailey Hayes, principal software engineer at SingleStore; and Fabrizio Pandini, a staff engineer at VMware — joined Alex Williams, founder and publisher of The New Stack, and myself for a livestream event.
S19:E6 - How to make learning databases fun and approachable ( Joe Karlsson)
In this episode, we talk about making databases fun and approachable with Joe Karlsson, Senior Developer Advocate at Single Store DB. Joe talks about going from hating coding to loving it, the relationship between databases and backend work, and how people can bring creativity and fun into their learning and their work in databases.
Show Links
Partner with Dev & CodeNewbie! (sponsor)
SingleStore
Node.js
JavaScript
Python
SQL
Joe Karlsson
Joe's career has taken him from building out database best practices and demos for MongoDB, architecting and building one largest eCommerce websites in North America at Best Buy, and teaching at one of the most highly-rated software development boot camps on Earth. Joe is also a TEDx Speaker, film buff, and avid TikToker and Tweeter.