Why your company should (try to) build its own AI SRE
As AI coding agents accelerate software development, they also create new challenges for site reliability engineers (SREs), who are increasingly responsible for debugging systems that no single human fully understands. In this episode ofThe New Stackpodcast, Sam Farid and Nate Heinrich of Chronosphere argue that AI agents should also be used for root-cause analysis, helping teams diagnose failures more quickly as model capabilities continue to improve.
Rather than immediately purchasing a commercial solution, they recommend organizations first build an in-house AI SRE. The process of documenting systems, dependencies, and operational knowledge creates valuable context that enables AI agents to troubleshoot effectively while improving institutional knowledge. Although Chronosphere offers its own AI SRE platform, the hosts emphasize that building an internal prototype helps teams understand their needs before evaluating vendor tools. As AI-generated code becomes more common, organizations that invest in mapping their systems and leveraging AI for operations will be better equipped to reduce downtime and support increasingly complex software environments.
Learn more from The New Stack around AI SREs:
5 ways SRE AI agents are set to augment human capabilities
The Future of AI in SRE: Preventing Failures, Not Fixing Them
AI Reliability Engineering: Welcome to the Third Age of SRE
Join our community of newsletter subscribers to stay on top of the news and at the top of your game.
Insights from a Vendor Insider with Ian Smith
It turns out, you don’t need to step outside to observe the clouds. On this episode, we’re joined by Chronosphere Field CTO Ian Smith. He and Corey delve into the innovative solutions Chronosphere offers, share insights from Ian’s experience in the industry, and discuss the future of cloud-native technologies. Whether you're a seasoned cloud professional or new to the field, this conversation with Ian Smith is packed with valuable perspectives and actionable takeaways.
Show Highlights:
(0:00) Intro
(0:42) Chronosphere sponsor read
(1:53) The role of Chief of Staff at Chronosphere
(2:45) Getting recognized in the Gartner Magic Quadrant
(4:42) Talking about the buying process
(8:26) The importance of observability
(10:18) Guiding customers as a vendor
(12:19) Chronosphere sponsor read
(12:46) What should you do as an observability buyer
(16:01) Helping orgs understand observability
(19:56) Avoiding toxicly positive endorsements
(24:15) Being transparent as a vendor
(27:43) The myth of "winner take all"
(30:02) Short term fixes vs. long term solutions
(33:54) Where you can find more from Ian and Chronosphere
About Ian Smith
Ian Smith is Field CTO at Chronosphere where he works across sales, marketing, engineering and product to deliver better insights and outcomes to observability teams supporting high-scale cloud-native environments. Previously, he worked with observability teams across the software industry in pre-sales roles at New Relic, Wavefront, PagerDuty and Lightstep.
Links
Chronosphere: https://chronosphere.io/?utm_source=duckbill-group&utm_medium=podcast
Ian’s Twitter: https://x.com/datasmithing
Ian’s LinkedIn: https://www.linkedin.com/in/ismith314159/
Sponsor
Chronosphere: https://chronosphere.io/?utm_source=duckbill-group&utm_medium=podcast
How to Know If You’re Building the Right Internal Tools
In this episode of The New Stack Makers, Rob Skillington, co-founder and CTO of Chronosphere, discusses the challenges engineers face in building tools for their organizations. Skillington emphasizes that the "build or buy" decision oversimplifies the issue of tooling and suggests that understanding the abstractions of a project is crucial. Engineers should consider where to build and where to buy, creating solutions that address the entire problem. Skillington advises against short-term thinking, urging innovators to consider the long-term landscape.
Drawing from his experience at Uber, Skillington highlights the importance of knowing the audience and customer base, even when they are colleagues. He shares a lesson learned when building a visualization platform for engineers at Uber, where understanding user adoption as a key performance indicator upfront could have improved the project's outcome.
Skillington also addresses the "not invented here syndrome," noting its prevalence in organizations like Microsoft and its potential impact on tool adoption. He suggests that younger companies, like Uber, may be more inclined to explore external solutions rather than building everything in-house. The conversation provides insights into Skillington's experiences and the considerations involved in developing internal tools and platforms.
Learn more from The New Stack about Software Engineering, Observability, and Chronosphere:
Cloud Native Observability: Fighting Rising Costs, Incidents
A Guide to Measuring Developer Productivity
4 Key Observability Best Practices
Chronosphere on Crafting a Cloud-Native Observability Strategy with Rachel Dines
Rachel Dines, Head of Product and Technical Marketing at Chronosphere, joins Corey on Screaming in the Cloud to discuss why creating a cloud-native observability strategy is so critical, and the challenges that come with both defining and accomplishing that strategy to fit your current and future observability needs. Rachel explains how Chronosphere is taking an open-source approach to observability, and why it’s more important than ever to acknowledge that the stakes and costs are much higher when it comes to observability in the cloud.
About Rachel
Rachel leads product and technical marketing for Chronosphere. Previously, Rachel wore lots of marketing hats at CloudHealth (acquired by VMware), and before that, she led product marketing for cloud-integrated storage at NetApp. She also spent many years as an analyst at Forrester Research. Outside of work, Rachel tries to keep up with her young son and hyper-active dog, and when she has time, enjoys crafting and eating out at local restaurants in Boston where she’s based.
Links Referenced:
Chronosphere: https://chronosphere.io/
LinkedIn: https://www.linkedin.com/in/rdines/
Build vs. buy doesn't matter. Tool adoption does.
SPONSORED BY CHRONOSPHEREEpisode notes:
Chronosphere is introducing Lens, a cloud-native observability tool to view data flows between services.
At Uber, Rob created M3, an open-source metrics engine compatible with Prometheus.
Congrats to Lifeboat badge winner, ralf htp, for their answer to How to read an image in Python OpenCV.
If you want to meet Rob and the Chronosphere team, they’ll be at AWS re:Invent from Nov. 27 to Dec. 1.
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Cloud Native Observability: Fighting Rising Costs, Incidents
Observability in multi-cloud environments is becoming increasingly complex, as highlighted by Martin Mao, CEO and co-founder of Chronosphere. This challenge has two main components: a rise in customer-facing incidents, which demand significant engineering time for debugging, and the ineffectiveness and high cost of existing tools. These issues are creating a problematic return on investment for the industry.
Mao discussed these observability challenges on The New Stack Makers podcast with host Heather Joslyn, emphasizing the need to help teams prioritize alerts and encouraging a shift left approach for security responsibility among developers. With the adoption of distributed cloud architectures, organizations are not only dealing with a surge in data but also facing a cultural shift towards DevOps, where developers are expected to be more accountable for their software in production.
Historically, operations teams handled software in production, but in the cloud-native world, developers must take on these responsibilities themselves. Many current observability tools were designed for centralized operations teams, which creates a gap in addressing developer needs.
Mao suggests that cloud-native observability tools should empower developers to run and maintain their software in production, providing insights into the complex environments they work in. Moreover, observability tools can assist developers in understanding the intricacies of their software, such as its dependencies and operational aspects.
To streamline the data obtained from observability efforts and manage costs, Chronosphere introduced the "Observability Data Optimization Cycle." This framework starts with establishing centralized governance to set budgets for teams generating data. The goal is to optimize data usage to extract value without incurring unnecessary costs. This approach applies financial operations (FinOps) concepts to the observability space, helping organizations tackle the challenges of cloud-native observability.
Learn more from The New Stack about Observability and Chronosphere:
Observability Overview, News and Trends
4 Key Observability Best Practices
Top Ways to Reduce Your Observability Costs
Top 4 Factors for Cloud Native Observability Tool Selection
Unpacking the Costs and Value of Observability with Martin Mao
Martin Mao, CEO & Cofounder at Chronosphere, joins Corey on Screaming in the Cloud to discuss the trends he sees in the observability industry. Martin explains why he feels measuring observability costs isn’t nearly as important as understanding the velocity of observability costs increasing, and why he feels efficiency is something that has to be built into processes as companies scale new functionality. Corey and Martin also explore how observability can now be used by business executives to provide top line visibility and value, as opposed to just seeing observability as a necessary cost.
About Martin
Martin is a technologist with a history of solving problems at the largest scale in the world and is passionate about helping enterprises use cloud native observability and open source technologies to succeed on their cloud native journey. He's now the Co-Founder & CEO of Chronosphere, a Series C startup with $255M in funding, backed by Greylock, Lux Capital, General Atlantic, Addition, and Founders Fund. He was previously at Uber, where he led the development and SRE teams that created and operated M3. Previously, he worked at AWS, Microsoft, and Google. He and his family are based in the Seattle area, and he enjoys playing soccer and eating meat pies in his spare time.
Links Referenced:
Chronosphere: https://chronosphere.io/
LinkedIn: https://www.linkedin.com/in/martinmao/
Ops teams are pets, not cattle (ep. 556)
A common refrain you’ll hear these days is that servers should be scaled out, easy to replace, and interchangeable—cattle, not pets. But for the ops folks who run those servers the opposite is true. You can’t just throw any of them into an incident where they may not know the stack or system and expect everything to work out. Every operator has a set of skills that they’ve built up through research or experience, and teams should value them as such. They’re people, not pets, and certainly not cattle—you can’t just get a new one when you burn out your existing ones.
On this episode of the podcast—sponsored by Chronosphere—we talk with Paige Cruz, Senior Developer Advocate at Chronosphere, about how teams can reduce the cognitive load on ops, the best ways to prepare for inevitable failures, and where the worst place to page Paige is.
Episode notes:
Chronosphere provides an observability platform for ops people, so naturally, the company has an interest in the happiness of those people.
If you’re interested in the history of the pets vs. cattle concept , this covers it pretty well.
Previously, we spoke with the CEO of Chronosphere about making incidents easier to manage.
We’ve covered this topic on the blog before, and two articles came up during our conversation with Paige.
You can connect with Paige on Twitter, where she has a pretty apropos handle.
Congrats to Stellar Question badge winner Bruno Rocha for asking How can I read large text files line by line, without loading them into memory?, which at least 100 users liked enough to bookmark.
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Chronosphere Nudges Observability Standards Toward Maturity
DETROIT — Rob Skillington’s grandfather was a civil engineer, working in an industry that, in over a century, developed processes and know-how that enabled the creation of buildings, bridges and road.
“A lot of those processes matured to a point where they could reliably build these things,” said Skillington, co-founder and chief technology officer at Chronosphere, an observability platform. “And I think about observability as that same maturity of engineering practice. When it comes to building software that actually is useful in the world, it is this process that helps you actually achieve the deployment and operation of these large scale systems that we use every day.”
Skillington spoke about the evolution of observability, and his company’s recent donation of an open source project to Prometheus, in this episode of The New Stack Makers podcast. Heather Joslyn, features editor of TNS, hosted the conversation.
This On the Road edition of The New Stack Makers was recorded at KubeCon + CloudNativeCon North America, in the Motor City. The episode was sponsored by Chronosphere.
A Donation to the Prometheus Project
Helping observability practices grow as mature and reliable as civil engineering rules that help build sturdy skyscrapers is a tough task, Skillington suggested.
In the cloud era, he said, “you have to really prepare the software for a whole set of runtime environments. And so the challenges around that is really about making it consistent, well understood and robust.”
At KubeCon in late October, Chronosphere and PromLabs (founded by Julius Volz, creator of Prometheus) announced that they had donated their open source project PromLens to the Prometheus project, the open source monitoring and alerts primitive.
The donation is a way of placing a bet on a tool that integrates well with Kubernetes. “There's this real yearning for essentially a standard that can be built upon by everyone in the industry, when it comes to these core primitives, essentially,” Skillington said. “And Prometheus is one of those primitives. We want to continue to solidify that as a primitive that stands the test of time.”
“We can't build a self-driving car if we're always building a different car,” he added.
PromLens builds Prometheus queries in a sort of integrated development environment (IDE), Skillington said. It also makes it easier for more people in an organization to create queries and understand the meaning and seriousness of alerts.
The PromLens tool breaks queries into a visual format, and allows users to edit them through a UI. “Basically, it's kind of like a What You See Is What You Get editor, or WYSIWYG editor, for Prometheus queries,” Skillington said.
“Some of our customers have tens of thousands of these alerts to find in PromQL, which is the query language for Prometheus,” he noted. “Having a tool like an integrated development environment — where you can really understand these complex queries and iterate faster on, setting these up and getting back to your day job — is incredibly important.”
Check out the full episode for more on PromLens and the current state of observability.
The Ever-Changing World of Cloud Native Observability with Ian Smith
About Ian
Ian Smith is Field CTO at Chronosphere where he works across sales, marketing, engineering and product to deliver better insights and outcomes to observability teams supporting high-scale cloud-native environments. Previously, he worked with observability teams across the software industry in pre-sales roles at New Relic, Wavefront, PagerDuty and Lightstep.
Links Referenced:
Chronosphere: https://chronosphere.io
Last Tweet in AWS: lasttweetinaws.com
The Multi-Colored Brick Road to the Cloud with Rachel Dines
About Rachel
Rachel leads product and technical marketing for Chronosphere. Previously, Rachel wore lots of marketing hats at CloudHealth (acquired by VMware), and before that, she led product marketing for cloud-integrated storage at NetApp. She also spent many years as an analyst at Forrester Research. Outside of work, Rachel tries to keep up with her young son and hyper-active dog, and when she has time, enjoys crafting and eating out at local restaurants in Boston where she’s based.
Links:
Chronosphere: https://chronosphere.io
Twitter: https://twitter.com/RachelDines
Email: rachel@chronosphere.io
Open Core, Real-Time Observability Born in the Cloud with Martin Mao
About Martin
Martin Mao is the co-founder and CEO of Chronosphere. He was previously at Uber, where he led the development and SRE teams that created and operated M3. Prior to that, he was a technical lead on the EC2 team at AWS and has also worked for Microsoft and Google. He and his family are based in our Seattle hub and he enjoys playing soccer and eating meat pies in his spare time.
Links:
Chronosphere: https://chronosphere.io/
Email: contact@chronosphere.io
System Observability For The Cloud Native Era With Chronosphere
Summary
Collecting and processing metrics for monitoring use cases is an interesting data problem. It is eminently possible to generate millions or billions of data points per second, the information needs to be propagated to a central location, processed, and analyzed in timeframes on the order of milliseconds or single-digit seconds, and the consumers of the data need to be able to query the information quickly and flexibly. As the systems that we build continue to grow in scale and complexity the need for reliable and manageable monitoring platforms increases proportionately. In this episode Rob Skillington, CTO of Chronosphere, shares his experiences building metrics systems that provide observability to companies that are operating at extreme scale. He describes how the M3DB storage engine is designed to manage the pressures of a critical system component, the inherent complexities of working with telemetry data, and the motivating factors that are contributing to the growing need for flexibility in querying the collected metrics. This is a fascinating conversation about an area of data management that is often taken for granted.
Announcements
Hello and welcome to the Data Engineering Podcast, the show about modern data management
When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask.
Today’s episode of Data Engineering Podcast is sponsored by Datadog, the monitoring and analytics platform for cloud-scale infrastructure and applications. Datadog’s machine-learning based alerts, customizable dashboards, and 400+ vendor-backed integrations makes it easy to unify disparate data sources and pivot between correlated metrics and events for faster troubleshooting. By combining metrics, traces, and logs in one place, you can easily improve your application performance. Try Datadog free by starting a your 14-day trial and receive a free t-shirt once you install the agent. Go to dataengineeringpodcast.com/datadog today see how you can unify your monitoring today.
Your host is Tobias Macey and today I’m interviewing Rob Skillington about Chronosphere, a scalable, reliable and customizable monitoring-as-a-service purpose built for cloud-native applications.
Interview
Introduction
How did you get involved in the area of data management?
Can you start by describing what you are building at Chronosphere and your motivation for turning it into a business?
What are the biggest challenges inherent to monitoring use cases?
How does the advent of cloud native environments complicate things further?
While you were at Uber you helped to create the M3 storage engine. There are a wide array of time series databases available, including many purpose built for metrics use cases. What were the missing pieces that made it necessary to create a new system?
How do you handle schema design/data modeling for metrics storage?
How do the usage patterns of metrics systems contribute to the complexity of building a storage layer to support them?
What are the optimizations that need to be made for the read and write paths in M3?
How do you handle high cardinality of metrics and ad-hoc queries to understand system behaviors?
What are the scaling factors for M3?
Can you describe how you have architected the Chronosphere platform?
What are the convenience features built on top of M3 that you are creating at Chronosphere?
How do you handle deployment and scaling of your infrastructure given the scale of the businesses that you are working with?
Beyond just server infrastructure and application behavior, what are some of the other sources of metrics that you and your users are sending into Chronosphere?
How do those alternative metrics sources complicate the work of generating useful insights from the data?
In addition to the read and write loads, metrics systems also need to be able to identify patterns, thresholds, and anomalies in the data to alert on it with minimal latency. How do you handle that in the Chronosphere platform?
What are some of the most interesting, innovative, or unexpected ways that you have seen Chronosphere/M3 used?
What are some of the most interesting, unexpected, or challenging lessons that you have learned while building Chronosphere?
When is Chronosphere the wrong choice?
What do you have planned for the future of the platform and business?
Contact Info
LinkedIn
@roskilli on Twitter
robskillington on GitHub
Parting Question
From your perspective, what is the biggest gap in the tooling or technology for data management today?
Closing Announcements
Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used.
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story.
To help other people find the show please leave a review on iTunes and tell your friends and co-workers
Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat
Links
Chronosphere
Lidar
Cloud Native
M3DB
OpenTracing
Metrics/Telemetry
Graphite
Podcast.__init__ Episode
InfluxDB
Clickhouse
Podcast Episode
Prometheus
Inverted Index
Druid
Cardinality
Apache Flink
Podcast Episode
HDFS
Avro
Podcast Episode
Grafana
Tecton
Podcast Episode
Datadog
Podcast Episode
Kubernetes
Sourcegraph
The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA
Support Data Engineering Podcast
Monitoring, Metrics and M3, with Martin Mao and Rob Skillington
Martin Mao and Rob Skillington are co-founders of Chronosphere; CEO and CTO respectively. They both worked on the monitoring team at Uber, where they created M3: a metrics platform with an open source time-series database built for scale. They join Craig and Adam to talk about monitoring, metrics and M3 on the last episode of 2019.
Do you have something cool to share? Some questions? Let us know:
web: kubernetespodcast.com
mail: kubernetespodcast@google.com
twitter: @kubernetespod
Chatter of the week Test message from Delta Airlines
News of the week CSI migration and CSI volume snapshots
AKS Private Clusters in preview
GKE maintenance Windows and exclusions is GA
Google Cloud E2 VMs: introduction and understanding dynamic resource management
New features in Cloud Run for Anthos
Best practices for performing forensics on containers
Infrastructure at Cliqz, and introducing Hydra
Envoy CVEs Istio security bulletin
The Top 3 Service Mesh Developments in 2019 by Zack Jory
Istio Service Mesh Explained in 5 Minutes by Ram Vennam
Ambassador Edge Stack
Solo.io WebAssembly Hub Episode 55, with Idit Levine
Kafka Envoy Protocol Filter
Talos 0.3 beta
AutoTiKV tuning
OpenPolicyAgent's KubeCon recap Episode 42, with John Murray
A first look at Antrea from Alex Brand
TODO: read this article by Patrick DeVivo
Does Testing Kubernetes Conformance Leave You in the Dark? Get Progress Updates as Tests Run by John Schnake
Demystifying Kubernetes as a Service – How Alibaba Cloud Manages 10,000s of Kubernetes Clusters
How Jaeger Helped Grafana Labs Improve Query Performance and Root Out Tough Bugs
Adopting Kubernetes at Quora by Taylor Barrella,
CNCF announces schedule for Bengaluru/Delhi Forums
Links from the interview M3 website
M3: Uber's Open Source, Large-scale Metrics Platform for Prometheus
Before: Graphite and its Whisper database
Prometheus Why pull rather than push?
AlertManager
PromQL
RRDtool
M3 on GitHub: open source from the start
Chronosphere
Rob's 2019 KubeCon's talks: EU: M3 and Prometheus, Monitoring at Planet Scale for Everyone
NA: Deep Linking Metrics and Traces with OpenTelemetry, OpenMetrics and M3
Twitter: Rob Skillington
Martin Mao
M3
Chronosphere