Rob Skillington

In this episode of The New Stack Makers, Rob Skillington, co-founder and CTO of Chronosphere, discusses the challenges engineers face in building tools for their organizations. Skillington emphasizes that the "build or buy" decision oversimplifies the issue of tooling and suggests that understanding the abstractions of a project is crucial. Engineers should consider where to build and where to buy, creating solutions that address the entire problem. Skillington advises against short-term thinking, urging innovators to consider the long-term landscape. Drawing from his experience at Uber, Skillington highlights the importance of knowing the audience and customer base, even when they are colleagues. He shares a lesson learned when building a visualization platform for engineers at Uber, where understanding user adoption as a key performance indicator upfront could have improved the project's outcome. Skillington also addresses the "not invented here syndrome," noting its prevalence in organizations like Microsoft and its potential impact on tool adoption. He suggests that younger companies, like Uber, may be more inclined to explore external solutions rather than building everything in-house. The conversation provides insights into Skillington's experiences and the considerations involved in developing internal tools and platforms. Learn more from The New Stack about Software Engineering, Observability, and Chronosphere: Cloud Native Observability: Fighting Rising Costs, Incidents A Guide to Measuring Developer Productivity  4 Key Observability Best Practices

Rob Skillington

CTO

Chronosphere

DETROIT — Rob Skillington’s grandfather was a civil engineer, working in an industry that, in over a century, developed processes and know-how that enabled the creation of buildings, bridges and road.   “A lot of those processes matured to a point where they could reliably build these things,” said Skillington, co-founder and chief technology officer at Chronosphere, an observability platform. “And I think about observability as that same maturity of engineering practice. When  it comes to building software that actually is useful in the world, it is this process that helps you actually achieve the deployment and operation of these large scale systems that we use every day.”   Skillington spoke about the evolution of observability, and his company’s recent donation of an open source project to Prometheus, in this episode of The New Stack Makers podcast. Heather Joslyn, features editor of TNS, hosted the conversation.   This On the Road edition of The New Stack Makers was recorded at KubeCon + CloudNativeCon North America, in the Motor City. The episode was sponsored by Chronosphere. A Donation to the Prometheus Project Helping observability practices grow as mature and reliable as civil engineering rules that help build sturdy skyscrapers is a tough task, Skillington suggested.   In the cloud era, he said, “you have to really prepare the software for a whole set of runtime environments. And so the challenges around that is really about making it consistent, well understood and robust.”   At KubeCon in late October, Chronosphere and PromLabs (founded by Julius Volz, creator of Prometheus) announced that they had donated their open source project PromLens to the Prometheus project, the open source monitoring and alerts primitive.   The donation is a way of placing a bet on a tool that integrates well with Kubernetes. “There's this real yearning for essentially a standard that can be built upon by everyone in the industry, when it comes to these core primitives, essentially,” Skillington said. “And Prometheus is one of those primitives. We want to continue to solidify that as a primitive that stands the test of time.”   “We can't build a self-driving car if we're always building a different car,” he added.   PromLens builds Prometheus queries in a sort of integrated development environment (IDE), Skillington said. It also makes it easier for more people in an organization to create queries and understand the meaning and seriousness of alerts.   The PromLens tool breaks queries into a visual format, and allows users to edit them through a UI. “Basically, it's kind of like a What You See Is What You Get editor, or  WYSIWYG editor, for Prometheus queries,” Skillington said.   “Some of our customers have tens of thousands of these alerts to find in PromQL, which is the query language for Prometheus,” he noted. “Having a tool like an integrated development environment — where you can really understand these complex queries and iterate faster on, setting these up and getting back to your day job — is incredibly important.”   Check out the full episode for more on PromLens and the current state of observability.

Rob Skillington

CTO

Chronosphere

Summary Collecting and processing metrics for monitoring use cases is an interesting data problem. It is eminently possible to generate millions or billions of data points per second, the information needs to be propagated to a central location, processed, and analyzed in timeframes on the order of milliseconds or single-digit seconds, and the consumers of the data need to be able to query the information quickly and flexibly. As the systems that we build continue to grow in scale and complexity the need for reliable and manageable monitoring platforms increases proportionately. In this episode Rob Skillington, CTO of Chronosphere, shares his experiences building metrics systems that provide observability to companies that are operating at extreme scale. He describes how the M3DB storage engine is designed to manage the pressures of a critical system component, the inherent complexities of working with telemetry data, and the motivating factors that are contributing to the growing need for flexibility in querying the collected metrics. This is a fascinating conversation about an area of data management that is often taken for granted. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management When you’re ready to build your next pipeline, or want to test out the projects you hear about on the show, you’ll need somewhere to deploy it, so check out our friends at Linode. With their managed Kubernetes platform it’s now even easier to deploy and scale your workflows, or try out the latest Helm charts from tools like Pulsar and Pachyderm. With simple pricing, fast networking, object storage, and worldwide data centers, you’ve got everything you need to run a bulletproof data platform. Go to dataengineeringpodcast.com/linode today and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Modern Data teams are dealing with a lot of complexity in their data pipelines and analytical code. Monitoring data quality, tracing incidents, and testing changes can be daunting and often takes hours to days. Datafold helps Data teams gain visibility and confidence in the quality of their analytical data through data profiling, column-level lineage and intelligent anomaly detection. Datafold also helps automate regression testing of ETL code with its Data Diff feature that instantly shows how a change in ETL or BI code affects the produced data, both on a statistical level and down to individual rows and values. Datafold integrates with all major data warehouses as well as frameworks such as Airflow & dbt and seamlessly plugs into CI workflows. Go to dataengineeringpodcast.com/datafold today to start a 30-day trial of Datafold. Once you sign up and create an alert in Datafold for your company data, they will send you a cool water flask. Today’s episode of Data Engineering Podcast is sponsored by Datadog, the monitoring and analytics platform for cloud-scale infrastructure and applications. Datadog’s machine-learning based alerts, customizable dashboards, and 400+ vendor-backed integrations makes it easy to unify disparate data sources and pivot between correlated metrics and events for faster troubleshooting. By combining metrics, traces, and logs in one place, you can easily improve your application performance. Try Datadog free by starting a your 14-day trial and receive a free t-shirt once you install the agent. Go to dataengineeringpodcast.com/datadog today see how you can unify your monitoring today. Your host is Tobias Macey and today I’m interviewing Rob Skillington about Chronosphere, a scalable, reliable and customizable monitoring-as-a-service purpose built for cloud-native applications. Interview Introduction How did you get involved in the area of data management? Can you start by describing what you are building at Chronosphere and your motivation for turning it into a business? What are the biggest challenges inherent to monitoring use cases? How does the advent of cloud native environments complicate things further? While you were at Uber you helped to create the M3 storage engine. There are a wide array of time series databases available, including many purpose built for metrics use cases. What were the missing pieces that made it necessary to create a new system? How do you handle schema design/data modeling for metrics storage? How do the usage patterns of metrics systems contribute to the complexity of building a storage layer to support them? What are the optimizations that need to be made for the read and write paths in M3? How do you handle high cardinality of metrics and ad-hoc queries to understand system behaviors? What are the scaling factors for M3? Can you describe how you have architected the Chronosphere platform? What are the convenience features built on top of M3 that you are creating at Chronosphere? How do you handle deployment and scaling of your infrastructure given the scale of the businesses that you are working with? Beyond just server infrastructure and application behavior, what are some of the other sources of metrics that you and your users are sending into Chronosphere? How do those alternative metrics sources complicate the work of generating useful insights from the data? In addition to the read and write loads, metrics systems also need to be able to identify patterns, thresholds, and anomalies in the data to alert on it with minimal latency. How do you handle that in the Chronosphere platform? What are some of the most interesting, innovative, or unexpected ways that you have seen Chronosphere/M3 used? What are some of the most interesting, unexpected, or challenging lessons that you have learned while building Chronosphere? When is Chronosphere the wrong choice? What do you have planned for the future of the platform and business? Contact Info LinkedIn @roskilli on Twitter robskillington on GitHub Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don’t forget to check out our other show, Podcast.__init__ to learn about the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at dataengineeringpodcast.com/chat Links Chronosphere Lidar Cloud Native M3DB OpenTracing Metrics/Telemetry Graphite Podcast.__init__ Episode InfluxDB Clickhouse Podcast Episode Prometheus Inverted Index Druid Cardinality Apache Flink Podcast Episode HDFS Avro Podcast Episode Grafana Tecton Podcast Episode Datadog Podcast Episode Kubernetes Sourcegraph The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA Support Data Engineering Podcast

Rob Skillington

CTO

Chronosphere

Martin Mao and Rob Skillington are co-founders of Chronosphere; CEO and CTO respectively. They both worked on the monitoring team at Uber, where they created M3: a metrics platform with an open source time-series database built for scale. They join Craig and Adam to talk about monitoring, metrics and M3 on the last episode of 2019. Do you have something cool to share? Some questions? Let us know: web: kubernetespodcast.com mail: kubernetespodcast@google.com twitter: @kubernetespod Chatter of the week Test message from Delta Airlines News of the week CSI migration and CSI volume snapshots AKS Private Clusters in preview GKE maintenance Windows and exclusions is GA Google Cloud E2 VMs: introduction and understanding dynamic resource management New features in Cloud Run for Anthos Best practices for performing forensics on containers Infrastructure at Cliqz, and introducing Hydra Envoy CVEs Istio security bulletin The Top 3 Service Mesh Developments in 2019 by Zack Jory Istio Service Mesh Explained in 5 Minutes by Ram Vennam Ambassador Edge Stack Solo.io WebAssembly Hub Episode 55, with Idit Levine Kafka Envoy Protocol Filter Talos 0.3 beta AutoTiKV tuning OpenPolicyAgent's KubeCon recap Episode 42, with John Murray A first look at Antrea from Alex Brand TODO: read this article by Patrick DeVivo Does Testing Kubernetes Conformance Leave You in the Dark? Get Progress Updates as Tests Run by John Schnake Demystifying Kubernetes as a Service – How Alibaba Cloud Manages 10,000s of Kubernetes Clusters How Jaeger Helped Grafana Labs Improve Query Performance and Root Out Tough Bugs Adopting Kubernetes at Quora by Taylor Barrella, CNCF announces schedule for Bengaluru/Delhi Forums Links from the interview M3 website M3: Uber's Open Source, Large-scale Metrics Platform for Prometheus Before: Graphite and its Whisper database Prometheus Why pull rather than push? AlertManager PromQL RRDtool M3 on GitHub: open source from the start Chronosphere Rob's 2019 KubeCon's talks: EU: M3 and Prometheus, Monitoring at Planet Scale for Everyone NA: Deep Linking Metrics and Traces with OpenTelemetry, OpenMetrics and M3 Twitter: Rob Skillington Martin Mao M3 Chronosphere

Martin Mao

CEO

Chronosphere

Rob Skillington

CTO

Chronosphere