Observability: the present and future, with Charity Majors
Supported by Our Partners
• Sonar — Trust your developers – verify your AI-generated code.
• Vanta —Automate compliance and simplify security with Vanta.
—
In today's episode of The Pragmatic Engineer, I'm joined by Charity Majors, a well-known observability expert – as well as someone with strong and grounded opinions. Charity is the co-author of "Observability Engineering" and brings extensive experience as an operations and database engineer and an engineering manager. She is the cofounder and CTO of observability scaleup Honeycomb.
Our conversation explores the ever-changing world of observability, covering these topics:
• What is observability? Charity’s take
• What is “Observability 2.0?”
• Why Charity is a fan of platform teams
• Why DevOps is an overloaded term: and probably no longer relevant
• What is cardinality? And why does it impact the cost of observability so much?
• How OpenTelemetry solves for vendor lock-in
• Why Honeycomb wrote its own database
• Why having good observability should be a prerequisite to adding AI code or using AI agents
• And more!
—
Timestamps
(00:00) Intro
(04:20) Charity’s inspiration for writing Observability Engineering
(08:20) An overview of Scuba at Facebook
(09:16) A software engineer’s definition of observability
(13:15) Observability basics
(15:10) The three pillars model
(17:09) Observability 2.0 and the shift to unified storage
(22:50) Who owns observability and the advantage of platform teams
(25:05) Why DevOps is becoming unnecessary
(27:01) The difficulty of observability
(29:01) Why observability is so expensive
(30:49) An explanation of cardinality and its impact on cost
(34:26) How to manage cost with tools that use structured data
(38:35) The common worry of vendor lock-in
(40:01) An explanation of OpenTelemetry
(43:45) What developers get wrong about observability
(45:40) A case for using SLOs and how they help you avoid micromanagement
(48:25) Why Honeycomb had to write their database
(51:56) Companies who have thrived despite ignoring conventional wisdom
(53:35) Observability and AI
(59:20) Vendors vs. open source
(1:00:45) What metrics are good for
(1:02:31) RUM (Real User Monitoring)
(1:03:40) The challenges of mobile observability
(1:05:51) When to implement observability at your startup
(1:07:49) Rapid fire round
—
The Pragmatic Engineer deepdives relevant for this episode:
• How Uber Built its Observability Platform https://newsletter.pragmaticengineer.com/p/how-uber-built-its-observability-platform
• Building an Observability Startup https://newsletter.pragmaticengineer.com/p/chronosphere
• How to debug large distributed systems https://newsletter.pragmaticengineer.com/p/antithesis
• Shipping to production https://newsletter.pragmaticengineer.com/p/shipping-to-production
—
See the transcript and other references from the episode at https://newsletter.pragmaticengineer.com/podcast
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.
Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe
Observability & Engineering Management, with Charity Majors
Charity Majors is the co-founder and CTO of honeycomb.io. She pioneered the concept of modern Observability, drawing on her years of experience building and managing massive distributed systems at Parse (acquired by Facebook), then subsequently at Facebook, and at Linden Lab building Second Life. She is the co-author of Observability Engineering and Database Reliability Engineering (O'Reilly). She loves free speech, free software and single malt scotch.
Do you have something cool to share? Some questions? Let us know:
- web: kubernetespodcast.com
- mail: kubernetespodcast@google.com
- twitter: @kubernetespod
News of the week CNCF Blog: Vitess 20 is now Generally Available
Vitess Blog: Announcing Vitess 20
Anthropic Blog: Claude 3.5 Sonnet
KubeCon India 2024 CFP
Apps on Azure Blog: Announcing support of OCI v1.1 specification in Azure Container Registry
VMware Tanzu Blog: Announcing VMware Tanzu Greenplum 7.2: Powering Your Business with Enhanced Performance and Advanced Capabilities
VMware Tanzu Blog: Join the public beta for GenAI on Tanzu Platform today!
CNCF: Adobe End User Journey Report
Links from the interview Honeycomb.io
O'Reilly Book: Observability Engineering
O'Reilly Book: Database Reliability Engineering
Charity's blog site: charity.wtf
Charity Blog: Questionable Advice: "My boss says we don't need any engineering managers. Is he right?"
Daniel H. Pink book: "Drive: The Surprising Truth About What Motivates Us"
In which, "He examines the three elements of true motivation—autonomy, mastery, and purpose-and offers smart and surprising techniques for putting these into action in a unique book that will change how we think and transform how we live."
Charity blog on Stack Overflow: "Generative AI is not going to build your engineering team for you"
In which she talks about how the tech industry is an apprenticeship industry.
Charity Majors in the Google Cloud Next 2024 Developer Keynote
honeycomb.io blog: "How Time Series Databases Work—And Where They Don't" by Alex Vondrak
honeycomb.io blog: "Why Observability Requires a Distributed Column Store" by Alex Vondrak
Links from the post-interview chat CNCF Kubernetes Community Days (KCDs)
CNCF Kubernetes Community Days (KCDs) on GitHub
Julia Evans Blog
Wizard Zines by Julia Evans
"Help! I Have a Manager!" zine by Julia Evans
Aja Hammerly aka "thagomizer" blog
"The Toaster Parable"
"Manager Toolkit: Manage The Person In Front Of You"
"Manager Toolkit: Useful Manager Phrases for 1:1s"
"Manager Toolkit: You Talk, I Type"
Shifting from Observability 1.0 to 2.0 with Charity Majors
This week on Screaming in the Cloud, Corey is joined by good friend and colleague, Charity Majors. Charity is the CTO and Co-founder of Honeycomb.io, the widely popular observability platform. Corey and Charity discuss the ins and outs of observability 1.0 vs. 2.0, why you should never underestimate the power of software to get worse over time, and the hidden costs of observability that could be plaguing your monthly bill right now. The pair also shares secrets on why speeches get better the more you give them and the basic role they hope AI plays in the future of computing. Check it out!
Show Highlights:
(00:00 - Reuniting with Charity Majors: A Warm Welcome
(03:47) - Navigating the Observability Landscape: From 1.0 to 2.0
(04:19) - The Evolution of Observability and Its Impact
(05:46) - The Technical and Cultural Shift to Observability 2.0
(10:34) - The Log Dilemma: Balancing Cost and Utility
(15:21) - The Cost Crisis in Observability
(22:39) - The Future of Observability and AI's Role
(26:41) - The Challenge of Modern Observability Tools
(29:05) - Simplifying Observability for the Modern Developer
(30:42) - Final Thoughts and Where to Find More
About Charity
Charity is an ops engineer and accidental startup founder at honeycomb.io. Before this she worked at Parse, Facebook, and Linden Lab on infrastructure and developer tools, and always seemed to wind up running the databases. She is the co-author of O'Reilly's Database Reliability Engineering, and loves free speech, free software, and single malt scotch.
Links:
https://charity.wtf/
Honeycomb Blog: https://www.honeycomb.io/blog
Twitter: @mipsytipsy
Charity Majors: Taking an Outsider's Approach to a Startup
In the early 2000s, Charity Majors was a homeschooled kid who’d gotten a scholarship to study classical piano performance at the University of Idaho.
“I realized, over the course of that first year, that music majors tended to still be hanging around the music department in their 30s and 40s,” she said. “And nobody really had very much money, and they were all doing it for the love of the game. And I was just like, I don't want to be poor for the rest of my life.”
Fortunately, she said, it was pretty easy at that time to jump into the much more lucrative tech world. “It was buzzing, they were willing to take anyone who knew what Unix was,” she said of her first tech job, running computer systems for the university.
Eventually, she dropped out of college, she said, “made my way to Silicon Valley, and I’ve been here ever since.”
Majors, co-founder and chief technology officer of the six-year-old Honeycomb.io, an observability platform company, told her story for The New Stack’s podcast series, The Tech Founder Odyssey, which spotlights the personal journeys of some of the most interesting technical startup creators in the cloud native industry.
It’s been a busy year for her and the company she co-founded with Christine Yen, a colleague from Parse, a mobile application development company that was bought by Facebook. In May, O’Reilly published “Observability Engineering,” which Majors co-wrote with George Miranda and Liz Fong-Jones. In June, Gartner named Honeycomb.io as a Leader in the Magic Quadrant for Application Performance Monitoring and Observability.
Thus far Honeycomb.io, now employing about 200 people, has raised just under $97 million, including a $50 million Series C funding round it closed in October, led by Insight Partners (which owns The New Stack).
This Tech Founder Odyssey conversation was co-hosted by Colleen Coll and Heather Joslyn of TNS.
‘Rage-Driven Development’
Honeycomb.io grew from efforts at Parse to solve a stubborn observability problem: systems crashed frequently, and rarely for the same reasons each time. “We invested a lot in the last generation of monitoring technology, we had all these dashboards, we have all these graphs,” Majors said. “But in order to figure out what's going on, you kind of had to know in advance what was going to break.”
Once Parse was acquired by Facebook, Majors, Yen and their teams began piping data into a Facebook tool called Scuba, which ”was aggressively hostile to users,” she recalled.
But, “it did one thing really well, which is let you slice and dice in real time on dimensions that have very high cardinality,” meaning those that contain lots of unique terms. This set it apart from the then-current monitoring technologies, which were built around assessing low cardinality dimensions.
Scuba allowed Majors’ organization to gain more control over its reliability problem. And it got her and Yen thinking about how a platform tool that could analyze high cardinality data about system health in real time. “Everything is a high cardinality dimension now,” Majors said. “And [with] the old generation of tools, you hit a wall really fast and really hard.”
And so, Honeycomb.io was created to build that platform. “My entire career has been rage-driven development,” she said. “Like: sounds cool, I'm gonna go play with that. This isn't working — I'm gonna go fix it from anger.”
A Reluctant CEO
Yen now holds the CEO role at Honeycomb.io, but Majors wound up with the job for roughly the first half of the company’s life.
Did Majors like being the boss? “Hated it,” she said. “Constitutionally what you want in a CEO is someone who is reliable, predictable, dependable, someone who doesn't mind showing up every Tuesday at 10:30 to talk to the same people.
“I am not structured. I really chafe against that stuff.”
However, she acknowledged, she may have been the right leader in the startup’s beginning: “It was a state of chaos, like we didn't think we were going to survive. And that's where I thrive.”
Fortunately, in Honeycomb.io’s early days, raising money wasn’t a huge challenge, due to its founders’ background at Facebook. “There were people who were coming to us, like, do you want $2 million for a seed thing? Which is good, because I've seen the slides that we put together, and they are laughable. If I had seen those slides as an investor, I would have run the other way.”
The “pedigree” conferred on her by investors due to her association with Facebook didn’t sit comfortably with her. “I really hated it,” she said. “Because I did not learn to be a better engineer at Facebook. And part of me kind of wanted to just reject it. But I also felt this like responsibility on behalf of all dropouts, and queer women everywhere, to take the money and do something with it. So that worked out.”
Majors, a frequent speaker at tech conferences, has established herself as a thought leader in not only observability but also engineering management. For other women, people of color, or people in the tech field with an unconventional story, she advised “investing a little bit in your public speaking skills, and making yourself a bit of a profile. Being externally known for what you do is really helpful because it counterbalances the default assumptions that you're not technical or that you're not as good.”
She added, “if someone can Google your name plus a technology, and something comes up, you're assumed to be an expert. And I think that that really works to people's advantage.“
Majors had a lot more to say about how her outsider perspective has shaped the way she approaches hiring, leadership and scaling up her organization. Check out this latest episode of the Tech Founder Odyssey.
Managing Humans with Charity Majors
Links Referenced:
Honeycomb: https://www.honeycomb.io/
Personal Blog: https://charity.wtf/
Honeycomb Blog: https://www.honeycomb.io/blog
Episode 195: Ad-hoc promotion and quitting a huge company with Charity Majors
<p>We’re excited to have special guest <a href="https://twitter.com/mipsytipsy">Charity Majors</a> on the show! Charity is the CTO and former CEO of <a href="https://www.honeycomb.io">Honeycomb</a>. She has worked at Second Life, Parse, Facebook, and more. She blogs at <a href="https://charity.wtf/">charity.wtf</a>.</p>
<p>Dave, Jamison, and Charity answer these questions:</p>
<ol>
<li>
<p>I’ve had the role of tech lead informally for the past two years at a fast-growing tech startup. We were a team of 6 developers, and now we are 16. Recently, we had a department meeting in which the Software Development VP communicated that we have 3 teams and I was the tech lead of two of them. I was surprised. He hasn’t mentioned his decision of splitting the teams nor that I’ve been officially promoted to tech lead. I was expecting a one-on-one where he would “pop the question”: Will you be my tech lead?</p>
<p>I asked him privately if that meant I would be officially promoted and would have my title changed. He said that he was going to have this conversation with the HR Manager and would get back to me, but potentially.</p>
<p>He doesn’t spend time on one-on-ones, nor is he very good at managing people although he’s good technically. How weird is this situation? A manager tells his team that they now have a tech lead along some org changes. I haven’t been informed, haven’t had my title changed yet, and haven’t been offered a raise yet.</p>
</li>
<li>
<p>Hi! I love your show and have been listening to it almost since day one. I was an engineer for about 10 years, and I’ve been a manager for about 1 year, and I love my team. They’re high performers, we have a high level of trust. I also like my boss! But the larger org has some issues, and in time-honored Soft Skills Engineering tradition, I plan to quit. I would like to stay in management. So I have these questions:</p>
<p>1) My employer is a very large public company. How much should I care about negative headlines and Wall Street’s opinion?</p>
<p>2) How long should I stay in my role as a manager before looking for a new job?</p>
<p>3) How do I message this to my team when I leave?</p>
</li>
</ol>
Honeycomb, Complex Systems, Saving Sanity
Charity Majors joined the show to talk about debugging complex systems, using go to save one’s sanity, hiring smart people who can learn, and collectively working to make “on-call” life not miserable.
Join the discussion
Changelog++ members support our work, get closer to the metal, and make the ads disappear. Join today!
Sponsors:
Linode – Our cloud server of choice. Get one of the fastest, most efficient SSD cloud servers for only $5/mo. Use the code changelog2017 to get 4 months free!
Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform.
Toptal – Scale your team and hire from the top 3% of developers and designers with Toptal. Email adam@changelog.com for a personal introduction.
Compose – Production ready, cloud hosted databases. Pick your flavor - MongoDB, Elasticsearch, RethinkDB, Redis, Postgres, etcd, or RabbitMQ. When you’re ready to sign up use our special URL compose.com/changelog to get 60-days free on Compose
Featuring:
Charity Majors – Website, GitHub, X
Erik St. Martin – GitHub, X
Carlisia Thompson – GitHub, LinkedIn, X
Brian Ketelsen – GitHub, X
Show Notes:
Honeycomb :: Powerful, Exploratory Learning with Richer Data
go package libhoney (it’s the APM of the future!)
How We Moved Our API From Ruby to Go and Saved Our Sanity
CHARITY.WTF
Database Reliability Engineering book
Interesting Go Projects and News
Charity wants to give big shout outs (shouts out?) to Naitik Shah and Matt Silverlock!
Go 1.8 is released
Implementing a Debugger: The Fundamentals
Building a Go Debugger
Gobot - 1.2 Released
Pixterm - Draw images in your ANSI terminal with true color
1.8 Release Parties Everywhere
Change to Go CoC
Free Software Friday!
Each week on the show we give a shout out to an open source project or community that’s made an impact in our day to day developer lives.
Brian - Eclipse Che
Erik - Kube-Lego
Carlisia - Visual Studio Code
Something missing or broken? PRs welcome!
