In the second episode of the VOID podcast, Courtney Wang, an SRE at Reddit, said that he was inspired to start writing more in-depth narrative incident reports after reading the write-up of the Slack January 4th, 2021 outage. That incident report, along with many other excellent ones, was penned by Laura Nolan and I've been trying to get her on this podcast since I started it. So, this is a very exciting episode for me. And for you all, it's going to be a bit different because instead of just discussing a single incident that Laura has written about, we get to lean on and learn from her accumulated knowledge doing this for quite a few organizations. And she's come with opinions.  A fun fact about this episode, I was going to title it "Laura Nolan and Control Plane Incidents," but the automated transcription service that I use, which is typically pretty spot on (thanks, Descript!), kept changing "plane" to "pain" and well, you're about to find out just how ironic that actually is... We discussed: A set of incidents she's been involved with that featured some form of control plane or automation as a contributing factor to the incident.What we can learn from fields of study like Resilience Engineering, such as the notion of Joint Cognitive SystemsOther notable incidents that have similar factorsWays that we can better factor in human-computer collaboration in tooling to help  make our lives easier when it comes to handling incidentsReferences:Slack's Outage on Jan 4th 2021A Terrible, Horrible, No-Good, Very Bad Day at SlackGoogle's "satpocalypse"Meta (Facebook) outageReddit Pi-day outageIronies of Automation (Lissane Bainbridge)

What happens when you use your own incident management software to manage your own incidents but said incident takes out your own incident management product? Tune in to find out...We chat with engineer Lawrence Jones about:How their product is designed, and how that both contributed to, but also helped them quickly resolve, the incidentThe role that organizational scaling (hiring lots of folks quickly) can play in making incident response challengingWhat happens when reality doesn't line up with your assumptions about how your system(s) worksThe importance of taking a step back and making sure the team is taking care of each other when you can get a break in the urgency of an incident

In this episode we take a delightful detour from our usual VOID programming to have Emily Ruppe, a Solutions Engineer at Jeli.io and member of the Learning From Incidents (LFI) community, on the program to discuss the upcoming LFI Conference happening in Denver in February. Find out more about the goals and some of the featured speakers for the event, and we hope to see you there!Discussed in this episode:Jeli.ioLearning From IncidentsThe LFI Conference (Feb 15-16, 2023 in Denver, CO)

If you or anyone you know  has listened to Spotify, you're likely familiar with their year end Wrapped tradition. You get a viral, shareable little summary of your favorite songs, albums and artists from the year. In this episode, I chat with Clint Byrum, an engineer whose team helps keep Spotify for Artists running, which in turn keeps well, Spotify running. Each year, the team looks back at the incidents they've had in their own form of Wrapped. They tested hypotheses with incident data that they've collected, found some interesting results and patterns, and helped push their team and larger organization to better understand what they can learn from incidents and how they can make their systems better support artists on their platform.We discussed:Metrics, both good and badMoving away from MTTR after they found it to be unreliableHow incident analysis is akin to archeologyGetting managers/executives interested in incident reviewsThe value of studying near misses along with actual incidents

At the end of January, 2021, a group of Reddit users organized what's called a "short squeeze." They  intended to wreak havoc on hedge funds that were shorting the stock of a struggling brick and mortar game retailer called GameStop. They were coordinating to buy more stock in the company and drive its price further up.In large part, they were successful—at least for a little while. One hedge fund lost somewhere around $2 billion and one Reddit user purportedly made off with around $13 million. Things managed to get even weirder from there, when online trading company Robinhood restricted trading for GameStop shares and sent its values plummeting losing three fourths of its value in just over an hour. But that's less relevant to this episode. What matters is that while all this was happening, traffic to a very specific page on Reddit, called a subreddit, r/wallstreetbets went to the moon. Long after the dust had settled, and the team had a chance to recover and reflect, some of the engineers wrote up an anthology of reports based on the numerous incidents they had that week. We talk to Courtney Wang, Garrett Hoffman, and Fran Garcia about those incidents, and their write-ups, in this episode.A few of the things we discussed include:The precarious dynamic where business successes (traffic surges based on cultural whims) are hard to predict, and can hit their systems in wild and surprising ways.How incidents like these have multiple contributing factors, not all of which are purely technicalHow much they learned about their company's processes, assumptions, organizational boundaries, and other "non-technical" factorsHow people are the source of resilience in these complex sociotechnical systemsCreating psychologically safe environments for people who respond to incidentsTheir motivation for investing so much time and energy into analyzing, writing, and publishing these incident reviewsWhat studying near misses illuminated for them about how their systems workResources mentioned in this episode include:Reddit's r/wallstreetsbets incident anthology, which links to all the reports we discuss."Work as imagined and work as done" by Steven Shorrock (video)

"We no longer felt confident about what the exact operational boundaries of our cluster were supposed to be."In early 2021, observability company Honeycomb dealt with a series of outages related to their Kafka architectural migration, culminating in a 12-hour incident, which is an extremely long outage for the company. In this episode, we chat with two engineers involved in these incidents, Liz Fong-Jones and Fred Hebert, about the backstory that is summarized in this meta-analysis they published in May. We cover a wide range of topics beyond the specific technical details of the incident (which we also discuss), including:Complex socio-technical systems and the kinds of failures that can happen in them (they're always surprises)Transparency and the benefits of companies sharing these outage reportsSafety margins, performance envelopes, and the role of expertise in developing a sense for themHoneycomb's incident response philosophy and processThe cognitive costs of responding to incidentsWhat we can (and can't) learn from incident reportsResources mentioned in the episode:Kafka Migration and Lessons Learned by  HoneycombManaging the Hidden Costs of Coordination by Laura McGuireAbove the Line, Below the Line by Richard Cook"Those found responsible have been sacked": Some observations on the usefulness of error by Richard Cook and Christopher P. NemethPublished in partnership with Indeed.