Courtney Nash

An insider's look at software-related incident reports.

Episode 2: Reddit and the Gamestop Shenanigans
At the end of January, 2021, a group of Reddit users organized what's called a "short squeeze." They  intended to wreak havoc on hedge funds that were shorting the stock of a struggling brick and mortar game retailer called GameStop. They were coordinating to buy more stock in the company and drive its price further up.In large part, they were successful—at least for a little while. One hedge fund lost somewhere around $2 billion and one Reddit user purportedly made off with around $13 million. Things managed to get even weirder from there, when online trading company Robinhood restricted trading for GameStop shares and sent its values plummeting losing three fourths of its value in just over an hour. But that's less relevant to this episode. What matters is that while all this was happening, traffic to a very specific page on Reddit, called a subreddit, r/wallstreetbets went to the moon. Long after the dust had settled, and the team had a chance to recover and reflect, some of the engineers wrote up an anthology of reports based on the numerous incidents they had that week. We talk to Courtney Wang, Garrett Hoffman, and Fran Garcia about those incidents, and their write-ups, in this episode.A few of the things we discussed include:The precarious dynamic where business successes (traffic surges based on cultural whims) are hard to predict, and can hit their systems in wild and surprising ways.How incidents like these have multiple contributing factors, not all of which are purely technicalHow much they learned about their company's processes, assumptions, organizational boundaries, and other "non-technical" factorsHow people are the source of resilience in these complex sociotechnical systemsCreating psychologically safe environments for people who respond to incidentsTheir motivation for investing so much time and energy into analyzing, writing, and publishing these incident reviewsWhat studying near misses illuminated for them about how their systems workResources mentioned in this episode include:Reddit's r/wallstreetsbets incident anthology, which links to all the reports we discuss."Work as imagined and work as done" by Steven Shorrock (video)
Dec 1 2021
44 mins
Episode 1: Honeycomb and the Kafka Migration