Fact of the Day 1

Share this post

October 20: Amazon’s Correction of Error Mechanism (Part 2)

www.factoftheday1.com

Discover more from Fact of the Day 1

My journey as a startup founder
Over 38,000 subscribers
Continue reading
Sign in

October 20: Amazon’s Correction of Error Mechanism (Part 2)

Let’s continue with the example from yesterday where a large server failure takes down the Amazon.com store.

Danny Sheridan
Oct 20, 2021
6
Share this post

October 20: Amazon’s Correction of Error Mechanism (Part 2)

www.factoftheday1.com
2
Share

Suggest a topic for a future fact.

Source: Dave Anderson, Scarlet Ink
Published: June 2021

Amazon’s Correction of Error Mechanism (Part 2)

Let’s continue with the example from yesterday where a large server failure takes down the Amazon.com store. Once the 5 Whys identify the root cause of the issue, action items are assigned.

5 Whys

  1. Why did the site go down? All the servers were turned off.

  2. Why were the servers turned off? A script was run which shut them off.

  3. Why was a script able to turn off so many servers? There are no permission differences between turning off 1 server vs many.

  4. Why did the team not notice immediately? There were no alarms/notifications in place.

  5. Why was someone shutting down servers? Because hardware was being replaced.

Action items

  1. Write a script for restarting servers ASAP, owned by __ employee.

  2. Improve existing alarms by the end of the week, owned by __ employee.

  3. Design automated recovery system for future outages by the end of the quarter, owned by __ employee.

Leave a comment

6
Share this post

October 20: Amazon’s Correction of Error Mechanism (Part 2)

www.factoftheday1.com
2
Share
2 Comments
Share this discussion

October 20: Amazon’s Correction of Error Mechanism (Part 2)

www.factoftheday1.com
Jim
Oct 21, 2021

Great post as always. Don't we need to evaluate the cost of these action items? Thanks

Expand full comment
Reply
Share
1 reply by Danny Sheridan
1 more comment...
Top
New
Community

No posts

Ready for more?

© 2023 Fact of the Day 1
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great writing