{"id":71385,"date":"2023-02-20T09:03:45","date_gmt":"2023-02-20T09:03:45","guid":{"rendered":"https:\/\/www.cryptocabaret.com\/?p=71385"},"modified":"2023-02-20T09:03:45","modified_gmt":"2023-02-20T09:03:45","slug":"4-questions-open-source-engineers-should-ask-to-mitigate-risk-at-scale","status":"publish","type":"post","link":"https:\/\/www.cryptocabaret.com\/?p=71385","title":{"rendered":"4 questions open source engineers should ask to mitigate risk at scale"},"content":{"rendered":"

4 questions open source engineers should ask to mitigate risk at scale<\/span>
\nkathryn.xtang@\u2026<\/a><\/span>
\nMon, 02\/20\/2023 – 03:00<\/span><\/p>\n

\n

At Shopify, we use and maintain a lot of open source projects<\/a>, and every year we prepare for Black Friday Cyber Monday<\/a> (BFCM) and other high-traffic events to make sure our merchants can sell to their buyers. To do this, we built an infrastructure platform at a large scale<\/a> that is highly complex, interconnected, globally distributed, requiring thoughtful technology investments from a network of teams. We\u2019re changing how the internet works, where no single person can oversee the full design and detail at our scale.<\/p>\n

Over BFCM 2022, we served 75.98M requests per minute to our commerce platform at peak. That\u2019s 1.27M requests per second. Working at this massive scale in a complex and interdependent system, it would be impossible to identify and mitigate every possible risk. This article breaks down a high-level risk mitigation process into four questions that can be applied to nearly any scenario to help you make the best use of your time and resources available.<\/p>\n

1. What are the risks?<\/h2>\n

To inform mitigation decisions, you must first understand the current state of affairs. We expand our breadth of knowledge by learning from people from all corners of the platform. We run \u201cwhat could go wrong\u201d (WCGW) exercises where anyone building or interested in infrastructure can highlight a risk. These can be technology risks, operational risks, or something else. Having this unfiltered list is a great way to get a broad understanding of what could<\/em> happen.<\/p>\n

The goal here is visibility<\/strong>.<\/p>\n

2. What is worth mitigating?<\/h2>\n

Great brainstorming leaves us with a large and daunting list of risks. With limited time to fix things, the key is to prioritize what is most important to our business. To do this, we vote on risks, then gather technical experts to discuss highest ranked risks in more detail, including their likelihood and severity. We make decisions about what and how to mitigate, and which team will own each action item.<\/p>\n

The goal here is to optimize\u00a0how we spend our time<\/strong>.<\/p>\n

3. Who makes what decisions?<\/h2>\n

In any organization, there are times when waiting for a perfect consensus is not possible or not effective. Shopify moves tremendously fast because we make sure to identify decision makers, then empower them to gather input, weigh risks\/rewards, and come to a decision. Often the decision is best made by the subject matter expert or who bears the most benefit or repercussions of whatever direction we choose.<\/p>\n

The goal here is to align incentives and accountability<\/strong>.<\/p>\n<\/p>\n

\n
Our favorite resources about open source<\/div>\n
\n
Git cheat sheet<\/a><\/div>\n
Advanced Linux commands cheat sheet<\/a><\/div>\n
Open source alternatives<\/a><\/div>\n
Free online course: RHEL technical overview<\/a><\/div>\n
Check out more cheat sheets<\/a><\/div>\n<\/p><\/div>\n<\/p><\/div>\n

4. How do you communicate?<\/h2>\n

We move fast but still need to keep stakeholders and close collaborators informed. We summarize key findings and risks from our WCGW exercises so that we all land on the same page about our risk profile. This may include key risks or single points of failure. We over-communicate so that we\u2019re aligned and aware and stakeholders have opportunities to interject.<\/p>\n

The goal here is alignment and awareness<\/strong>.<\/p>\n

Solving the right things when there is uncertainty<\/h2>\n

Underlying all these questions is the uncertainty in our working environment. You never have all the facts or know exactly which components will fail when and how. The best way to deal with uncertainty is by using probability.<\/p>\n

Expert poker players know that great bets don\u2019t always yield great outcomes, and bad bets don\u2019t always yield bad outcomes. What\u2019s important is to bet on the probability of outcomes, where over enough rounds, your results will converge to expectation. The same applies in engineering, where we constantly make bets and learn from them. Great bets require clearly distinguishing the quality of your decisions versus outcomes. It means not over-indexing on bad decisions that led to lucky outcomes or great decisions that happen to run into very unlucky scenarios.<\/p>\n

Knowing that we can\u2019t control everything also helps us stay calm, which is vital for us to practice good judgment in high-pressure situations.<\/p>\n

When it comes to BFCM (and life in general), no one can predict the future or fully protect against all risks. The question is, what would you change looking back? In hindsight, would you feel confident that you prioritized the most important things and made thoughtful bets using the information available? Did you facilitate meaningful discussions with the right people? Could you justify your actions to your customers and their customers?<\/p>\n

\n

This article originally appeared on Planning in Bets: Risk Mitigation at Scale<\/a> and is republished with permission.<\/em><\/p>\n<\/div>\n

\n

What do you do with a finite amount of time to deal with an infinite number of things that can go wrong?\u00a0<\/p>\n<\/div>\n

\n
\n
\"A<\/div>\n
Image by: <\/span><\/p>\n

Opensource.com<\/p>\n<\/div>\n<\/article>\n<\/div>\n

\n
Business<\/a><\/div>\n
DevOps<\/a><\/div>\n
SCaLE<\/a><\/div>\n<\/p><\/div>\n
What to read next<\/div>\n

\n \"Creative<\/a>This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.<\/div>\n
\n
\n
Register<\/a> or Login<\/a> to post a comment.<\/div>\n<\/p><\/div>\n<\/section>\n

Powered by WPeMatico<\/a><\/small><\/p>\n","protected":false},"excerpt":{"rendered":"

4 questions open source engineers should ask to mitigate risk at scale kathryn.xtang@\u2026 Mon, 02\/20\/2023 – 03:00 At Shopify, we use and maintain a lot of open source projects, and every year we prepare for Black Friday Cyber Monday (BFCM) and other high-traffic events to make sure our merchants can sell to their buyers. To […]<\/p>\n","protected":false},"author":1,"featured_media":71386,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[307],"tags":[],"class_list":["post-71385","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-open-source"],"_links":{"self":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts\/71385","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=71385"}],"version-history":[{"count":0,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/posts\/71385\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=\/wp\/v2\/media\/71386"}],"wp:attachment":[{"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=71385"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=71385"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.cryptocabaret.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=71385"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}