5 Lessons about Metrics from Amazon
5 Lessons about Metrics from Amazon
You Can't Manage What You Can't Measure And You Can't Measure What You Can't Manage
For those following along at home with my 30 Days of Stream of Consciousness Essay Writing (thanks!!), you might recall one from a few days ago where I blasted out 61 Lessons I learned (one a day) while working at Amazon Go one summer a few years ago.
I think Amazon does Metrics really well. Like scary well. Mostly.
This impressed me since I think that it’s something a lot of people and companies do pretty poorly and it’s v important. like VVV important. As a public service (and definitely not because I’m currently trying to figure out what metrics matter in Web3 startups in order to value them properly), I decided to pull out the lessons most closely related to metrics and to give those a deeper treatment, add some context, and maybe put together a few actionable takeaways you, dear reader. Here we go!
Metric Selection Really Drives Decision-Making
You have to figure this piece out first before you make any real commitments otherwise you’ll end up building your entire machine around hitting targets that don’t matter.
The Context:
I watched this play out in great and terrible ways. The logistics team had a phenomenal understanding of the core metrics surrounding how you reduce costs (I mean duh, it’s Amazon). I heard people talk time and time again about reducing inventory touches which everyone understood to be one of the most significant cost drivers in the supply chain. Great!
In other teams though, an alphabet soup of acronyms for metrics that people knew the letters of, but didn’t know the numerator or denominator or even what the acronym might stand for! This combined with a dizzying barrage of reporting on 15+ metrics during the weekly business review meetings (all given relatively equal weight and airtime) makes for a tough environment in which to get oriented on what really matters.
How can you make quality decisions about important things without knowing what’s important?
The Takeaway:
You can’t manage what you can’t measure sometimes turns into you you don’t manage anything you’re measuring if you try to manage everything under the sun… Strategy is about choice and your choice of metrics is your decision to focus on what elements of operational effectiveness best help support your strategy. Focusing on the metrics that inform you about the strategic decisions you should be making is paramount to your success.
Metrics Should Change with your Level and Scope
Tactical decision makers need to think about different metrics than strategic decision makers. Everyone should be universally aware of business drivers, but should explicitly have different metrics they are responsible for reporting on and improving.
The Context:
There’s piles and piles written about having the one most important metric in a company and that’s broadly true, but as organizations gain complexity theirs metrics sometimes need to adjust to reflect that complexity. For any project or vertical within Amazon there is a veritable encyclopedia of previous metrics that teams can look up and leverage and who is/was responsible for driving that metric and how they did so. Repeated trials have shown Amazonians that different types of employees (Instock vs Category vs Pricing many others) have different scopes of control (levers that they can use to change the numbers like negotiating hard or changing what items appear at the top of the page) and when you can tie their metric to their span of control, great results can happen. Conversely, if you tie a metric they can’t control to someone ← nothing will happen lol.
The Takeaway:
Don’t think of just raw metrics. Think of Input Metrics and Output Metrics. In a basic form -
Price x Quantity Sold = Revenue
Price and Quantity Sold are the Input Metrics, to the Output Metric of Revenue.
Lower level Team members should be tied to individual Input Metrics while more senior levels, who are in charge of the Pricing and Quantity teams, should be tied to the Output metric.
The trick here is to realize that every Output Metric is an Input Metric for someone else so it’s actually Inputs all the way up until you get to Bezos…oh wait I mean Jassy.
Tell Narratives Grounded with Metrics and End States
As long as you’re not trapped in Office Space in your day to day, most of your meetings are hopefully valuable. Many of them likely involve you reporting about something you’re in charge of. It’s crucial to remember that the important part is the story behind the numbers not the numbers themselves.
The Context:
I saw some poor souls who entered a meeting where they were the star of the show and they told an amazing story…with no numbers to back it up. Likewise, I watched some brilliant people absolutely blow everyone’s mind with their complex calculations, but have no coherent story to connect the dots throughout. Oof.
The Takeaway:
Recognize that often you’re supposed to do the analysis for others to digest and we invented cooking for a reason. There’s a balance to strike between the qualitative and quantitative aspects of it (Amazon definitely skewed quantitative), but no big Amazon initiative gets done with a faux Press Release and PR FAQ.
Use the numbers to tell the story, but never forget that.
Always Do a Basic Visual (like a Scatterplot)
Whenever you create something that someone seems interested in, just do it.
The Context:
I spent a year as an Aide in the military. I engaged closely with some relatively senior Amazon leaders. I have had pretty good exposure to people at high levels of organizations and man, people respond to pretty pictures. It lowers their cognitive processing load (which is already probably overwhelmed), reduces the time they need to make a decision (horrifyingly), and honestly they don’t care about your report. They do, but they don’t and they have like 10 more reports to read.
The Takeaway:
Use Excel. Use Python. Nobody cares how, just make the pictures. It will take you like 5 minutes once you figure it out and you’ll look like a wizard leaving everyone else in the dust. Also! CRUCIALLY! You might learn something new and unexpected because data visualizations reveal cool and weird things.
Don't Hide Behind Basis Points
This one is a bit tricky, but sometimes reporting becomes a goal unto itself.
The Context:
If things are hard and you want to avoid solving your real problems then try to calculate the 10th decimal point of some metric you’re tracking and you’ll be really busy for a long time. This isn’t useful, but people do this all the time and feel good about it because they tried really really hard to measure the thing they’re supposed to be improving.
The Takeaway:
When thinking about the data you’re seeing always ask if you’re looking at the right level of depth. Do you need 3 decimal points or is “number go up” good enough? If so, round it off and make your team move on.
Once you have that figured out, always take a breath and think (and explain to yourself and others) "What do I need to believe for this result to be possible or true?"
This frames your interpretation of the mystical basis points with reality and forces you to acknowledge your assumptions. Then you can make actual decisions.
Phew! Metrics amirite?
I hope this added value to your day.
Please share this with someone who might find this interesting!
If you have any thoughts or questions about this essay - Let’s Chat
To hear more from me, add me on Twitter or Farcaster,
and, of course, please subscribe to Wysr