In the first part of this three-part series on metrics, we discussed vanity metrics. Vanity metrics are typically raw numbers taken without context, such as page views and app downloads. They don’t correlate directly to customer value. And they don’t provide guidance for what future product changes you should make.

Outcome-oriented metrics, on the other hand, do provide guidance on how to evolve and improve your products over time. Outcome-focused metrics typically do 3 things:

1. They link actions to results.
2. They focus on delivering customer value.
3. They provide insight into the health of your product or business.

Remember the differences between vanity and outcome-based metrics. Vanity metrics might make us feel good, but they don’t help us improve or optimize our business. Outcome-based metrics provide insight into the health of your product.

Tweet

Where should you start when you’re ready to embrace an outcome-focused approach to measuring success?

Start with your company’s strategy.

When your company has a strong, clear vision, it should serve as your north star and guide you to defining the most important metric. Based on that vision, what is the one metric that matters most to the success of your company and that you can rally your team around?

Let’s look at a great example of leading from strategy with Zappos. Their CEO literally wrote the book on modern customer service. Zappos disrupted online retail sales by focusing on its support department as an opportunity to market and generate revenue rather than as a cost center. Their entire strategy revolves around creating loyalty among its customers using effective KPIs that lead to what they call ‘wow’ moments. And they did this because they found that repeat customers spend more than first-time customers and drive referrals.

Here’s another example from Sonos, where I worked for nearly twelve years as a product management leader. The Sonos mission is to fill every home with music. We used that as our north star when defining key outcomes. For example, ‘fastest time to music’ was a key outcome by which we would measure the success of a particular feature. The rationale was the faster the music starts playing, the better Sonos is doing its job of filling the home with music.

Ask yourself, what is the key outcome you can focus on and set actionable KPIs against based on your companies strategy?

With your company strategy in hand, I recommend selecting a useful framework for measuring key outcomes and the one I like is Pirate Metrics.

Pirate Metrics Framework

Pirate Metrics were coined by Dave McClure back in 2007. Here’s a link to his presentation where he first describes this framework. He breaks it down into 5 components.

Acquisition – how well are you getting customers to your site or app?

Activation – are your customers having a great ‘first run’ experience?

Retention – how often are your customers coming back?

Referral – are they telling others about your product?

Revenue – are they paying for your service? Are you able to monetize your customers?

Let’s dig deeper into this and come up with some examples to help you better understand pirate metrics.

To better understand how to apply pirate metrics, pretend we’re product managers for lynda.com, an online learning platform that lets you watch videos on how to do everything from learning to knit to learning to code and everything in between.  What are some outcomes that we might care about that could be measured using the pirate metric framework? 

1. For starters, let’s look at acquisition. We don’t necessarily care how many people come to the lynda.com homepage. That’s a vanity metric. What we probably do care about, though, is how many people are signing up for a free trial of the service within a specific time period, say each month.
2. From there, how can we measure activation and whether or not our customers are having a great first run experience? A good metric, building on the previous one, would be what percentage of the people signing up for a free trial are watching a video from start to finish? If they’re only watching a few seconds of a video and then leaving, you could hypothesize that they’re not having a great first run experience.
3. To measure retention, we could measure what percentage of those people are coming back to watch another video within the month. This is probably one of the most important metrics product managers care about. Knowing your MAU, or monthly average users might even be your company’s top line metric, which is the case for a company like Facebook.
4. For measuring referrals, we could measure what % of people are sharing links of videos to their network. Some companies like to measure NPS, also known as the Net Promoter Score. It’s based on asking customers if they would recommend your product to friends or colleagues.  I’ll talk more about this in a few minutes.
5. The final, and probably most important metric, especially if you work for a SAAS product is measuring how much revenue your customers generate. ARPU, or average revenue per user, is often the top-line metric for SAAS business. How might we measure this? One possibility is to measure the % of people who go from the trial to paying for a monthly subscription, which most companies refer to as their key conversion rate.

Putting outcome-based metrics into practice

Now that you have a framework to help you identify all of the possible types of outcomes you can measure, let’s talk about how to harness this information and put it into practice.

We can boil it down to four key steps:

1. What is the key outcome based on your company’s vision and strategy? What is the most important thing you can improve upon? Perhaps it’s an increase in your conversation rate.
2. Form a hypothesis. The important thing to remember is to start small and look for the most meaningful lever that you can pull and focus on that. Don’t change five different things on your landing page and then start measuring your conversation rate. You won’t be able to correlate which of those changes affected your conversion rate. Change only one thing. Your hypothesis could be something like: “If we reduce the price of our product by 10%, we’ll see an increase in our conversation rate of at least 11%.”
3. Build your experiment. This is where you need to dissect your hypothesis into key components so that you can collect the correct data to validate if your hypothesis is correct or not. Before you build anything, make sure you know what your current state, or baseline metric, is. In this case, make sure you can state what your current conversion rate is. Once you have your experiment ready, set up your analytics to measure the KPI against the current baseline and the goal that you’ve set.
4. Measure and analyze. Once you’ve got the new data coming in from your experiment, you should be able to quickly analyze if it was a success or not.

• If it wasn’t, that means your hypothesis was incorrect. Remember that you should not view these moments as failures or a waste of your precious developer resources. This is a learning moment. Failing fast and learning early is key to allowing you to eventually zero in on what works.
• If the experiment was mildly successful, I encourage you to tweak your hypothesis based on the new data you have.
• If the experiment was mildly successful, I encourage you to tweak your hypothesis based on the new data you have.

If it was wildly successful, celebrate! And then look for the next lever you can pull to help your business be even more successful.

What I just described with those 4 steps is a framework for continuous learning based on the Lean Startup Methodology.

By creating this virtuous loop of building experiments, measuring your KPIs and learning each step of the way, you can quickly and successfully create value for your customers (and your business).

Key takeaways from this 3-part series

We’ve covered a lot of ground in this series of articles on the various types of product data that exist, so here’s a quick recap of the key take-aways:

1. Measure for outcomes, not vanity metrics. Tie actions to results.
2. Define KPIs based on your company’s strategy and top-line metric.
3. Experiment with small changes and foster a culture of continuous learning.
4. Make sure your data is correct. Examine a sample of your data before moving on.
5. Choose the right type of visualization for the data. From Data to Viz is a great resource for this.

If you found this article (or series) useful, please share it with your network. You might also enjoy this article about how to achive product/market fit.

Sign up to receive an update when I post my next blog article!