A simple guide to the product discovery process

Product discovery processProduct discovery is one of my favorite product management activities! I love every aspect of it and think it’s the best process for figuring out the right products and features to build.

Follow this simple guide to product discovery and make a tremendous positive impact on your company’s business.

Step 1 – Empathize

The first step in effective product discovery is empathizing with your users and customers. I distinguish between the two because customers may be paying for your product but not necessarily using it. And vice versa. It’s important to consider both constituents when conducting the discovery process.

This important step in the process is one that a lot of product teams overlook. To empathize with your customers, you need to put yourself in their shoes. Get to know them and understand what their challenges and joys are by talking to them. If you can visit them in person, that’s the best way. Thankfully, it’s easy to also interview users via video chat tools like Zoom and Google Hangouts.

Before you meet with your customers, though, create a script that includes the purpose for your visit along with a list of questions that you will ask each and every one. It’s important to do this so that you can establish a solid baseline for understanding what’s working for them today and what their problems are. Identifying patterns and trends in these problems is how you will succeed in the next step. Before we move on, though, it’s important to stress that you should be talking to your customers and users at least once a week. This shouldn’t be a one-time activity, but rather a continuous process.

Step 2 – Hypothesize

Now that you’ve talked to your users, you probably will see trends quickly emerge. In my experience, it doesn’t take too many customer visits for patterns to emerge. Before jumping into solution mode, though, it’s critical to formulate a number of hypotheses based on the data you’ve gathered.

What does a hypothesis look like? I’m going to use a hypothetical (no pun intended!) example. Let’s say you’re a product manager for the Domino’s Pizza app and you’ve talked to several customers. img_46cd519d9064-1You realize that the majority of them are not completing their order with the app and, instead, are placing an order by phone. You could then look at your app data to see where people are dropping off to formulate a hypothesis such as “If the app made it easier to see what specials and coupons are available, more people would place an order within the app.”

Before moving on to the next step, you should prioritize your list of hypotheses so that you can focus on the most important one. Once you’ve determined the hypothesis you think could have the biggest impact, it’s time for step 3.

Step 3 – Ideate

Do your engineers or marketing peers ever complain that they don’t get to come up with ideas for what products, features or services to build? This is the perfect time to get them involved. Actually, if possible, you should include them in the previous phases, too, but that is often not feasible for development teams that are in “delivery mode” (more on that in a future post!). Teresa Torres recently wrote a great article about why engineers should participate in the discovery process and it’s a good read.

Person writing in a note book

Schedule a half-day meeting, call it a workshop for bonus points, and invite a member from each of the functions that are involved in bringing your product to market (marketing, design, engineering, customer support are good starting points) and you may just find that the energy produced from everyone working together could light up a room.

The goal of rapid ideation is to quickly identify a number of solutions that would allow you to test your hypotheses.

I’ve done this exercise with teams before and we’ve timeboxed the entire process at 90 minutes. You can do this quickly. There’s no excuse not to be using rapid ideation on a weekly basis with your teams!

Here’s how to ideate rapidly:

  1. Break out and work individually at this point to generate the most ideas.
  2. Give everyone some post-its and colorful sharpies and have them start writing or sketching ideas, one per post-it note. I prefer to use extreme post-it notes because I’ve found that the basic ones don’t always stick to whiteboards and you could come back the next day to a pile of post-its on the floor.
  3. Limitscreen shot 2019-01-13 at 8.07.21 am the amount of time to this activity to a short period like 10 minutes. I find it’s useful to have a simple, visual timer available that everyone can see that they know how much time is left.
  4. Have everyone cluster their post-its on a whiteboard. There will inevitably be some duplication and overlap… this is good. Great minds often think alike!
  5. Give everyone colored dots (no more than 3) and have them vote on which idea resonates based on the likelihood for proving your hypothesis.
  6. Select the idea with the most dots. If there’s a tie, this is where you can use your product management experience and instincts to select the one you think is the best.
  7. Make a list of the ideas, especially the top ones so that you can come back to them later.

You’re now ready to prototype based on that top idea.

Step 4 – Prototype

Based on your hypothesis, what is the simplest, quickest prototype you can create to validate your idea? It could be as simple as a sketch on a piece of paper. It might be a black and white digital image you can show customers on their phone. It could be a low-fidelity prototype using a tool such as Balsamiq or Proto.io.

This is an actual mock-up created by Appinventiv for Domino’s Pizza*

It should be something you can literally throw out when you’re done testing it. In other words, it should not be something written in code that could be tweaked over time. This is to prevent teams from becoming too attached to their ideas. A big mistake teams often make is continuing to invest in something once it’s created even if it didn’t test well (or, even worse, wasn’t tested at all)!

You or someone on your team should be able to create a prototype in less than a day. If they’re taking longer than that, they are likely getting bogged down in details that can be figured out later.

Step 5 – Test

Validating your prototype is also something you should be able to do in a day. Find a few target users and have them attempt to use it. If it’s a paper-based mock-up, ask them to tell you what they would tap on, what information they would need to be able to perform their job, etc. Collect feedback on your prototype, from usability issues to whether or not the prototype helps them do their job easier. Find out what’s working and what’s not working and then go back to the drawing board to refine your hypothesis until you’ve reached the desired outcome. You can do this by tweaking the original prototype or you can take one of the other solutions from your ideation phase and mock that up for testing.

Ultimately, you should be working towards the smallest, shippable “thing” that achieves the desired outcome. From here, I encourage you to create a small backlog, not of features, but of hypotheses you’d like to explore and experiment with. Developing an experimental mindset that focuses on delivering upon outcomes and not features is the key to having the most impact on making your customers happy (as well as growing your company’s business).

Have you tried using the discovery process before? Have questions about how to do this? Get in touch. I’d love to hear from you!

* Before I came up with the hypothetical Domino’s Pizza example to show what a hypothesis is, I had no idea there was a company that had done work for Domino’s to improve their user experience for coupons and specials. Credit to Appinventiv for the mock-up shown above. 

Why you should use value-based pricing

value based pricing model

I recently read this fantastic article on value-based pricing by Ian Blair, which provides an in-depth guide on how to set your company’s pricing strategy. Why is this so important? Value-based pricing, when done right, can significantly increase your ARPU. I highly recommend you give this article a read, but here are the key takeaways:

First off, what is value-based pricing?
In its simplest terms, value-based pricing is pricing based on how much your customers will pay for your product.

Pricing should be based on your product’s differentiating feature(s)
Many companies provide a laundry list of features that they offer, typically with a pricing sheet that offers options that are good, better and best. The vast majority of features listed are usually commodities — that is, they are features that are commonly offered by all competitors. You may consider these table-stakes, but remember that this is not what determines the value of your product offering. How your product differs, how it’s better than the competition — that is what determines the value of your product or service.

Buyer personas are key to determining value-based pricing strategy
If you’re not already talking to your customers, you need to do this as your very first step in determining your pricing strategy. Ian includes some great sample questions to ask your customers and prospective customers to inform your pricing strategy. I personally think the most important question to ask is “What is the single most valuable feature that the buyer loves?”  The answer to this will likely be your key differentiator.

Determine the maximum and minimum price customers are willing to pay
Once you’ve talked to your customers (and remember, this is something you should be continuously doing!), come up with a price range that you think customers will pay. I’ve found in-person interviews aren’t the best forum for getting customers to open up about pricing. I recommend conducting a survey that allows you to ask your customers and prospects questions such as:

Screen Shot 2018-12-07 at 9.22.22 AM

That first question should help you validate the pricing floor — or the lowest price you can reasonably charge. Anything below this and you’re leaving money on the table, making it difficult to increase rates in the future by establishing a low anchor price and/or you’re damaging your company’s brand.

The next question should be asked to validate the ceiling price or the highest price you can reasonably charge.

Screen Shot 2018-12-07 at 9.22.01 AM

I recently conducted a survey for a client designed to set the company’s new pricing strategy and found this type of questioning to be a very effective approach. The data should quickly allow you to identify the ideal pricing sweet spot.

Are you using value-based pricing? If so, I’d love to know more about what you’ve found to be effective, too.

Lessons I’ve learned as a mentor

mentorI recently concluded a 6-month mentorship program created by Jeremy Horn, aka The Product Guy, that is aptly called The Product Mentor. The program is designed to pair mentors and mentees from around the world, across all industries, from start-up to enterprise.

My mentee, Merziyah Poonawala, is based in New York. Since I’m based in Santa Barbara, we conducted all of our mentoring activities virtually. Merziyah works for an agency and deals with lots of clients. As a coach and consultant, I, too, deal with lots of clients. This common ground gave our mentoring relationship a solid foundation upon which to identify concrete goals that she could work towards.

As we concluded the mentoring program last week, I looked back and identified the lessons I learned as a mentor.

1. Mentoring is a learning opportunity for both mentees and mentors.

Although my primary goal was to help my mentee become more successful, the mentee/mentor relationship caused me to challenge myself in areas in which I, too, could become more proficient. Treat the mentor/mentee relationship as a two-way street so that you can both grow and develop.

2. Make sure you’re on the same page.

A big focus of The Product Mentor program is on KPIs. While many open-ended mentoring relationships may not be as structured, I found it extremely helpful to use KPIs as the basis for identifying what success looks like for my mentee. Because we worked together to craft goals to help her achieve a successful outcome — how to become a more data-informed product manager — this gave us something concrete to discuss each week as we did our weekly check-ins.

3. Context matters.

Because of the nature of her job, Merziyah had a couple of real-world projects we were able to use as a way of putting her goals and objectives into practice. This proved to be extremely valuable. For example, at one point during the 6 months we worked together, she felt there was potentially a lack of trust coming from one of her clients. Because one of Merziyah’s KPIs was to use data to communicate more effectively, I coached her on using data to help her client understand the state of the project, thus building credibility and trust.

4. It pays to be mindful and engaged.

Being a mentor requires a commitment…a commitment to really engage with your mentee on a meaningful level. If you’re going to be a mentor, don’t just phone it in. Use that precious time to really focus on what your mentee is telling you and look for as many teachable moments as possible.

Want to learn more?

If you’re interested in becoming a product management mentor or mentee, click here to learn more about The Product Mentor program. Jeremy has included a lot of great content on this site, including videos of past mentor presentations. My presentation was about using metrics that matter (here’s a hint: outcome-based metrics are a must!), which you can find here.

TPM-Mentor-Award

I’d like to thank Jeremy for allowing me to be a part of this wonderful program. As I’ve said, this was a wonderful learning experience. Being recognized as an Outstanding Mentor for my participation in the program was the icing on the cake!

I’d also like to thank my mentee, Merziyah, for being such a great partner in this program and I wish her the best of luck as she continues to strive to become an even better product manager.

10 best holiday gifts for product managers

It’s a week since Black Friday which means we’re officially in the holiday season and many of you may be asked by your loved ones “What do you want for Christmas/Hanukkah/Kwanzaa?”

I’ve pulled together a few items that are some of my favorite things. Feel free to share this gift guide with those who might be struggling to pick the perfect present for their beloved product manager!

Sennheiser CX 5.00i Black In-Ear Headset

Screen Shot 2018-11-28 at 10.57.40 AMMost PMs work in open workspace environments, so a good pair of noise-canceling headphones is a must-have to help them block out the noise and focus on making their products even better. I love my Sennheisers! You will have a hard time finding earbuds as good and affordable as these. Plus, they have a good microphone which is a bonus if you’re doing a lot of calls from your desk.

 

 

Sonos One (virtual assistant included!)

Screen Shot 2018-11-28 at 11.01.33 AMFor the PM that works from home occasionally, this is a home-office must-have. The Sonos One lets you play your favorite music and control it with your voice or choice of apps. Whether you’re most productive when listening to your favorite Spotify playlist or want to catch up on product management podcasts (I have an article about my favorite podcasts coming soon), the Sonos One is ideal. And because Alexa is built in, you can use it as a virtual assistant to book meetings, check the weather and save you time.

 

A Desktop Air Purifier

Screen Shot 2018-11-28 at 2.26.52 PMI’m lucky that I work from home now, but I remember the days of being in an office surrounded by co-workers that had a cold, were just getting over a cold or said “really, it’s just allergies”. Stay healthy by keeping a desktop air purifier at your desk. This one is nice and small for a personal space.

 

 

A balance board by FluidStance

Screen Shot 2018-11-28 at 2.28.40 PMThis is a life-changer! If you have a standing desk, this will help you focus on maintaining good posture while — bonus! — strengthening your core and improving your balance. I found that it also prevented back strain associated with standing too long.  

 

 

A good journal for taking notes

Screen Shot 2018-11-28 at 2.05.28 PMAlthough I do most of my work on my laptop, there are times when I prefer to take notes the old-fashioned way. In fact, research has shown that writing notes (rather than typing) helps you retain information better. I prefer this journal that has dotted pages so that you can easily create boxes and diagrams.

 

Books!

There are so many great books I’ve enjoyed this past year, so I’ll just highlight a couple that I would recommend as gifts.

Screen Shot 2018-11-28 at 2.08.06 PMINSPIRED: How to Create Tech Products Customers Love

 

 

 

Screen Shot 2018-11-28 at 2.10.08 PMTools of Titans: The Tactics, Routines, and Habits of Billionaires, Icons, and World-Class Performers

 

 

Screen Shot 2018-11-28 at 2.11.46 PMSprint: How to Solve Big Problems and Test New Ideas in Just Five Days

 

 

 

Project Starter Kit for Arduino

Screen Shot 2018-11-28 at 2.24.25 PMArduino is the hot open-source prototyping platform for anyone who wants to create interactive physical environments. Perfect for you makers out there!

 

 

 

A crystal ball

Screen Shot 2018-11-28 at 2.18.26 PMWho wouldn’t love to gaze into a crystal ball to get answers to the universe’s most challenging questions, such as “What does 2019 have in store for me?!?”

 

 

 

 

 

How to organize a hackathon

You’ve convinced your company to conduct a Hackathon… now what?!? Over at the Product Collective Slack group, someone recently asked for advice on how to launch a hackathon at their company. Since this is something I initiated at Sonos years ago, I shared some suggestions. I thought others might find it useful, so I’m sharing my advice here, too.

When we started Hack Week at Sonos years ago, we kept the first few events really basic, which I highly recommend if you are just starting out.

Here are some of the things we did that were successful.

  1. Invite everyone to participate. At Sonos, we were lucky because our hack week teams included people from practically every department in the company, from software development and design to human resources and finance.
  2. Cancel stand-up meetings, grooming sessions, etc. so that as many participants can participate as possible.
  3. Book conference rooms for teams to use during the week AND for a few hours on the last day for demos. Since most meetings should be canceled during this time, rooms should be plentiful!
  4. Order lunch for everyone participating each day. This gives people a chance to mingle during the day and talk about their projects.
  5. Have a happy hour on the first day. Again, this social element really helps the participants get those creative juices flowing.
  6. Ask teams to sign up on a wiki page that includes the team name, team members, and a title for their project. This will come in handy later.
  7. Promote the demos to the entire company and invite everyone to attend. You especially want your leaders there.
  8. Invite a few ‘experts’ to work with teams on their ideas in an advisory role.
  9. Create a wiki page packed with tips on how to give good demos.
  10. Schedule demo day! The last day of the week should be dedicated to all of the teams giving demos of their projects. Pick a fixed time for each demo (10 minutes is a good guideline). Depending on the number of teams, schedule the rooms needed for demos accordingly.
  11. If possible, record the demos and post online for people to watch again (or if they missed it).
  12. Make it a competition! Have people who watch the demos vote for their favorite. Or have a panel of judges pick the winner. Be sure to have prizes for the winners, but don’t feel pressured to spend a lot of money. Just make it fun!
  13. Repeat every quarter!
  14. Consider doing a retrospective following your first few hackathons to continuously improve the process.

Good luck!

The Problem with NPS – and What You Can Do About It

NPS is a metric that has become a standard for measuring customer loyalty and satisfaction by many companies. Built on the power of one simple question — how likely is it that you would recommend [my product] to a friend or colleague? — it’s now used by companies of all sizes in virtually every industry all over the world.

The Problem with NPS

NPS is easy to measure. It produces a number you can track over time. It feels ‘legitimate’ because it’s based on customer feedback. So what’s not to like about it?

In reality, NPS has all the earmarks of a vanity metric.

That’s because the data is taken out of context and, more importantly, you can’t take action on it as a stand-alone metric, so it’s simply not useful. Even worse, it can be a major distraction from improving your company’s top-line metric (which, hopefully, isn’t NPS!).

At one of the companies I worked for a long time ago, we bought into NPS so much that we built our own survey tool and integrated it into our app to measure NPS. The problem we ran into, however, is that the metric by itself didn’t help us identify WHY someone might not recommend our product. We would usually monitor our NPS score after a big software release and, if the number increased, would say “Wow, that was a great release!”

Unfortunately, our software releases were pretty monolithic, meaning there were a lot of new features, UX improvements and bug fixes going into each release. Yes, I was grinding it out in a Feature Factory. We couldn’t point to any one thing we did that could be attributed to the increase in the NPS score. I could talk more about the dangers of monolithic software releases and how to deploy software in smaller, more frequent cycles, but that’s a whole ‘nother article. You should focus on the smallest, incremental improvements based on a framework like the lean startup methodology by Eric Ries.

lean-startup-methodology_diagram
Lean Startup by Eric Ries

 

Sidenote: Want to learn more about the dangers of Feature Factories? Check out John Cutler’s great article on the topic here

How to derive outcomes from NPS

If your company is committed to measuring NPS, here’s a tip that you can use to understand the ‘why’ behind your NPS score and potentially increase it.

Follow up the standard NPS question with one additional question, “What would it take for you to recommend my product to someone you know?”, and target the people who are not your promoters, which means they rated their likelihood to recommend your product an 8 or lower. You can then do an analysis of those open-ended responses to identify key trends in the data.

Screen Shot 2018-10-04 at 8.29.00 AM

Perhaps you work for a SAAS business that charges a monthly subscription fee and a large number of people say the price is too expensive and that’s why they won’t recommend to a friend. Using the lean analytics model, you could conduct pricing experimentation to see if there is a lower pricing model that results in driving up your referrals (and growing your company’s business).

Want to know more about measuring key outcomes? Click here!

Measure What Matters

Everybody knows metrics are important for product managers. The key is using the right data to inform and influence your product decisions.

There are 3 types of data: The good, the bad, and, yes, the ugly

The Good, the Bad and the Ugly

Let’s start off with the bad — Vanity Metrics

What are vanity metrics? They’re a way of measuring something without context, they are easily manipulated, and they do not lead to actions you can take to grow your business.  Examples of vanity metrics include app downloads, page views, time spent on a website, twitter followers, facebook likes, etc.

The number of times your app has been downloaded does not provide a direct correlation to how successful your business is. If a million people have downloaded it, but only 1/10th are using it, that’s not good. Same with page views. Just because your website has lots of traffic does not mean you have a successful business.

It is important for companies to properly measure the right data so that they can get a handle on the true health of their business.

If you focus primarily on vanity metrics, you can get a false sense of success by focusing on metrics that might make you feel good, but don’t actually tell you how well your business is doing. This is an especially lesson for start-ups, which seem tempted to use vanity metrics because they haven’t quite figured out which metrics actually matter.

Let’s take a look at some examples of vanity metrics…

Screen Shot 2018-09-22 at 8.17.39 PM

On this chart, it looks like something really great happened on August 14th. Wow, web traffic really spiked that day! The problem that many people make is attributing that solitary spike to an action they took directly preceding it. Multiple people at their company may have done something that resulted in the spike. There could even be external events that caused the spike. The key takeaway here is if this spike didn’t lead to the company’s bottom line by generating revenue, it’s useless.

Speaking of useless, this entire page is useless!

google-analyticsMany product managers use Google Analytics. But if you use their default dashboard, pictured here, there’s nothing you can take action on based on these metrics! The thing to remember is that you should customize your dashboard to focus on the metrics that measure how well your product is achieving key outcomes. Data points such as the number of users, page views, and average session duration don’t help you measure what’s working. And you can’t improve what you’re not effectively measuring.

The Ugly – Flawed Charts

As often as product managers talk about making data-informed decisions, it’s amazing to me how often data is presented in ways that are completely misleading or downright wrong.

Many of you may remember the Venn diagram incident from Hillary Clinton’s campaign.Screen Shot 2018-09-22 at 8.23.26 PM

The general point her campaign team was trying to make was true. The majority of American’s support universal background checks including a majority of gun owners. If you think about it, though, the yellow circle should be completely inside of the blue circle because the gun owners referenced here are all Americans. But that’s now how Venn diagrams work! This information should’ve been presented using a different type of chart.

In this Google example, Screen Shot 2018-09-22 at 8.23.33 PMit appears the email designer messed up the doughnut chart. As you can see, the green line goes around more than half of the circle, even though the number referenced is 41%.

The chart below is bad for several reasons. If you follow the use of colors, you’ll see Iceland, Finland, Portugal and Spain all nicely represented with different colors. And then it shows the UK, Denmark, Australia, Venezuela, and Kenya as being part of the same country. This would’ve worked much better as a simple bar chart!Screen Shot 2018-09-22 at 8.23.42 PM

The lesson here is to make sure you use the right type of visualization for displaying your data and that your chart is correct.

Where can you go for help with selecting and creating charts?

If you’re struggling with deciding which type of chart to use, I highly recommend From Data to Viz.

Screen Shot 2018-08-15 at 7.52.49 AM
From Data to Viz

Whether you’re new to the world of creating graphs or you’ve been doing this for years, this website is a treasure trove of information about how to present data.  

Based on the type of data you have, such as chronological or numerical, it will guide you through the relevant types of visualizations that are best suited for your particular case. 

Depending on the type of chart you’ve selected, it will also inform you about the pitfalls to avoid.

Another great source of information is this book by Edward Tufte. The Visual Display of Quantitative Information is the bible in the industry. I can’t recommend this book enough! And, if you ever get the chance to attend one of his seminars, it’s well worth it!Screen Shot 2018-09-22 at 8.30.20 PM

The Good = Measuring for Outcomes

We’ve talked a lot about vanity metrics. Vanity metrics are typically raw numbers taken without context, such as page views and app downloads. They don’t correlate directly to customer value. And they don’t provide guidance for what future product changes you should make.

What do we mean by outcome-oriented metrics? Those are metrics that typically do 3 things:

  1. They link actions to results.
  2. They focus on delivering customer value.
  3. They provide insight into the health of your product or business.

Remember the differences between vanity and outcome-focused metrics. Vanity metrics might make us feel good, but they don’t help us improve or optimize our business.

Hopefully, by now, you know which metrics to avoid.  And you know a little more about the types of metrics that you should focus on.

Where should you start when you’re ready to embrace an outcome-focused approach to measuring success? It’s really quite simple.

sales-kpis

Start with your company’s strategy.

When your company has a strong, clear vision, that should serve as your north star and guide you to prioritize the most important metric. Based on that vision, what is the one metric that matters most to the success of your company and that you can rally your team around?

Let’s look at a great example of leading from strategy with Zappos. Their CEO literally wrote the book on modern customer service. Zappos disrupted online retail sales by focusing on its support department as an opportunity to market and generate revenue rather than as a cost center. Their entire strategy revolves around creating loyalty among its customers using effective KPIs that lead to what they call ‘wow’ moments. And they did this because they found that repeat customers spend more than first-time customers and drive referrals.

Here’s a more peronal, real-world example from Sonos, where I worked for nearly twelve years as a product management leader. The Sonos mission is to fill every home with music.sonos We would use that as our north star when defining key outcomes. For example, ‘fastest time to music’ was a key outcome by which we would measure the success of a particular feature. The rationale for that was the faster the music starts playing, the better Sonos is doing its job of filling the home with music.

Ask yourself, what is the key outcome you can focus on and set actionable KPIs against based on your companies strategy?

With your company strategy in hand, I recommend selecting a useful framework for measuring key outcomes and the one I like is Pirate Metrics.

Pirate Metrics

Pirate Metrics were coined by Dave McClure back in 2007. Here’s a link to his presentation where he first describes this framework. He breaks it down into 5 components.

Acquisition – how well are you getting customers to your site or app?

Activation – are your customers having a great ‘first run’ experience?

Retention – how often are your customers coming back?

Referral – are they telling others about your product?

Revenue – are they paying for your service? Are you able to monetize your customers?

Let’s dig deeper into this and come up with some examples to help you better understand pirate metrics.

To better understand how to apply pirate metrics, pretend we’re product managers for lynda.com, an online learning platform that lets you watch videos on how to do everything from learning to knit to learning to code and everything in between.  What are some outcomes that we might care about that could be measured using the pirate metric framework? 

Screen Shot 2018-09-22 at 7.00.16 AM

  1. For starters, let’s look at acquisition. We don’t necessarily care how many people come to the lynda.com homepage. That’s a vanity metric. What we probably do care about, though, is how many people are signing up for a free trial of the service within a specific time period, say each month.
  2. From there, how can we measure activation and whether or not our customers are having a great first run experience? A good metric, building on the previous one, would be what percentage of the people signing up for a free trial are watching a video from start to finish? If they’re only watching a few seconds of a video and then leaving, you could hypothesize that they’re not having a great first run experience.
  3. To measure retention, we could measure what percentage of those people are coming back to watch another video within the month. This is probably one of the most important metrics product managers care about. Knowing your MAU, or monthly average users might even be your company’s top line metric, which is the case for a company like Facebook.
  4. For measuring referrals, we could measure what % of people are sharing links of videos to their network. Some companies like to measure NPS, also known as the Net Promoter Score. It’s based on asking customers if they would recommend your product to friends or colleagues.  I’ll talk more about this in a few minutes.
  5. The final, and probably most important metric, especially if you work for a SAAS product is measuring how much revenue your customers generate. ARPU, or average revenue per user, is often the top-line metric for SAAS business. How might we measure this? One possibility is to measure the % of people who go from the trial to paying for a monthly subscription, which most companies refer to as their key conversion rate.

Putting outcome-based metrics to practice

Now that you have a framework to help you identify all of the possible types of outcomes you can measure, let’s talk about how to harness this information and put it into practice.

We can boil it down to four key steps:

  1. What is the key outcome based on your company’s vision and strategy? What is the most important thing you can improve upon? Perhaps it’s an increase in your conversation rate.
  2. Form a hypothesis. The important thing to remember is to start small and look for the most meaningful lever that you can pull and focus on that. Don’t change five different things on your landing page and then start measuring your conversation rate. You won’t be able to correlate which of those changes affected your conversion rate. Change only one thing. Your hypothesis could be something like: “If we reduce the price of our product by 10%, we’ll see an increase in our conversation rate of at least 11%.”
  3. Build your experiment. This is where you need to dissect your hypothesis into key components so that you can collect the correct data to validate if your hypothesis is correct or not. Before you build anything, make sure you know what your current state, or baseline metric, is. In this case, make sure you can state what your current conversion rate is. Once you have your experiment ready, set up your analytics to measure the KPI against the current baseline and the goal that you’ve set.
  4. Measure and analyze. Once you’ve got the new data coming in from your experiment, you should be able to quickly analyze if it was a success or not.
    • If it wasn’t, that means your hypothesis was incorrect. Remember that you should not view these moments as failures or a waste of your precious developer resources. This is a learning moment. Failing fast and learning early is key to allowing you to eventually zero in on what works.
    • If the experiment was mildly successful, I encourage you to tweak your hypothesis based on the new data you have.
    • If it was wildly successful, celebrate! And then look for the next lever you can pull to help your business be even more successful.

What I just described with those 4 steps is a framework for continuous learning. Taken from the Lean Startup Methodology, can be used by anyone.

lean-startup-methodology_diagram
Lean Startup Methodology by Eric Ries

By creating this virtuous loop of building experiments, measuring your KPIs and learning each step of the way, you can quickly and successfully create value for your customers (and your business).

Key Takeaways

  1. Measure outcomes, not vanity metrics. Tie actions to results.
  2. Make sure your data is correct. Examine a sample of your data before moving on.
  3. Choose the right type of visualization for the data. From Data to Viz is a great resource for this.
  4. Define KPIs based on your company’s strategy and top-line metric.
  5. Experiment with small changes and foster a culture of continuous learning.

 

%d bloggers like this: