(This is part two of a series.  Previously I criticized the use of the “One Metric That Matters” approach to business management. You can read the previous post here.)

Above is a chart from a funding pitch deck from a real startup. Roughly 99% of this chart was altered further to protect everyone involved. This chart was used in the funding deck because it looks like the type of chart you would think every investor would want to see. It goes up and to the right, shows an exponential growth curve. 

In this post I’ll argue that growth is not something you can or should manage with just one metric. As with most aspects of a business, trying to apply the “One Metric That Matters” (OMTM) to a company you wish to be successful is a myopic recipe for failure. Instead you must use a collection of metrics to provide a realistic assessment.

Growth seems pretty simple, right? “The numbers need to go up and to the right” in a scale that suggests you and your company are “riding a rocket.” Perhaps you think you “have the bull by the horns” or the “tiger by the tail.” It sounds exciting, a little terrifying, but positive, right?

Vanity and Theater

We as a culture are obsessed with growth, and the media has fanned this obsession with what Lean Startup author Eric Ries refers to in his writing and speaking as “vanity metrics” employed in the service of “success theater.” This is the publicizing of some number that is so large as to appear sensational. Its purpose is to give the press something to write about, entice prospective investors, and make your competitors jealous. The truth is that the number means nothing, and certainly provides no insights that your competition can use against you. 

One of my favorite recordings from the Commonwealth Club of California is this conversation between Eric Ries and 500 Startups founder Dave McClure. The example they describe is that if you are talking about having a million clicks to your website a week, that could be from one million individuals that each visit once and then abandon it, never again to return. Those million clicks could also be a single user who is clicking a million times over and over (probably a bot, or your mom).

It’s true that there are some stories of incredible startups that became giant, like Instagram and Facebook, and in retrospect we can look at the history of the growth of those companies and see giant numbers. We make the mistake of believing that giant numbers cause a huge success, when more accurately it is the huge success that produces giant numbers as a bi-product.

For every company that was as successful as Instagram or Facebook (that’s a very small club), there are hundreds that started out similarly but died. We can look through the dead pool and find examples of startups that everybody thought would have been the next big thing. Here’s an article by Business Insider reporting on what happened to some particularly overhyped companies. 

Opportunity Costs

Let’s return to the chart from earlier. The main problem with this chart is that it is based on a single number, cumulative signups. It is impossible for a cumulative signup number to decrease over time, unless your company is proactively deleting accounts and factoring that into the equation. Success theater, however, is an arms race, and no company is going to intentionally take an action to make this chart look less sensational by doing something like deleting users. This is the same problem with metrics like total installs or total number of app store downloads, or any number that is aggregated and totalled like this.

What’s worse, spending your precious time, talent, and dollars on activities that serve success theater and vanity metrics becomes very expensive when you consider the opportunity costs involved. A startup that chooses options that make the numbers more sensational over what makes for a better customer experience is a startup that has gotten seduced into providing a sub-par software stack, database architecture, user features, and company workflow. 

A Better Way

Stop being so vain! Instead of spending any time on these sort of vanity numbers to track and flaunt your “growth,” there’s a better way. Below are two sets of analysis that in the long run will provide you with a clearer, better picture of the growth and health of your company:

 

Cohort Analysis

 Look at the behavior of users over time. An easier way to do this than looking at them individually is to look at them in cohorts. A cohort in this context is grouping users by time based on a milestone. It’s like a graduating class. If you are making weekly changes to your product, try grouping individual users into weekly cohorts. These might be individuals who completed the gauntlet of your signup process using the set of features that were available at that time. For a slower moving company, you may look at a monthly view. The important thing is to make sure that the size of each cohort is statistically significant. A weekly cohort of a total of just 3 users isn’t going to give you information you can trust.

If a new feature is added that improves your product that improvement will be reflected in the numbers for that cohort. If you break the experience or just make it worse somehow, then you’ll see that change because your future cohorts will not look as good for as long as the ones in the previous week.

There’s more about cohort analysis I can and should discuss but I’ll save it for another post. One of my favorite references for generating cohort analysis in Python is this post by Greg Reda. Please comment or contact me if you have specific questions about cohort analysis and I’ll be glad to help.

DAU:MAU

This is one of my favorite measurements. It is incredibly easy to do.  Also, there is a wealth of publicly available data that can be used to compare your metrics with other companies. It seems to be fairly robust across industries too, although you should always check this before blindly trusting external data. DAU is an acronym for “daily active users.” MAU stands for “monthly active users.”

The way this works is that each 24-hour period you count the total number of users that engaged your product. Assuming you are logging a user id number as well as a timestamp, this is fairly easy to calculate. If a user id appears inside of a 24-hour window, you add it to that day’s tally. Every user only gets counted once in each day. That means if a user clicks 1,000 times in a single day, they still are only counted one time.

Then you do this same analysis again, but instead of a 1-day window, you look at a 30-day window. That gives you your MAU number. To calculate this, some average the last thirty days of DAU and stick with calendar months. I’ve also seen this calculated as taking a single day and then calculating backwards for the last 30 days. I prefer the latter approach.

Finally you simply divide your DAU by your MAU and you get a ratio. That result is your DAU:MAU ratio, and it is a number I believe is very useful. I’m not alone in this assessment; you can read more about DAU:MAU here, here and here.

Another Disclaimer

It’s important to note that you should use both approaches, and more. Paying attention to DAU:MAU but not doing cohort analysis will make it difficult for you to understanding the longevity and retention of your users. Conversely, ignoring DAU:MAU to focus only on cohort analysis means you’ll miss spikes or drops in behavior.

Besides the above examples there are other important indicators of the health of the business that your company should monitor. For instance, how comfortable are you with the balance sheet? Have you considered alternative runway scenarios in your forecast?

Despite this, if you are reviewing DAU:MAU plus cohort analysis charts regularly, you are doing better than at least 80% of companies I’ve worked with. Do this now so that when we work together we can jump past the basics.

 

That’s it, for now. I hope this encourages you and your company to do less playing around with vanity metrics and success theater. I hope it will lead to you seeing a more accurate picture of the growth and health of your company. If you like this post and would like to see more like it, please let me know!