Metrics that count, versus Vanity Metrics

Background – The Lean Start-Up

Lean start-up is a method for developing businesses and products first proposed in 2011 by Eric Ries. Based on his previous experience working in several American start-ups, Ries claims that start-ups can shorten their product development cycles by adopting a combination of business-hypothesis-driven experimentation, iterative product releases, and what he calls validated learning. Ries' overall claim is that if start-ups invest their time into iteratively building products or services to meet the needs of early adopter customers, they can reduce the market risks and sidestep the need for large amounts of initial project funding and expensive product launches and failures.

The Lean Start-up approach fosters companies that are both more capital efficient and that leverage human creativity more effectively. Inspired by lessons from lean manufacturing (for example Toyota), it relies on validated learning, rapid scientific experimentation, as well as a number of counter-intuitive practices that shorten product development cycles, measure actual progress without resorting to the so-called vanity metrics, and learn what customers really want. It enables a company to shift directions with agility, altering plans inch by inch, minute by minute. Rather than wasting time creating elaborate business plans, The Lean Start-up offers entrepreneurs in companies of all sizes a way to test their vision continuously, to adapt and adjust before it’s too late. Ries provides a scientific approach to creating and managing successful start-ups in an age when companies need to innovate more than ever.

Measure

A start-up´s objective should be to:

  1. Rigorously measure where it is right now, confronting the hard truths that assessment reveals and then…
  2. Devise experiments to learn how to move the real numbers closer to the ideal reflected in the business plan.

A disciplined, systematic approach to figuring out if progress is being made and discovering if the organisation is actually achieving validated learning. This is called innovation accounting; an alternative to traditional accounting designed specifically for start-ups, and begins by turning the leap-of-faith assumptions into a quantifiable financial model. 

Innovation Accounting works in three steps:

  1. Use a Minimal Viable Product (MVP) to establish real data on where the company is right now
  2. Start-ups must attempt to tune the engine from the baseline toward the ideal. 
  3. After the start-up has made all the micro changes and product optimizations it can move its baseline toward the ideal, the company reaches a decision point: to pivot or preserve.

The innovation accounting framework makes it clear when the company is stuck and needs to change direction. The importance of innovation accounting is based on only 5% of entrepreneurship being the big idea, the business model, the whiteboard strategizing, and the splitting up of the spoils. The other 95% is the gritty work that is measured by innovation accounting: product prioritizing decisions, deciding which customers to target or listen to, and having the courage to subject a grand vision to constant testing and feedback.

Learning milestone

The MVP provides the first example of a learning milestone. An MVP allows a start-up to fill in real baseline data in it growth model namely conversion rates. Sign up and trial rates, customer life value etc. and this is valuable as the foundation for learning about customers and their reactions to product even if that foundation begins with extremely bad news.

Tuning the engine: Once the baseline has been established, the start-up can work toward the second learning milestone: tuning the engine. Every product development, marketing, or any other initiative that a start-up undertakes should be targeted at improving one of the drivers of its growth models.

Actionable metrics

Start-ups love to point to big growth numbers, and the press loves to publish them. Examples can include the likes of one million downloads, 10 million registered users, 200 million tweets per day. These growth metrics can often be signs of traction (which is why they are reported), but just as often they are not. It is important to distinguish between real metrics and what Eric Ries calls vanity metrics.

Actionable metrics can lead to informed business decisions and subsequent action. These are in contrast to vanity metrics that are measurements that give the rosiest picture possible but do not accurately reflect the key drivers of a business.

Vanity metrics for one company may be actionable metrics for another. For example, a company specializing in creating web based dashboards for financial markets might view the number of web page views per person as a vanity metric as their revenue is not based on number of page views. However, an online magazine with advertising would view web page views as a key metric as page views are directly correlated to revenue.

A typical example of a vanity metric is 'the number of new users gained per day'. While a high number of users gained per day seems beneficial to any company, if the cost of acquiring each user through expensive advertising campaigns is significantly higher than the revenue gained per user, then gaining more users could quickly lead to bankruptcy.

Vanity metrics are things like registered users, downloads, and raw page views. They are easily manipulated, and do not necessarily correlate to the numbers that really matter: active users, engagement, the cost of getting new customers, and ultimately revenues and profits. The latter are more actionable metrics. The real data is retention and repeat usage. Start-ups that focus on the real metrics can make their products better, attract more customers, and retain them for longer.

It is important for start-ups to properly instrument the data they track so that they can get a handle on the true health of their business. If they track only the vanity metrics, they can get a false sense of success. Just because a start-up can produce a chart that is up and to the right does not mean it has a great business. A mobile application could have millions of downloads but only a few hundred thousand active users, or a freemium website might see exploding traffic growth but barely any conversions to paying users.

Many start-ups, of course, track one set of numbers internally and selectively share another set of vanity numbers externally with the press or potential funders. The worst is when start-ups try to pitch raw growth numbers (we are up 400%), but without any context (400% from what, 1,000 users or 100,000?). One should therefore always ask for more meaningful numbers, but those are not always available or forthcoming.

The vanity metrics aren’t completely useless, just don’t be fooled by them. There are ways to back into real numbers from the vanity metrics. Venture Capitalist Fred Wilson names it his 30/10/10 rule: 30 percent of downloads or registered users are active once a month, 10 percent are active once a day, and 10 percent of the daily users will be the maximum number of concurrent users. These are the patterns he is seeing in his portfolio companies and the start-ups pitching him.

Start-ups would be better off, however, reporting real metrics from the start. Vanity metrics can catch up to them, especially if those numbers do not correspond to the real numbers. Facebook is a great example of a company that focuses on the right numbers. Even in its college-only days, it would always talk about daily active users (the users who come back every day) and how fast it took them to take over a particular campus. If more start-ups would measure and share the right metrics from the start, the rest would begin to focus on them too.