Homework answers / question archive / Praise for Lean Analytics “Your competition will use this book to outgrow you

Praise for Lean Analytics “Your competition will use this book to outgrow you

Business

Share With

Praise for Lean Analytics “Your competition will use this book to outgrow you.” Mike Volpe—CMO, Hubspot “Everyone has data, the key is figuring out what pieces will improve your learning and decision making. Everyone knows they need metrics, but finding ones that are specific, measurable, actionable, relevant, and timely is a huge challenge. In Lean Analytics, Ben and Alistair have done a masterful job showing us how to use data and metrics to peer through the haze of uncertainty that surrounds creating new businesses and products. This book is a huge gift to our industry.” Zach Nies—Chief Technologist, Rally Software “Lean Analytics is the missing piece of Lean Startup, with practical and detailed research, advice and guidance that can help you succeed faster in a startup or large organization.” Dan Martell—CEO and Founder, Clarity “Entrepreneurs need their own reality distortion field to tilt at improbable windmills. But that delusion can be their undoing if they start lying to themselves. This book is the antidote. Alistair and Ben have written a much-needed dose of reality, and entrepreneurs who ignore this datadriven approach do so at their peril.” Brad Feld—Managing Director, Foundry Group; Co-founder, TechStars; and Creator, the Startup Revolution series of books “Lean Analytics will take you from Minimum Viable Product to Maximally Valuable Product. It’s as useful for product managers at today’s multi-billion dollar companies as it is for entrepreneurs who aspire to build those of tomorrow.” John Stormer—Senior Director of New Products, Salesforce “The bad news is, there will always be people out there smarter than you. The good news is, Alistair and Ben are those guys. Using Lean Analytics will give you the edge you need.” Julien Smith—New York Times bestselling author of Trust Agents and The Flinch “At Twitter, analytics has been key to understanding our users and growing our business. Smart startups need to embrace a data-driven approach if they’re going to compete on a level playing field, and this book shows you how.” Kevin Weil—Director of Product, Revenue, Twitter “A must-read on how to integrate analytics deep into an emerging product, and take the guesswork out of business success.” Peter Yared—CTO/CIO, CBS Interactive “Lean Analytics is a detailed explanation of the data-driven approach to running a business. Thoughtfully composed by two experienced entrepreneurs, this is a book I will make part of my training materials at Sincerely, Inc., and all future companies.” Matt Brezina—Founder, Sincerely, Inc., and Xobni “Pearson’s Law states, ‘That which is measured improves.’ Croll and Yoskovitz extend our understanding of Lean management by bringing rigorous measurement techniques to a new frontier: the earliest stages of new product development and launch. If entrepreneurs apply their frameworks, they should see reduced waste and big improvements in startup success rates. Thomas Eisenmann—Howard H. Stevenson Professor of Business Administration, Harvard Business School “This isn’t just a book about web analytics or business analytics—it’s a book about what organizations should and shouldn’t measure, and how to transform that data into actionable practices that will help them succeed. Alistair and Benjamin have compiled a robust set of case studies that illustrate the power of getting analytics right, and, if taken to heart, their tips and takeaways will make entrepreneurs, marketers, product and engineering folks better at what they do.” Rand Fishkin—CEO and Co-founder, Moz “I bet you’d never imagined that success depends on your ability to fail. Fail faster, fail forward. And the secret to that success is your ability to learn and iterate quickly using data. Qualitative and quantitative. Let Alistair and Ben show you how to get to startup nirvana smarter!” Avinash Kaushik—Author, Web Analytics 2.0 “Lean Analytics shows you how to move insanely fast by getting your metrics to tell you when you’re failing and how to do something about it. Tons of honest, meaningful advice—a must-read for Founders who want to win.” Sean Kane—Co-founder, F6S and Springboard Accelerator “There are only two skills that are guaranteed to reduce the chances of startup failure. One is clairvoyance; the other is in this book. Every entrepreneur should read it.” Dharmesh Shah—Founder and CTO, HubSpot “First you need to build something people love. Then you need to attract and engage people to find and use it. Having a deep understanding of your data and metrics is fundamental in achieving this at scale. Lean Analytics is a detailed, hands-on approach to learning what it means to track the right metrics and use them to build the right products.” Josh Elman—VC, Greylock Partners “Lean Analytics is the natural evolution of the Lean Startup movement, which began as a humble blog and has blossomed into a global movement. This book delivers concrete, hard-won insights spanning all business models and company stages. It’s a must-read for any business leader who’s looking to succeed in an increasingly data-driven world.” Mark MacLeod—Chief Corporate Development Officer, FreshBooks “A vital part of the founder’s toolkit. If you’re starting a company, you need to read this.” Mark Peter Davis—Venture Capitalist and Incubator “Lean Analytics is packed with practical, actionable advice and engaging case studies. You need to read this book to understand how to use data to build a better business.” Paul Joyce—Co-founder and CEO, Geckoboard “Get this book now. Even if you’re only thinking about starting something, Lean Analytics will help. It’s a dose of tough love that will greatly increase your chances of survival and success. Start off on the right foot and read this book; you won’t regret it.” Dan Debow—Co-CEO and Founder, Rypple; SVP, Work.com “Stop thinking and just buy this book. It’s the secret sauce. If you’re an entrepreneur, it’s required reading.” Greg Isenberg—CEO, fiveby.tv; Venture Partner, Good People Ventures “This is a treasure for the Lean Startup movement—a dense collection of actionable advice, backed by real case studies. Lean concepts are easy to understand but often difficult to put into practice, but Lean Analytics makes the path clear and gives you the tools to measure your progress.” Jason Cohen—CEO, WP Engine “With this book, Alistair and Ben bring a framework and lessons together for the thousands of new startups looking to do things fast and right. Time is everything as markets get continuously more efficient, and even over-capitalized quickly. Lean Analytics is great learning material for this generation of web and mobile startups.” Howard Lindzon—Co-founder and CEO, Stocktwits; Managing Partner, Social Leverage; Creator, Wallstrip “Alistair and Ben are trusted leaders in their field already. With this book, they let you see how they got there.” Chris Brogan—CEO and President, Human Business Works “With Lean Analytics, Ben and Alistair have, for the first time that I’ve seen, put real case studies and numbers together in an easy-to-read form, with actual successful startups as examples. These insights are hugely powerful for both early-stage founders and those at a later stage. It’s one of the few books that I know I’ll be going back to time and time again.” Joel Gascoigne—Founder and CEO, Buffer “Daniel Patrick Moynihan famously said, ‘Everyone is entitled to his own opinion, but not to his own facts.’ This is never more true than in business. One of the best things about working with Alistair Croll is how he cuts through opinion with facts, turning marketing into learning, and product development into a conversation with customers.” Tim O’Reilly—Founder and CEO, O’Reilly Media, Inc. “Not more numbers, but actionable metrics. In Lean Analytics, Alistair and Ben teach you how to cut through the fog of data and focus on the right key metrics that make the difference between succeeding and failing.” Ash Maurya—Founder and CEO, Spark59 and WiredReach; author, Running Lean “We live in a day and age where data and analytics can (finally!) be used by anyone and everyone. If you’re not leveraging the power of data and analytics to figure out what works and what doesn’t, then you’re working in the dark. Listen to Alistair and Ben: they’re not only the light switch to get you out of the dark, but they know how the entire power plant runs. I can’t think of two people I would turn to quicker if I had a startup and wanted to leverage the power of data to make my business a success.” Mitch Joel—President, Twist Image; author, Ctrl Alt Delete “Many entrepreneurs are overwhelmed by data they don’t know what to do with and by metrics that aren’t helpful in running their business. Lean Analytics tells important stories from many businesses— with real data—to provide a framework to define the right metrics and use them to execute better. Highly recommended!” Mike Greenfield—Founder, Circle of Moms and Team Rankings “Lean Analytics helps you cut through the clutter and show you how to measure what really matters.” Rajesh Setty—Serial Entrepreneur and Business Alchemist, rajeshsetty.com “I’ve heard way too many early entrepreneurs (myself included!) bristle at letting data drive product design. ‘It’s my product—how could users know better than me?’ This book, with its wealth of relatable stories and examples, lays out in clear terms exactly how and why analytics can help. It’s a shortcut to a lesson that can otherwise take painfully long to learn.” Dan Melinger—Co-founder and CEO, Socialight “Ben and Alistair are startup experts in their own right, but they really went out of their way to solicit advice and input from as many other real-world practitioners as possible when writing this book. Their effort really pays off—Lean Analytics is chock-full of high-quality techniques for building your startup, put in terms that even a first-time entrepreneur can understand.” Bill D’Alessandro—Partner, Skyway Ventures “Are you in search of what to measure, how to measure it, and how to act on that data in order to grow your startup? Lean Analytics gives you exactly that.” Rob Walling—Author, Start Small, Stay Small: A Developer’s Guide to Launching a Startup “You need this book if you’re an entrepreneur looking to get an edge with your data.” Massimo Farina—Co-founder, Static Pixels “Every entrepreneur’s goal is to follow the most efficient path to success, but you rarely know that path going in. Lean Analytics demonstrates the process of leveraging very specific metrics to find your business’s unique path, in a way that new entrepreneurs and veterans alike can understand.” Ryan Vaughn—Founder, Varsity News Network Lean Analytics Use Data to Build a Better Startup Faster Alistair Croll Benjamin Yoskovitz Beijing · Cambridge · Farnham · Köln · Sebastopol · Tokyo Lean Analytics by Alistair Croll and Benjamin Yoskovitz Copyright © 2013 Alistair Croll, Benjamin Yoskovitz. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (safari.oreilly.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com. Editor: Mary Treseler Production Editor: Holly Bauer Copyeditor: Rachel Monaghan Proofreader: Jilly Gagnon Indexer: Lucie Haskins Cover Designer: Mark Paglietti Interior Designers: Ron Bilodeau and Monica Kamsvaag Illustrator: Kara Ebrahim March 2013: First Edition. Revision History for the First Edition: 2013-02-19 First release See http://oreilly.com/catalog/errata.csp?isbn=0636920026334 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Lean Analytics and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trademark claim, the designations have been printed in caps or initial caps. Although the publisher and author have used reasonable care in preparing this book, the information it contains is distributed “as is” and without warranties of any kind. This book is not intended as legal or financial advice, and not all of the recommendations may be suitable for your situation. Professional legal and financial advisors should be consulted, as needed. Neither the publisher nor the author shall be liable for any costs, expenses, or damages resulting from use of or reliance on the information contained in this book. ISBN: 978-1-449-33567-0 [CW] For Riley, who’s already mastered the art of asking “why” five times. —Alistair For my brother, Jacob, who passed away too soon, but inspires me still to challenge myself and take risks. —Ben Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Part One: Stop Lying to Yourself Chapter 1 We’re All Liars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Chapter 2 How to Keep Score. . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter 3 Deciding What to Do with Your Life . . . . . . . . . . . . . . . . 31 Chapter 4 Data-Driven Versus Data-Informed . . . . . . . . . . . . . . . . 37 Part Two: Finding the Right Metric for Right Now Chapter 5 Analytics Frameworks. . . . . . . . . . . . . . . . . . . . . . . . . 45 Chapter 6 The Discipline of One Metric That Matters. . . . . . . . . . . . 55 Chapter 7 What Business Are You In?. . . . . . . . . . . . . . . . . . . . . . 63 Chapter 8 Model One: E-commerce. . . . . . . . . . . . . . . . . . . . . . . 71 xiii Chapter 9 Model Two: Software as a Service (SaaS). . . . . . . . . . . . . 89 Chapter 10 Model Three: Free Mobile App . . . . . . . . . . . . . . . . . . 103 Chapter 11 Model Four: Media Site. . . . . . . . . . . . . . . . . . . . . . . 113 Chapter 12 Model Five: User-Generated Content. . . . . . . . . . . . . . 125 Chapter 13 Model Six: Two-Sided Marketplaces. . . . . . . . . . . . . . . 137 Chapter 14 What Stage Are You At?. . . . . . . . . . . . . . . . . . . . . . . 153 Chapter 15 Stage One: Empathy. . . . . . . . . . . . . . . . . . . . . . . . . 159 Chapter 16 Stage Two: Stickiness . . . . . . . . . . . . . . . . . . . . . . . . 203 Chapter 17 Stage Three: Virality . . . . . . . . . . . . . . . . . . . . . . . . . 227 Chapter 18 Stage Four: Revenue . . . . . . . . . . . . . . . . . . . . . . . . 241 Chapter 19 Stage Five: Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Chapter 20 Model + Stage Drives the Metric You Track. . . . . . . . . . . 265 Part Three: Lines in the Sand Chapter 21 Am I Good Enough? . . . . . . . . . . . . . . . . . . . . . . . . . 273 Chapter 22 E-commerce: Lines in the Sand. . . . . . . . . . . . . . . . . . 293 Chapter 23 SaaS: Lines in the Sand . . . . . . . . . . . . . . . . . . . . . . . 299 Chapter 24 Free Mobile App: Lines in the Sand . . . . . . . . . . . . . . . 309 xiv Index Chapter 25 Media Site: Lines in the Sand . . . . . . . . . . . . . . . . . . . 321 Chapter 26 User-Generated Content: Lines in the Sand. . . . . . . . . . 331 Chapter 27 Two-Sided Marketplaces: Lines in the Sand. . . . . . . . . . 341 Chapter 28 What to Do When You Don’t Have a Baseline . . . . . . . . . 347 Part Four: Putting Lean Analytics to Work Chapter 29 Selling into Enterprise Markets. . . . . . . . . . . . . . . . . . 353 Chapter 30 Lean from Within: Intrapreneurs. . . . . . . . . . . . . . . . . 371 Chapter 31 Conclusion: Beyond Startups . . . . . . . . . . . . . . . . . . . 389 Appendix References and Further Reading. . . . . . . . . . . . . . . . . 393 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 Index xv Foreword For some reason, the Lean Startup movement has proven excellent at producing bumper stickers. Odds are, if you’re reading this, you know some of our most popular additions to the business lexicon: pivot, minimum viable product, Build-Measure-Learn, continuous deployment, or Steve Blank’s famous “get out of the building.” Some of these you can already buy on a t-shirt. Given that the past few years of my life have been dedicated to promoting these concepts, I am not now trying to diminish their importance. We are living through a transformation in the way work is done, and these concepts are key elements of that change. The Lean Series is dedicated to bringing this transformation to life by moving beyond the bumper stickers and diving deep into the details. Lean Analytics takes this mission to a whole new level. On the surface, this new world seems exciting and bold. Innovation, new sources of growth, the glory of product/market fit and the agony of failures and pivots all make for riveting drama. But all of this work rests on a foundation made of far more boring stuff: accounting, math, and metrics. And the traditional accounting metrics—when applied to the uncertainties of innovation—are surprisingly dangerous. We call them vanity metrics, the numbers that make you feel good but seriously mislead. Avoiding them requires a whole new accounting discipline, which I call “innovation accounting.” xvii Trust me, as an entrepreneur, I had no interest in accounting as a subject. To be honest, in far too many of my companies, the accounting was incredibly simple anyway: revenue, margins, free cash flows—they were all zero. But accounting is at the heart of our modern management techniques. Since the days of Frederick Winslow Taylor, we have assessed the skill of managers by comparing their results to the forecast. Beat the plan, get a promotion. Miss the plan, and your stock price declines. And for some kinds of products, this works just fine. Accurate forecasting requires a long and stable operating history from which to make the forecast. The longer and more stable, the more accurate. And yet who really feels like the world is getting more and more stable every day? Whenever conditions change, or we attempt to change them by introducing a truly new product, accurate forecasting becomes nearly impossible. And without that yardstick, how do we evaluate if we’re making progress? If we’re busy building the wrong product, why should we be proud to be doing it on time and on budget? This is the reason we need a new understanding of how to measure progress, both for ourselves as entrepreneurs and managers, as investors in the companies we fund, and the teams under our purview. That is why an accounting revolution is required if we’re to succeed in this new era of work. And Ben and Alistair have done the incredibly hard work of surveying the best thinking on the metrics and analytics, gathering in-depth examples, and breaking new ground in presenting their own frameworks for figuring out which metrics matter, and when. Their work collecting industry-wide benchmarks to use for a variety of key metrics is worth the price of admission all by itself. This is not a theoretical work, but a guide for all practitioners who seek new sources of growth. I wish you happy hunting. Eric Ries San Francisco February 4, 2013 xviii Foreword Preface The Lean Startup movement is galvanizing a generation of entrepreneurs. It helps you identify the riskiest parts of your business plan, then finds ways to reduce those risks in a quick, iterative cycle of learning. Most of its insights boil down to one sentence: Don’t sell what you can make; make what you can sell. And that means figuring out what people want to buy. Unfortunately, it’s hard to know what people really want. Many times, they don’t know themselves. When they tell you, it’s often what they think you want to hear.* What’s worse, as a founder and entrepreneur, you have strong, almost overwhelming preconceptions about how other people think, and these color your decisions in subtle and insidious ways. Analytics can help. Measuring something makes you accountable. You’re forced to confront inconvenient truths. And you don’t spend your life and your money building something nobody wants. Lean Startup helps you structure your progress and identify the riskiest parts of your business, then learn about them quickly so you can adapt. Lean Analytics is used to measure that progress, helping you to ask the most important questions and get clear answers quickly. * http://www.forbes.com/sites/jerrymclaughlin/2012/05/01/would-you-do-this-to-boost-sales-by20-or-more/ xix In this book we show you how to figure out your business model and your stage of growth. We’ll explain how to find the One Metric That Matters to you right now, and how to draw a line in the sand so you know when to step on the gas and when to slam on the brakes. Lean Analytics is the dashboard for every stage of your business, from validating whether a problem is real, to identifying your customers, to deciding what to build, to positioning yourself favorably with a potential acquirer. It can’t force you to act on data—but it can put that data front and center, making it harder for you to ignore, and preventing you from driving off the road entirely. Who This Book Is For This book is for the entrepreneur trying to build something innovative. We’ll walk you through the analytical process, from idea generation to achieving product/market fit and beyond, so this book both is for those starting their entrepreneurial journey as well as those in the middle of it. Web analysts and data scientists may also find this book useful, because it shows how to move beyond traditional “funnel visualizations” and connect their work to more meaningful business discussions. Similarly, business professionals involved in product development, product management, marketing, public relations, and investing will find much of the content relevant, as it will help them understand and assess startups. Most of the tools and techniques we’ll cover were first applied to consumer web applications. Today, however, they matter to a far broader audience: independent local businesses, election managers, business-to-business startups, rogue civil servants trying to change the system from within, and “intrapreneurs” innovating within big, established organizations.* In that respect, Lean Analytics is for anyone trying to make his or her organization more effective. As we wrote this book, we talked with tiny family businesses, global corporations, fledgling startups, campaign organizers, charities, and even religious groups, all of whom were putting lean, analytical approaches to work in their organizations. How This Book Works There’s lots of information in this book. We interviewed over a hundred founders, investors, intrapreneurs, and innovators, many of whom shared * xx An intrapreneur is an entrepreneur within a large organization, often fighting political rather than financial battles and trying to promote change from within. Preface their stories with us, and we’ve included more than 30 case studies. We’ve also listed more than a dozen best-practice patterns you can apply right away. And we’ve broken the content into four big parts. • Part I focuses on an understanding of Lean Startup and basic analytics, and the data-informed mindset you’ll need to succeed. We review a number of existing frameworks for building your startup and introduce our own, analytics-focused one. This is your primer for the world of Lean Analytics. At the end of this section, you’ll have a good understanding of fundamental analytics. • Part II shows you how to apply Lean Analytics to your startup. We look at six sample business models and the five stages that every startup goes through as it discovers the right product and the best target market. We also talk about finding the One Metric That Matters to your business. When you’re done, you’ll know what business you’re in, what stage you’re at, and what to work on. • Part III looks at what’s normal. Unless you have a line in the sand, you don’t know whether you’re doing well or badly. By reading this section, you’ll get some good baselines for key metrics and learn how to set your own targets. • Part IV shows you how to apply Lean Analytics to your organization, changing the culture of consumer- and business-focused startups as well as established businesses. After all, data-driven approaches apply to more than just new companies. At the end of most chapters, we’ve included questions you can answer to help you apply what you’ve read. The Building Blocks Lean Analytics doesn’t exist in a vacuum. We’re an extension of Lean Startup, heavily influenced by customer development and other concepts that have come before. It’s important to understand those building blocks before diving in. Customer Development Customer development—a term coined by entrepreneur and professor Steve Blank—took direct aim at the outdated, “build it and they will come” waterfall method of building products and companies. Customer development is focused on collecting continuous feedback that will have a material impact on the direction of a product and business, every step of the way. Preface xxi Blank first defined customer development in his book The Four Steps to the Epiphany (Cafepress.com) and refined his ideas with Bob Dorf in The Startup Owner’s Manual (K & S Ranch). His definition of a startup is one of the most important concepts in his work: A startup is an organization formed to search for a scalable and repeatable business model. Keep that definition in mind as you read the rest of this book. Lean Startup Eric Ries defined the Lean Startup process when he combined customer development, Agile software development methodologies, and Lean manufacturing practices into a framework for developing products and businesses quickly and efficiently. First applied to new companies, Eric’s work is now being used by organizations of all sizes to disrupt and innovate. After all, Lean isn’t about being cheap or small, it’s about eliminating waste and moving quickly, which is good for organizations of any size. One of Lean Startup’s core concepts is build→measure→learn—the process by which you do everything, from establishing a vision to building product features to developing channels and marketing strategies, as shown in Figure P-1. Within that cycle, Lean Analytics focuses on the measure stage. The faster your organization iterate through the cycle, the more quickly you’ll find the right product and market. If you measure better, you’re more likely to succeed. Figure P-1. The build→measure→learn cycle xxii Preface The cycle isn’t just a way of improving your product. It’s also a good reality check. Building the minimum product necessary is part of what Eric calls innovation accounting, which helps you objectively measure how you’re doing. Lean Analytics is a way of quantifying your innovation, getting you closer and closer to a continuous reality check—in other words, to reality itself. We’d Like to Hear from You Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 (800) 998-9938 (in the United States or Canada) (707) 829-0515 (international or local) (707) 829-0104 (fax) We have a web page for this book where we list errata, examples, and any additional information. You can access this page at: http://oreil.ly/lean_analytics The authors also maintain a website for this book at: http://leananalyticsbook.com/ To comment or ask technical questions about this book, send email to: bookquestions@oreilly.com For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Safari® Books Online Safari Books Online (www.safaribooksonline.com) is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business. Preface xxiii Technology professionals, software developers, web designers, and business and creative professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training. Safari Books Online offers a range of product mixes and pricing programs for organizations, government agencies, and individuals. Subscribers have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and dozens more. For more information about Safari Books Online, please visit us online. Thanks and Acknowledgments This book took a year to write, but decades to learn. It was more of a team effort than most, with dozens of founders, investors, and innovators sharing their stories online and off. Our personal blog readers, as well as the hundreds of subscribers to our Lean Analytics blog who gave us feedback, deserve much of the credit for the clever parts; we deserve all of the blame for the bad bits. Mary Treseler was the voice of our readers and called us out when we strayed too far into jargon. Our families stayed amazingly patient and helped with several rounds of reading and editing. We sent early copies of critical chapters to reviewers, who verified our assumptions and checked our math, and many of them contributed so much useful feedback that they’re practically co-authors. Sonia Gaballa of Nudge Design did great work with our website, and the production team at O’Reilly put up with our unreasonable demands and constant changes. And folks at Totango, Price Intelligently, Chartbeat, Startup Compass, and others all dug into anonymized customer data to enlighten us on things like Software as a Service, pricing, engagement, and average metrics. But most of all, we want to thank people who challenged us, shared with us, and opened their kimonos to tell us the good and bad parts of startups, often having to fight for approval to talk publicly. Some weren’t able to, despite their best efforts, and we’ll leave their stories for another day—but every piece of feedback helped shape this book and our understanding of how analytics and Lean Startup methods intertwine. xxiv Preface P ar t O n e : Stop Lying to Yourself In this part of the book, we’ll look at why you need data to succeed. We’ll tackle some basic analytical concepts like qualitative and quantitative data, vanity metrics, correlation, cohorts, segmentation, and leading indicators. We’ll consider the perils of being too data-driven. And we’ll even think a bit about what you should be doing with your life. It depends on what the meaning of the word “is” is. William Jefferson Clinton C hap t er 1 We’re All Liars Let’s face it: you’re delusional. We’re all delusional—some more than others. Entrepreneurs are the most delusional of all. Entrepreneurs are particularly good at lying to themselves. Lying may even be a prerequisite for succeeding as an entrepreneur—after all, you need to convince others that something is true in the absence of good, hard evidence. You need believers to take a leap of faith with you. As an entrepreneur, you need to live in a semi-delusional state just to survive the inevitable rollercoaster ride of running your startup. Small lies are essential. They create your reality distortion field. They are a necessary part of being an entrepreneur. But if you start believing your own hype, you won’t survive. You’ll go too far into the bubble you’ve created, and you won’t come out until you hit the wall—hard—and that bubble bursts. You need to lie to yourself, but not to the point where you’re jeopardizing your business. That’s where data comes in. Your delusions, no matter how convincing, will wither under the harsh light of data. Analytics is the necessary counterweight to lying, the yin to the yang of hyperbole. Moreover, data-driven learning is the cornerstone of success in startups. It’s how you learn what’s working and iterate toward the right product and market before the money runs out. 3 We’re not suggesting that gut instinct is a bad thing. Instincts are inspiration, and you’ll need to listen to your gut and rely on it throughout the startup journey. But don’t disembowel yourself. Guts matter; you’ve just got to test them. Instincts are experiments. Data is proof. The Lean Startup Movement Innovation is hard work—harder than most people realize. This is true whether you’re a lone startup trying to disrupt an industry or a rogue employee challenging the status quo, tilting at corporate windmills and steering around bureaucratic roadblocks. We get it. Entrepreneurship is crazy, bordering on absurd. Lean Startup provides a framework by which you can more rigorously go about the business of creating something new. Lean Startup delivers a heavy dose of intellectual honesty. Follow the Lean model, and it becomes increasingly hard to lie, especially to yourself. There’s a reason the Lean Startup movement has taken off now. We’re in the midst of a fundamental shift in how companies are built. It’s vanishingly cheap to create the first version of something. Clouds are free. Social media is free. Competitive research is free. Even billing and transactions are free.* We live in a digital world, and the bits don’t cost anything. That means you can build something, measure its effect, and learn from it to build something better the next time. You can iterate quickly, deciding early on if you should double down on your idea or fold and move on to the next one. And that’s where analytics comes in. Learning doesn’t happen accidentally. It’s an integral part of the Lean process. Management guru and author Peter Drucker famously observed, “If you can’t measure it, you can’t manage it.”† Nowhere is this truer than in the Lean model, where successful entrepreneurs build the product, the go-tomarket strategy, and the systems by which to learn what customers want— simultaneously. * When we say “free,” we mean “free from significant upfront investment.” Plenty of cloud and billing services cost money—sometimes more money than you’d spend doing it yourself— once your business is under way. But free, here, means free from outlay in advance of finding your product/market fit. You can use PayPal, or Google Wallet, or Eventbrite, or dozens of other payment and ticketing systems, and pass on the cost of the transaction to your consumers. † 4 In Management: Tasks, Responsibilities, Practices (HarperBusiness), Drucker wrote, “Without productivity objectives, a business does not have direction. Without productivity measurements, it does not have control.” Part One: Stop Lying to Yourself Poking a Hole in Your Reality Distortion Field Most entrepreneurs have been crushed, usually more than once. If you haven’t been solidly trounced on a regular basis, you’re probably doing it wrong, and aren’t taking the risks you need to succeed in a big way. But there’s a moment on the startup rollercoaster where the whole thing comes right off the rails. It’s truly failed. There’s little more to do than turn off the website and close down the bank account. You’re overwhelmed, the challenges are too great, and it’s over. You’ve failed. Long before the actual derailment, you knew this was going to happen. It wasn’t working. But at the time, your reality distortion field was strong enough to keep you going on faith and fumes alone. As a result, you hit the wall at a million miles an hour, lying to yourself the whole time. We’re not arguing against the importance of the reality distortion field— but we do want to poke a few holes in it. Hopefully, as a result, you’ll see the derailment in time to avoid it. We want you to rely less on your reality distortion field, and rely more on Lean Analytics. Case Study Photography—Growth Within | Airbnb Growth Airbnb is an incredible success story. In just a few years, the company has become a powerhouse in the travel industry, providing travelers with an alternative to hotels, and providing individuals who have rooms, apartments, or homes to rent with a new source of income. In 2012, travelers booked over 5 million nights with Airbnb’s service. But it started small, and its founders—adherents to the Lean Startup mindset—took a very methodical approach to their success. At SXSW 2012, Joe Zadeh, Product Lead at Airbnb, shared part of the company’s amazing story. He focused on one aspect of its business: professional photography. It started with a hypothesis: “Hosts with professional photography will get more business. And hosts will sign up for professional photography as a service.” This is where the founders’ gut instincts came in: they had a sense that professional photography would help their business. But rather than implementing it outright, they built a Concierge Minimum Viable Product (MVP) to quickly test their hypothesis. Chapter 1: We’re All Liars 5 What Is a Concierge MVP? The Minimum Viable Product is the smallest thing you can build that will create the value you’ve promised to your market. But nowhere in that definition does it say how much of that offering has to be real. If you’re considering building a ride-sharing service, for example, you can try to connect drivers and passengers the old-fashioned way: by hand. This is a concierge approach. It recognizes that sometimes, building a product—even a minimal one—isn’t worth the investment. The risk you’re investigating is, “Will people accept rides from others?” It’s emphatically not, “Can I build software to match drivers and passengers?” A Concierge MVP won’t scale, but it’s fast and easy in the short term. Now that it’s cheap, even free, to launch a startup, the really scarce resource is attention. A concierge approach in which you run things behind the scenes for the first few customers lets you check whether the need is real; it also helps you understand which things people really use and refine your process before writing a line of code or hiring a single employee. Initial tests of Airbnb’s MVP showed that professionally photographed listings got two to three times more bookings than the market average. This validated the founders’ first hypothesis. And it turned out that hosts were wildly enthusiastic about receiving an offer from Airbnb to take those photographs for them. In mid-to-late 2011, Airbnb had 20 photographers in the field taking pictures for hosts—roughly the same time period where we see the proverbial “hockey stick” of growth in terms of nights booked, shown in Figure 1-1. 6 Part One: Stop Lying to Yourself Figure 1-1. It’s amazing what you can do with 20 photographers and people’s apartments Airbnb experimented further. It watermarked photos to add authenticity. It got customer service to offer professional photography as a service when renters or potential renters called in. It increased the requirements on photo quality. Each step of the way, the company measured the results and adjusted as necessary. The key metric Airbnb tracked was shoots per month, because it had already proven with its Concierge MVP that more professional photographs meant more bookings. By February 2012, Airbnb was doing nearly 5,000 shoots per month and continuing to accelerate the growth of the professional photography program. Summary • Airbnb’s team had a hunch that better photos would increase rentals. • They tested the idea with a Concierge MVP, putting the least effort possible into a test that would give them valid results. • When the experiment showed good results, they built the necessary components and rolled it out to all customers. Chapter 1: We’re All Liars 7 Analytics Lessons Learned Sometimes, growth comes from an aspect of your business you don’t expect. When you think you’ve found a worthwhile idea, decide how to test it quickly, with minimal investment. Define what success looks like beforehand, and know what you’re going to do if your hunch is right. Lean is a great way to build businesses. And analytics ensures that you’ll collect and analyze data. Both fundamentally transform how you think about starting and growing a company. Both are more than processes— they’re mindsets. Lean, analytical thinking is about asking the right questions, and focusing on the one key metric that will produce the change you’re after. With this book, we hope to provide you with the guidance, tools, and evidence to embrace data as a core component of your startup’s success. Ultimately, we want to show you how to use data to build a better startup faster. 8 Part One: Stop Lying to Yourself C hap t er 2 How to Keep Score Analytics is about tracking the metrics that are critical to your business. Usually, those metrics matter because they relate to your business model— where money comes from, how much things cost, how many customers you have, and the effectiveness of your customer acquisition strategies. In a startup, you don’t always know which metrics are key, because you’re not entirely sure what business you’re in. You’re frequently changing the activity you analyze. You’re still trying to find the right product, or the right target audience. In a startup, the purpose of analytics is to find your way to the right product and market before the money runs out. What Makes a Good Metric? Here are some rules of thumb for what makes a good metric—a number that will drive the changes you’re looking for. A good metric is comparative. Being able to compare a metric to other time periods, groups of users, or competitors helps you understand which way things are moving. “Increased conversion from last week” is more meaningful than “2% conversion.” A good metric is understandable. If people can’t remember it and discuss it, it’s much harder to turn a change in the data into a change in the culture. 9 A good metric is a ratio or a rate. Accountants and financial analysts have several ratios they look at to understand, at a glance, the fundamental health of a company.* You need some, too. There are several reasons ratios tend to be the best metrics: • Ratios are easier to act on. Think about driving a car. Distance travelled is informational. But speed—distance per hour—is something you can act on, because it tells you about your current state, and whether you need to go faster or slower to get to your destination on time. • Ratios are inherently comparative. If you compare a daily metric to the same metric over a month, you’ll see whether you’re looking at a sudden spike or a long-term trend. In a car, speed is one metric, but speed right now over average speed this hour shows you a lot about whether you’re accelerating or slowing down. • Ratios are also good for comparing factors that are somehow opposed, or for which there’s an inherent tension. In a car, this might be distance covered divided by traffic tickets. The faster you drive, the more distance you cover—but the more tickets you get. This ratio might suggest whether or not you should be breaking the speed limit. Leaving our car analogy for a moment, consider a startup with free and paid versions of its software. The company has a choice to make: offer a rich set of features for free to acquire new users, or reserve those features for paying customers, so they will spend money to unlock them. Having a full-featured free product might reduce sales, but having a crippled product might reduce new users. You need a metric that combines the two, so you can understand how changes affect overall health. Otherwise, you might do something that increases sales revenue at the expense of growth. A good metric changes the way you behave. This is by far the most important criterion for a metric: what will you do differently based on changes in the metric? • “Accounting” metrics like daily sales revenue, when entered into your spreadsheet, need to make your predictions more accurate. These metrics form the basis of Lean Startup’s innovation accounting, showing you how close you are to an ideal model and whether your actual results are converging on your business plan. * This includes fundamentals such as the price-to-earnings ratio, sales margins, the cost of sales, revenue per employee, and so on. 10 Part One: Stop Lying to Yourself • “Experimental” metrics, like the results of a test, help you to optimize the product, pricing, or market. Changes in these metrics will significantly change your behavior. Agree on what that change will be before you collect the data: if the pink website generates more revenue than the alternative, you’re going pink; if more than half your respondents say they won’t pay for a feature, don’t build it; if your curated MVP doesn’t increase order size by 30%, try something else. Drawing a line in the sand is a great way to enforce a disciplined approach. A good metric changes the way you behave precisely because it’s aligned to your goals of keeping users, encouraging word of mouth, acquiring customers efficiently, or generating revenue. Unfortunately, that’s not always how it happens. Renowned author, entrepreneur, and public speaker Seth Godin cites several examples of this in a blog post entitled “Avoiding false metrics.”* Funnily enough (or maybe not!), one of Seth’s examples, which involves car salespeople, recently happened to Ben. While finalizing the paperwork for his new car, the dealer said to Ben, “You’ll get a call in the next week or so. They’ll want to know about your experience at the dealership. It’s a quick thing, won’t take you more than a minute or two. It’s on a scale from 1 to 5. You’ll give us a 5, right? Nothing in the experience would warrant less, right? If so, I’m very, very sorry, but a 5 would be great.” Ben didn’t give it a lot of thought (and strangely, no one ever did call). Seth would call this a false metric, because the car salesman spent more time asking for a good rating (which was clearly important to him) than he did providing a great experience, which was supposedly what the rating was for in the first place. Misguided sales teams do this too. At one company, Alistair saw a sales executive tie quarterly compensation to the number of deals in the pipeline, rather than to the number of deals closed, or to margin on those sales. Salespeople are coin-operated, so they did what they always do: they followed the money. In this case, that meant a glut of junk leads that took two quarters to clean out of the pipeline—time that would have been far better spent closing qualified prospects. Of course, customer satisfaction or pipeline flow is vital to a successful business. But if you want to change behavior, your metric must be tied to the behavioral change you want. If you measure something and it’s not * http://sethgodin.typepad.com/seths_blog/2012/05/avoiding-false-metrics.html Chapter 2: How to Keep Score 11 attached to a goal, in turn changing your behavior, you’re wasting your time. Worse, you may be lying to yourself and fooling yourself into believing that everything is OK. That’s no way to succeed. One other thing you’ll notice about metrics is that they often come in pairs. Conversion rate (the percentage of people who buy something) is tied to time-to-purchase (how long it takes someone to buy something). Together, they tell you a lot about your cash flow. Similarly, viral coefficient (the number of people a user successfully invites to your service) and viral cycle time (how long it takes them to invite others) drive your adoption rate. As you start to explore the numbers that underpin your business, you’ll notice these pairs. Behind them lurks a fundamental metric like revenue, cash flow, or user adoption. If you want to choose the right metrics, you need to keep five things in mind: Qualitative versus quantitative metrics Qualitative metrics are unstructured, anecdotal, revealing, and hard to aggregate; quantitative metrics involve numbers and statistics, and provide hard numbers but less insight. Vanity versus actionable metrics Vanity metrics might make you feel good, but they don’t change how you act. Actionable metrics change your behavior by helping you pick a course of action. Exploratory versus reporting metrics Exploratory metrics are speculative and try to find unknown insights to give you the upper hand, while reporting metrics keep you abreast of normal, managerial, day-to-day operations. Leading versus lagging metrics Leading metrics give you a predictive understanding of the future; lagging metrics explain the past. Leading metrics are better because you still have time to act on them—the horse hasn’t left the barn yet. Correlated versus causal metrics If two metrics change together, they’re correlated, but if one metric causes another metric to change, they’re causal. If you find a causal relationship between something you want (like revenue) and something you can control (like which ad you show), then you can change the future. 12 Part One: Stop Lying to Yourself Analysts look at specific metrics that drive the business, called key performance indicators (KPIs). Every industry has KPIs—if you’re a restaurant owner, it’s the number of covers (tables) in a night; if you’re an investor, it’s the return on an investment; if you’re a media website, it’s ad clicks; and so on. Qualitative Versus Quantitative Metrics Quantitative data is easy to understand. It’s the numbers we track and measure—for example, sports scores and movie ratings. As soon as something is ranked, counted, or put on a scale, it’s quantified. Quantitative data is nice and scientific, and (assuming you do the math right) you can aggregate it, extrapolate it, and put it into a spreadsheet. But it’s seldom enough to get a business started. You can’t walk up to people, ask them what problems they’re facing, and get a quantitative answer. For that, you need qualitative input. Qualitative data is messy, subjective, and imprecise. It’s the stuff of interviews and debates. It’s hard to quantify. You can’t measure qualitative data easily. If quantitative data answers “what” and “how much,” qualitative data answers “why.” Quantitative data abhors emotion; qualitative data marinates in it. Initially, you’re looking for qualitative data. You’re not measuring results numerically. Instead, you’re speaking to people—specifically, to people you think are potential customers in the right target market. You’re exploring. You’re getting out of the building. Collecting good qualitative data takes preparation. You need to ask specific questions without leading potential customers or skewing their answers. You have to avoid letting your enthusiasm and reality distortion rub off on your interview subjects. Unprepared interviews yield misleading or meaningless results. Vanity Versus Real Metrics Many companies claim they’re data-driven. Unfortunately, while they embrace the data part of that mantra, few focus on the second word: driven. If you have a piece of data on which you cannot act, it’s a vanity metric. If all it does is stroke your ego, it won’t help. You want your data to inform, to guide, to improve your business model, to help you decide on a course of action. Whenever you look at a metric, ask yourself, “What will I do differently based on this information?” If you can’t answer that question, you probably shouldn’t worry about the metric too much. And if you don’t know which Chapter 2: How to Keep Score 13 metrics would change your organization’s behavior, you aren’t being datadriven. You’re floundering in data quicksand. Consider, for example, “total signups.” This is a vanity metric. The number can only increase over time (a classic “up and to the right” graph). It tells us nothing about what those users are doing or whether they’re valuable to us. They may have signed up for the application and vanished forever. “Total active users” is a bit better—assuming that you’ve done a decent job of defining an active user—but it’s still a vanity metric. It will gradually increase over time, too, unless you do something horribly wrong. The real metric of interest—the actionable one—is “percent of users who are active.” This is a critical metric because it tells us about the level of engagement your users have with your product. When you change something about the product, this metric should change, and if you change it in a good way, it should go up. That means you can experiment, learn, and iterate with it. Another interesting metric to look at is “number of users acquired over a specific time period.” Often, this will help you compare different marketing approaches—for example, a Facebook campaign in the first week, a reddit campaign in the second, a Google AdWords campaign in the third, and a LinkedIn campaign in the fourth. Segmenting experiments by time in this way isn’t precise, but it’s relatively easy.* And it’s actionable: if Facebook works better than LinkedIn, you know where to spend your money. Actionable metrics aren’t magic. They won’t tell you what to do—in the previous example, you could try changing your pricing, or your medium, or your wording. The point here is that you’re doing something based on the data you collect. Pattern | Eight Vanity Metrics to Watch Out For It’s easy to fall in love with numbers that go up and to the right. Here’s a list of eight notorious vanity metrics you should avoid. 1. Number of hits. This is a metric from the early, foolish days of the Web. If you have a site with many objects on it, this will be a big number. Count people instead. * 14 A better way is to run the four campaigns concurrently, using analytics to group the users you acquire into distinct segments. You’ll get your answer in one week rather than four, and control for other variables like seasonal variation. We’ll get into more detail about segmentation and cohort analysis later. Part One: Stop Lying to Yourself 2. Number of page views. This is only slightly better than hits, since it counts the number of times someone requests a page. Unless your business model depends on page views (i.e., display advertising inventory), you should count people instead. 3. Number of visits. Is this one person who visits a hundred times, or are a hundred people visiting once? Fail. 4. Number of unique visitors. All this shows you is how many people saw your home page. It tells you nothing about what they did, why they stuck around, or if they left. 5. Number of followers/friends/likes. Counting followers and friends is nothing more than a popularity contest, unless you can get them to do something useful for you. Once you know how many followers will do your bidding when asked, you’ve got something. 6. Time on site/number of pages. These are a poor substitute for actual engagement or activity unless your business is tied to this behavior. If customers spend a lot of time on your support or complaints pages, that’s probably a bad thing. 7. Emails collected. A big mailing list of people excited about your new startup is nice, but until you know how many will open your emails (and act on what’s inside them), this isn’t useful. Send test emails to some of your registered subscribers and see if they’ll do what you tell them. 8. Number of downloads. While it sometimes affects your ranking in app stores, downloads alone don’t lead to real value. Measure activations, account creations, or something else. Exploratory Versus Reporting Metrics Avinash Kaushik, author and Digital Marketing Evangelist at Google, says former US Secretary of Defense Donald Rumsfeld knew a thing or two about analytics. According to Rumsfeld: There are known knowns; there are things we know that we know. There are known unknowns; that is to say there are things that we now know we don’t know. But there are also unknown unknowns— there are things we do not know, we don’t know. Figure 2-1 shows these four kinds of information. Chapter 2: How to Keep Score 15 Figure 2-1. The hidden genius of Donald Rumsfeld The “known unknowns” is a reporting posture—counting money, or users, or lines of code. We know we don’t know the value of the metric, so we go find out. We may use these metrics for accounting (“How many widgets did we sell today?”) or to measure the outcome of an experiment (“Did the green or the red widget sell more?”), but in both cases, we know the metric is needed. The “unknown unknowns” are most relevant to startups: exploring to discover something new that will help you disrupt a market. As we’ll see in the next case study, it’s how Circle of Friends found out that moms were its best users. These “unknown unknowns” are where the magic lives. They lead down plenty of wrong paths, and hopefully toward some kind of “eureka!” moment when the idea falls into place. This fits what Steve Blank says a startup should spend its time doing: searching for a scalable, repeatable business model. Analytics has a role to play in all four of Rumsfeld’s quadrants: • It can check our facts and assumptions—such as open rates or conversion rates—to be sure we’re not kidding ourselves, and check that our business plans are accurate. • It can test our intuitions, turning hypotheses into evidence. • It can provide the data for our spreadsheets, waterfall charts, and board meetings. • It can help us find the nugget of opportunity on which to build a business. 16 Part One: Stop Lying to Yourself In the early stages of your startup, the unknown unknowns matter most, because they can become your secret weapons. Case Study of Moms Explores Its Way to | Circle Success Circle of Friends was a simple idea: a Facebook application that allowed you to organize your friends into circles for targeted content sharing. Mike Greenfield and his co-founders started the company in September 2007, shortly after Facebook launched its developer platform. The timing was perfect: Facebook became an open, viral place to acquire users as quickly as possible and build a startup. There had never been a platform with so many users and that was so open (Facebook had about 50 million users at the time). By mid-2008, Circle of Friends had 10 million users. Mike focused on growth above everything else. “It was a land grab,” he says, and Circle of Friends was clearly viral. But there was a problem. Too few people were actually using the product. According to Mike, less than 20% of circles had any activity whatsoever after their initial creation. “We had a few million monthly uniques from those 10 million users, but as a general social network we knew that wasn’t good enough and monetization would likely be poor.” So Mike went digging. He started looking through the database of users and what they were doing. The company didn’t have an in-depth analytical dashboard at the time, but Mike could still do some exploratory analysis. And he found a segment of users—moms, to be precise—that bucked the poor engagement trend of most users. Here’s what he found: • Their messages to one another were on average 50% longer. • They were 115% more likely to attach a picture to a post they wrote. • They were 110% more likely to engage in a threaded (i.e., deep) conversation. • They had friends who, once invited, were 50% more likely to become engaged users themselves. • They were 75% more likely to click on Facebook notifications. • They were 180% more likely to click on Facebook news feed items. • They were 60% more likely to accept invitations to the app. Chapter 2: How to Keep Score 17 The numbers were so compelling that in June 2008, Mike and his team switched focus completely. They pivoted. And in October 2008, they launched Circle of Moms on Facebook. Initially, numbers dropped as a result of the new focus, but by 2009, the team grew its community to 4.5 million users—and unlike the users who’d been lost in the change, these were actively engaged. The company went through some ups and downs after that, as Facebook limited applications’ abilities to spread virally. Ultimately, the company moved off Facebook, grew independently, and sold to Sugar Inc. in early 2012. Summary • Circle of Friends was a social graph application in the right place at the right time—with the wrong market. • By analyzing patterns of engagement and desirable behavior, then finding out what those users had in common, the company found the right market for its offering. • Once the company had found its target, it focused—all the way to changing its name. Pivot hard or go home, and be prepared to burn some bridges. Analytics Lessons Learned The key to Mike’s success with Circle of Moms was his ability to dig into the data and look for meaningful patterns and opportunities. Mike discovered an “unknown unknown” that led to a big, scary, gutsy bet (drop the generalized Circle of Friends to focus on a specific niche) that was a gamble—but one that was based on data. There’s a “critical mass” of engagement necessary for any community to take off. Mild success may not give you escape velocity. As a result, it’s better to have fervent engagement with a smaller, more easily addressable target market. Virality requires focus. Leading Versus Lagging Metrics Both leading and lagging metrics are useful, but they serve different purposes. A leading metric (sometimes called a leading indicator) tries to predict the future. For example, the current number of prospects in your sales funnel gives you a sense of how many new customers you’ll acquire in the future. 18 Part One: Stop Lying to Yourself If the current number of prospects is very small, you’re not likely to add many new customers. You can increase the number of prospects and expect an increase in new customers. On the other hand, a lagging metric, such as churn (which is the number of customers who leave in a given time period) gives you an indication that there’s a problem—but by the time you’re able to collect the data and identify the problem, it’s too late. The customers who churned out aren’t coming back. That doesn’t mean you can’t act on a lagging metric (i.e., work to improve churn and then measure it again), but it’s akin to closing the barn door after the horses have left. New horses won’t leave, but you’ve already lost a few. In the early days of your startup, you won’t have enough data to know how a current metric relates to one down the road, so measure lagging metrics at first. Lagging metrics are still useful and can provide a solid baseline of performance. For leading indicators to work, you need to be able to do cohort analysis and compare groups of customers over periods of time. Consider, for example, the volume of customer complaints. You might track the number of support calls that happen in a day—once you’ve got a call volume to make that useful. Earlier on, you might track the number of customer complaints in a 90-day period. Both could be leading indicators of churn: if complaints are increasing, it’s likely that more customers will stop using your product or service. As a leading indicator, customer complaints also give you ammunition to dig into what’s going on, figure out why customers are complaining more, and address those issues. Now consider account cancellation or product returns. Both are important metrics—but they measure after the fact. They pinpoint problems, but only after it’s too late to avert the loss of a customer. Churn is important (and we discuss it at length throughout the book), but looking at it myopically won’t let you iterate and adapt at the speed you need. Indicators are everywhere. In an enterprise software company, quarterly new product bookings are a lagging metric of sales success. By contrast, new qualified leads are a leading indicator, because they let you predict sales success ahead of time. But as anyone who’s ever worked in B2B (businessto-business) sales will tell you, in addition to qualified leads you need a good understanding of conversion rate and sales-cycle length. Only then can you make a realistic estimate of how much new business you’ll book. In some cases, a lagging metric for one group within a company is a leading metric for another. For example, we know that the number of quarterly bookings is a lagging metric for salespeople (the contracts are signed already), but for the finance department that’s focused on collecting Chapter 2: How to Keep Score 19 payment, they’re a leading indicator of expected revenue (since the revenue hasn’t yet been realized). Ultimately, you need to decide whether the thing you’re tracking helps you make better decisions sooner. As we’ve said, a real metric has to be actionable. Lagging and leading metrics can both be actionable, but leading indicators show you what will happen, reducing your cycle time and making you leaner. Correlated Versus Causal Metrics In Canada, the use of winter tires is correlated with a decrease in accidents. People put softer winter tires on their cars in cold weather, and there are more accidents in the summer.* Does that mean we should make drivers use winter tires year-round? Almost certainly not—softer tires stop poorly on warm summer roads, and accidents would increase. Other factors, such as the number of hours driven and summer vacations, are likely responsible for the increased accident rates. But looking at a simple correlation without demanding causality leads to some bad decisions. There’s a correlation between ice cream consumption and drowning. Does that mean we should ban ice cream to avert drowning deaths? Or measure ice cream consumption to predict the fortunes of funeral home stock prices? No: ice cream and drowning rates both happen because of summer weather. Finding a correlation between two metrics is a good thing. Correlations can help you predict what will happen. But finding the cause of something means you can change it. Usually, causations aren’t simple one-to-one relationships. Many factors conspire to cause something. In the case of summertime car crashes, we have to consider alcohol consumption, the number of inexperienced drivers on the road, the greater number of daylight hours, summer vacations, and so on. So you’ll seldom get a 100% causal relationship. You’ll get several independent metrics, each of which “explains” a portion of the behavior of the dependent metric. But even a degree of causality is valuable. You prove causality by finding a correlation, then running an experiment in which you control the other variables and measure the difference. This is hard to do because no two users are identical; it’s often impossible to subject a statistically significant number of people to a properly controlled experiment in the real world. * 20 http://www.statcan.gc.ca/pub/82-003-x/2008003/article/10648/c-g/5202438-eng.htm Part One: Stop Lying to Yourself If you have a big enough sample of users, you can run a reliable test without controlling all the other variables, because eventually the impact of the other variables is relatively unimportant. That’s why Google can test subtle factors like the color of a hyperlink,* and why Microsoft knows exactly what effect a slower page load time has on search rates.† But for the average startup, you’ll need to run simpler tests that experiment with only a few things, and then compare how that changed the business. We’ll look at different kinds of testing and segmentation shortly, but for now, recognize this: correlation is good. Causality is great. Sometimes, you may have to settle for the former—but you should always be trying to discover the latter. Moving Targets When picking a goal early on, you’re drawing a line in the sand—not carving it in stone. You’re chasing a moving target, because you really don’t know how to define success. Adjusting your goals and how you define your key metrics is acceptable, provided that you’re being honest with yourself, recognizing the change this means for your business, and not just lowering expectations so that you can keep going in spite of the evidence. When your initial offering—your minimum viable product—is in the market and you’re acquiring early-adopter customers and testing their use of your product, you won’t even know how they’re going to use it (although you’ll have assumptions). Sometimes there’s a huge gulf between what you assume and what users actually do. You might think that people will play your multiplayer game, only to discover that they’re using you as a photo upload service. Unlikely? That’s how Flickr got started. Sometimes, however, the differences are subtler. You might assume your product has to be used daily to succeed, only to find out that’s not so. In these situations, it’s reasonable to update your metrics accordingly, provided that you’re able to prove the value created. * http://gigaom.com/2009/07/09/when-it-comes-to-links-color-matters/ † http://velocityconf.com/velocity2009/public/schedule/detail/8523 Chapter 2: How to Keep Score 21 Case Study House Defines an “Active | HighScore User” HighScore House started as a simple application that allowed parents to list chores and challenges for their children with point values. Kids could complete the tasks, collect points, and redeem the points for rewards they wanted. When HighScore House launched its MVP, the company had several hundred families ready to test it. The founders drew a line in the sand: in order for the MVP to be considered successful, parents and kids would have to each use the application four times per week. These families would be considered “active.” It was a high, but good, bar. After a month or so, the percentage of active families was lower than this line in the sand. The founders were disappointed but determined to keep experimenting in an effort to improve engagement: • They modified the sign-up flow (making it clearer and more educational to increase quality signups and to improve onboarding). • They sent email notifications as daily reminders to parents. • They sent transactional emails to parents based on actions their kids took in the system. There was an incremental improvement each time, but nothing that moved the needle significantly enough to say that the MVP was a success. Then co-founder and CEO Kyle Seaman did something critical: he picked up the phone. Kyle spoke with dozens of parents. He started calling parents who had signed up, but who weren’t active. First he reached out to those that had abandoned HighScore House completely (“churned out”). For many of them, the application wasn’t solving a big enough pain point. That’s fine. The founders never assumed the market was “all parents”—that’s just too broad a definition, particularly for a first version of a product. Kyle was looking for a smaller subset of families where HighScore House would resonate, to narrow the market segment and focus. Kyle then called those families who were using HighScore House, but not using it enough to be defined as active. Many of these families responded positively: “We’re using HighScore House. It’s great. The kids are making their beds consistently for the first time ever!” 22 Part One: Stop Lying to Yourself The response from parents was a surprise. Many of them were using HighScore House only once or twice a week, but they were getting value out of the product. From this, Kyle learned about segmentation and which types of families were more or less interested in what the company was offering. He began to understand that the initial baseline of usage the team had set wasn’t consistent with how engaged customers were using the product. That doesn’t mean the team shouldn’t have taken a guess. Without that initial line in the sand, they would have had no benchmark for learning, and Kyle might not have picked up the phone. But now he really understood his customers. The combination of quantitative and qualitative data was key. As a result of this learning, the team redefined the “active user” threshold to more accurately reflect existing users’ behavior. It was okay for them to adjust a key metric because they truly understood why they were doing it and could justify the change. Summary • HighScore House drew an early, audacious line in the sand—which it couldn’t hit. • The team experimented quickly to improve the number of active users but couldn’t move the needle enough. • They picked up the phone and spoke to customers, realizing that they were creating value for a segment of users with lower usage metrics. Analytics Lessons Learned First, know your customer. There’s no substitute for engaging with customers and users directly. All the numbers in the world can’t explain why something is happening. Pick up the phone right now and call a customer, even one who’s disengaged. Second, make early assumptions and set targets for what you think success looks like, but don’t experiment yourself into oblivion. Lower the bar if necessary, but not for the sake of getting over it: that’s just cheating. Use qualitative data to understand what value you’re creating and adjust only if the new line in the sand reflects how customers (in specific segments) are using your product. Chapter 2: How to Keep Score 23 Segments, Cohorts, A/B Testing, and Multivariate Analysis Testing is at the heart of Lean Analytics. Testing usually involves comparing two things against each other through segmentation, cohort analysis, or A/B testing. These are important concepts for anyone trying to perform the kind of scientific comparison needed to justify a change, so we’ll explain them in some detail here. Segmentation A segment is simply a group that shares some common characteristic. It might be users who run Firefox, or restaurant patrons who make reservations rather than walking in, or passengers who buy first-class tickets, or parents who drive minivans. On websites, you segment visitors according to a range of technical and demographic information, then compare one segment to another. If visitors using the Firefox browser have significantly fewer purchases, do additional testing to find out why. If a disproportionate number of engaged users are coming from Australia, survey them to discover why, and then try to replicate that success in other markets. Segmentation works for any industry and any form of marketing, not just for websites. Direct mail marketers have been segmenting for decades with great success. Cohort Analysis A second kind of analysis, which compares similar groups over time, is cohort analysis. As you build and test your product, you’ll iterate constantly. Users who join you in the first week will have a different experience from those who join later on. For example, all of your users might go through an initial free trial, usage, payment, and abandonment cycle. As this happens, you’ll make changes to your business model. The users who experienced the trial in month one will have a different onboarding experience from those who experience it in month five. How did that affect their churn? To find out, we use cohort analysis. Each group of users is a cohort—participants in an experiment across their lifecycle. You can compare cohorts against one another to see if, on the whole, key metrics are getting better over time. Here’s an example of why cohort analysis is critical for startups. 24 Part One: Stop Lying to Yourself Imagine that you’re running an online retailer. Each month, you acquire a thousand new customers, and they spend some money. Table 2-1 shows your customers’ average revenues from the first five months of the business. January February March April May Total customers 1,000 2,000 3,000 4,000 5,000 Average revenue per customer $5.00 $4.50 $4.33 $4.25 $4.50 Table 2-1. Average revenues for five months From this table, you can’t learn much. Are things getting better or worse? Since you aren’t comparing recent customers to older ones—and because you’re commingling the purchases of a customer who’s been around for five months with those of a brand new one—it’s hard to tell. All this data shows is a slight drop in revenues, then a recovery. But average revenue is pretty static. Now consider the same data, broken out by the month in which that customer group started using the site. As Table 2-2 shows, something important is going on. Customers who arrived in month five are spending, on average, $9 in their first month—nearly double that of those who arrived in month one. That’s huge growth! January February March April May New users 1,000 1,000 1,000 1,000 1,000 Total users 1,000 2,000 3,000 4,000 5,000 Month 1 $5.00 $3.00 $2.00 $1.00 $0.50 $6.00 $4.00 $2.00 $1.00 $7.00 $6.00 $5.00 $8.00 $7.00 Month 2 Month 3 Month 4 Month 5 $9.00 Table 2-2. Comparing revenues by the month customers arrived Another way to understand cohorts is to line up the data by the users’ experience—in the case of Table 2-3, we’ve done this by the number of Chapter 2: How to Keep Score 25 months they’ve used the system. This shows another critical metric: how quickly revenue declines after the first month. Month of use Cohort 1 2 3 4 5 January $5.00 $3.00 $2.00 $1.00 $0.50 February $6.00 $4.00 $2.00 $1.00 March $7.00 $6.00 $5.00 April $8.00 $7.00 May $9.00 Averages $7.00 $5.00 $3.00 $1.00 $0.50 Table 2-3. Cohort analysis of revenue data A cohort analysis presents a much clearer perspective. In this example, poor monetization in early months was diluting the overall health of the metrics. The January cohort—the first row—spent $5 in its first month, then tapered off to only $0.50 in its fifth month. But first-month spending is growing dramatically, and the drop-off seems better, too: April’s cohort spent $8 in its first month and $7 in its second month. A company that seemed stalled is in fact flourishing. And you know what metric to focus on: drop-off in sales after the first month. This kind of reporting allows you to see patterns clearly against the lifecycle of a customer, rather than slicing across all customers blindly without accounting for the natural cycle a customer undergoes. Cohort analysis can be done for revenue, churn, viral word of mouth, support costs, or any other metric you care about. A/B and Multivariate Testing Cohort experiments that compare groups like the one in Table 2-2 are called longitudinal studies, since the data is collected along the natural lifespan of a customer group. By contrast, studies in which different groups of test subjects are given different experiences at the same time are called cross-sectional studies. Showing half of the visitors a blue link and half of them a green link in order to see which group is more likely to click that link is a cross-sectional study. When we’re comparing one attribute of a subject’s experience, such as link color, and assuming everything else is equal, we’re doing A/B testing. 26 Part One: Stop Lying to Yourself You can test everything about your product, but it’s best to focus on the critical steps and assumptions. The results can pay off dramatically: Jay Parmar, co-founder of crowdfunded ticketing site Picatic, told us that simply changing the company’s call to action from “Get started free” to “Try it out free” increased the number of people who clicked on an offer— known as the click-through rate—by 376% for a 10-day period. A/B tests seem relatively simple, but they have a problem. Unless you’re a huge web property—like Bing or Google—with enough traffic to run a test on a single factor like link color or page speed and get an answer quickly, you’ll have more things to test than you have traffic. You might want to test the color of a web page, the text in a call to action, and the picture you’re showing to visitors. Rather than running a series of separate tests one after the other—which will delay your learning cycle—you can analyze them all at once using a technique called multivariate analysis. This relies on statistical analysis of the results to see which of many factors correlates strongly with an improvement in a key metric. Figure 2-2 illustrates these four ways of slicing users into subgroups and analyzing or testing them. Figure 2-2. Cohorts, segments, A/B testing, and multivariate analysis, oh my The Lean Analytics Cycle Much of Lean Analytics is about finding a meaningful metric, then running experiments to improve it until that metric is good enough for you to move to the next problem or the next stage of your business, as shown in Figure 2-3. Chapter 2: How to Keep Score 27 Eventually, you’ll find a business model that is sustainable, repeatable, and growing, and learn how to scale it. Figure 2-3. The circle of life for analytical startups 28 Part One: Stop Lying to Yourself We’ve covered a lot of background on metrics and analytics in this chapter, and your head might be a bit full at this point. You’ve learned: • What makes a good metric • What vanity metrics are and how to avoid them • The difference between qualitative and quantitative metrics, between exploratory and reporting metrics, between leading and lagging metrics, and between correlated and causal metrics • What A/B testing is, and why multivariate testing is more common • The difference between segments and cohorts In the coming chapters, you’ll put all of these dimensions to work on a variety of business models and stages of startup growth. Exercise | Evaluating the Metrics You Track Take a look at the top three to five metrics that you track religiously and review daily. Write them down. Now answer these questions about them: • How many of those metrics are good metrics? • How many do you use to make business decisions, and how many are just vanity metrics? • Can you eliminate any that aren’t adding value? • Are there others that you’re now thinking about that may be more meaningful? Cross off the bad ones and add new ones to the bottom of your list, and let’s keep going through the book. Chapter 2: How to Keep Score 29 C hap t er 3 Deciding What to Do with Your Life As a founder, you’re trying to decide what to spend the next few years of your life working on. The reason you want to be lean and analytical about the process is so that you don’t waste your life building something nobody wants. Or, as Netscape founder and venture capitalist Marc Andreesen puts it, “Markets that don’t exist don’t care how smart you are.”* Hopefully, you have an idea of what you want to build. It’s your blueprint, and it’s what you’ll test with analytics. You need a way of quickly and consistently articulating your hypotheses around that idea, so you can go and verify (or repudiate) them with real customers. To do this, we recommend Ash Maurya’s Lean Canvas, which lays out a clear process for defining and adjusting a business model based on customer development. We’ll discuss Ash’s model later in this chapter. But the canvas is only half of what you need. It’s not just about finding a business that works—you also need to find a business that you want to work on. Strategic consultant, blogger, and designer Bud Caddell has three clear criteria for deciding what to spend your time on: something that you’re good at, that you want to do, and that you can make money doing. Let’s look at the Lean Canvas and Bud’s three criteria in more detail. * http://pmarca-archive.posterous.com/the-pmarca-guide-to-startups-part-4-the-only 31 The Lean Canvas The Lean Canvas is a one-page visual business plan that’s ongoing and actionable. It was created by Ash Maurya, and inspired by Alex Osterwalder’s Business Model Canvas.* As you can see in Figure 3-1, it consists of nine boxes organized on a single sheet of paper, designed to walk you through the most important aspects of any business. Figure 3-1. You can describe your entire business in nine small boxes The Lean Canvas is fantastic at identifying the areas of biggest risk and enforcing intellectual honesty. When you’re trying to decide if you’ve got a real business opportunity, Ash says you should consider the following: 1. Problem: Have you identified real problems people know they have? 2. Customer segments: Do you know your target markets? Do you know how to target messages to them as distinct groups? 3. Unique value proposition: Have you found a clear, distinctive, memorable way to explain why you’re better or different? 4. Solution: Can you solve the problems in the right way? * 32 http://www.businessmodelgeneration.com/canvas Part One: Stop Lying to Yourself 5. Channels: How will you get your product or service to your customers, and their money back to you? 6. Revenue streams: Where will the money come from? Will it be onetime or recurring? The result of a direct transaction (e.g., buying a meal) or something indirect (magazine subscriptions)? 7. Cost structure: What are the direct, variable, and indirect costs you’ll have to pay for when you run the business? 8. Metrics: Do you know what numbers to track to understand if you’re making progress? 9. Unfair advantage: What is the “force multiplier” that will make your efforts have greater impact than your competitors? We encourage every startup to use Lean Canvas. It’s an enlightening experience, and well worth the effort. What Should You Work On? The Lean Canvas provides a formal framework to help you choose and steer your business. But there’s another, more human, side to all of this. Do you want to do it? This doesn’t get asked enough. Investors say they look for passionate founders who really care about solving a problem. But it’s seldom called out as something to which you should devote much thought. If you’re going to survive as a founder, you have to find the intersection of demand (for your product), ability (for you to make it), and desire (for you to care about it). That trifecta is often overlooked, withering under the harsh light of data and a flood of customer feedback. But it shouldn’t. Don’t start a business you’re going to hate. Life is too short, and your weariness will show. Bud Caddell has an amazingly simple diagram of how people should choose what to work on, shown in Figure 3-2. Chapter 3: Deciding What to Do with Your Life 33 Figure 3-2. Bud Caddell’s diagram belongs on every career counselor’s wall Bud’s diagram shows three overlapping rings: what you like to do, what you’re good at, and what you can be paid to do. For each intersection between rings, he suggests a course of action: • If you want to do something and are good at it, but can’t be paid to do it, learn to monetize. • If you’re good at something and can be paid to do it, but don’t like doing it, learn to say no. • If you like to do something and can be paid to do it, but aren’t very good at it, learn to do it well. This isn’t just great advice for career counselors; when launching a new venture, you need to properly assess these three dimensions as well. 34 Part One: Stop Lying to Yourself First, ask yourself: can I do this thing I’m hoping to do, well? This is about your ability to satisfy your market’s need better than your competitors, and it’s a combination of design skill, coding, branding, and myriad other factors. If you identify a real need, you won’t be the only one satisfying it, and you’ll need all the talent you can muster in order to succeed. Do you have a network of friends and contacts who can give you an unfair advantage that improves your odds? Do you have the talent to do the things that matter really well? Never start a company on a level playing field— that’s where everyone else is standing. These same rules apply to people working in larger organizations. Don’t launch a new product or enter a new market unless your existing product and market affords you an unfair advantage. Young competitors with fewer legacies will be fighting you for market share, and your size should be an advantage, not a handicap. Second, figure out whether you like doing this thing. Startups will consume your life, and they’ll be a constant source of aggravation. Your business will compete with your friends, your partner, your children, and your hobbies. You need to believe in what you’re doing so that you’ll keep at it and ride through the good times and the bad. Would you work on it even if you weren’t being paid? Is it a problem worth solving, that you’ll brag about to others? Is it something that will take your career in the direction you want, and give you the right reputation within your existing organization? If not, maybe you should keep looking. Finally, be sure you can make money doing it.* This is about the market’s need. You have to be able to extract enough money from customers for the value you’ll deliver, and do so without spending a lot to acquire those customers—and the process of acquiring them and extracting their money has to scale independent of you as a founder. For an intrapreneur, this question needs to be answered simply to get approval for the project, but remember that you’re fighting the opportunity cost—whatever the organization could be doing instead, or the profitability of the existing business. If what you’re doing isn’t likely to have a material impact on the bottom line, maybe you should look elsewhere. This is by far the most important of the three; the other two are easy, because they’re up to you. But now you have to figure out if anyone will pay you for what you can and want to build. * Not everyone is hoping to make money with his or her startup. Some people are doing it for attention, or to fix government, or to make the world a better place. If that’s you, replace “money” with “produce the results I’m hoping to achieve” as you read this book. Chapter 3: Deciding What to Do with Your Life 35 In the early stages of a startup, you’ll be dealing with a lot of data. You’re awash in the tides of opinion, and buffeted by whatever feedback you’ve heard most recently. Never forget that you’re trying to answer three fundamental questions: • Have I identified a problem worth solving? • Is the solution I’m proposing the right one? • Do I actually want to solve it? Or, more succinctly: should I go build this thing? Exercise | Create a Lean Canvas Go to http://leancanvas.com to create your first canvas. Pick an idea or project you’re working on now, or something you’ve been thinking about. Spend 20 minutes on the canvas and see what it looks like. Fill in the boxes based on the numbered order, but feel free to skip boxes that you can’t fill out. We’ll wait. How did you do? Can you see what areas of your idea or business are the riskiest? Are you excited about tackling those areas of risk now that you see them described in the canvas? If you’re confident, share your Lean Canvas with someone else (an investor, advisor, or colleague) and use it as a discussion starter. 36 Part One: Stop Lying to Yourself C hap t er 4 Data-Driven Versus Data-Informed Data is a powerful thing. It can be addictive, making you overanalyze everything. But much of what we actually do is unconscious, based on past experience and pragmatism. And with good reason: relying on wisdom and experience, rather than rigid analysis, helps us get through our day. After all, you don’t run A/B testing before deciding what pants to put on in the morning; if you did, you’d never get out the door. One of the criticisms of Lean Startup is that it’s too data-driven. Rather than be a slave to the data, these critics say, we should use it as a tool. We should be data-informed, not data-driven. Mostly, they’re just being lazy, and looking for reasons not to do the hard work. But sometimes, they have a point: using data to optimize one part of your business, without stepping back and looking at the big picture, can be dangerous—even fatal. Consider travel agency Orbitz and its discovery that Mac users were willing to reserve a more expensive hotel room. CTO Roger Liew told the Wall Street Journal, “We had the intuition [that Mac users are 40% more likely to book a four- or five-star hotel than PC users and to stay in more expensive rooms], and we were able to confirm it based on the data.”* On the one hand, an algorithm that ignores seemingly unrelated customer data (in this case, whether visitors were using a Mac) wouldn’t have found this opportunity to increase revenues. On the other hand, an algorithm that * http://online.wsj.com/article/SB10001424052702304458604577488822667325882.html 37 blindly optimizes based on customer data, regardless of its relationship to the sale, may have unintended consequences—like bad PR. Data-driven machine optimization, when not moderated by human judgment, can cause problems. Years ago, Gail Ennis, then CMO of analytics giant Omniture, told one of us that users of the company’s content optimization tools had to temper machine optimization with human judgment. Left to its own devices, the software quickly learned that scantily clad women generated a far higher click-through rate on web pages than other forms of content. But that click-through rate was a short-term gain, offset by damage to the brand of the company that relied on it. So Omniture’s software works alongside curators who understand the bigger picture and provide suitable imagery for the machine to test. Humans do inspiration; machines do validation. In mathematics, a local maximum is the largest value of a function within a given neighborhood.* That doesn’t mean it’s the largest possible value, just the largest one in a particular range. As an analogy, consider a lake on a mountainside. The water isn’t at its lowest possible level—that would be sea level—but it’s at the lowest possible level in the area surrounding the lake. Optimization is all about finding the lowest or highest values of a particular function. A machine can find the optimal settings for something, but only within the constraints and problem space of which it’s aware, in much the same way that the water in a mountainside lake can’t find the lowest possible value, just the lowest value within the constraints provided. To understand the problem with constrained optimization, imagine that you’re given three wheels and asked to evolve the best, most stable vehicle. After many iterations of pitting different wheel layouts against one another, you come up with a tricycle-like configuration. It’s the optimal threewheeled configuration. Data-driven optimization can perform this kind of iterative improvement. What it can’t do, however, is say, “You know what? Four wheels would be way better!” Math is good at optimizing a known system; humans are good at finding a new one. Put another way, change favors local maxima; innovation favors global disruption. In his book River Out Of Eden (Basic Books), Richard Dawkins uses the analogy of a flowing river to describe evolution. Evolution, he explains, can create the eye. In fact, it can create dozens of versions of it, for * 38 http://en.wikipedia.org/wiki/Maxima_and_minima Part One: Stop Lying to Yourself wasps, octopods, humans, eagles, and whales. What it can’t do well is go backward: once you have an eye that’s useful, slight mutations don’t usually yield improvements. A human won’t evolve an eagle’s eye, because the intermediate steps all result in bad eyesight. Machine-only optimization suffers from similar limitations as evolution. If you’re optimizing for local maxima, you might be missing a bigger, more important opportunity. It’s your job to be the intelligent designer to data’s evolution. Many of the startup founders with whom we’ve spoken have a fundamental mistrust of leaving their businesses to numbers alone. They want to trust their guts. They’re uneasy with their companies being optimized without a soul, and see the need to look at the bigger picture of the market, the problem they’re solving, and their fundamental business models. Ultimately, quantitative data is great for testing hypotheses, but it’s lousy for generating new ones unless combined with human introspection. Pattern | How to Think Like a Data Scientist Monica Rogati, a data scientist at LinkedIn, gave us the following 10 common pitfalls that entrepreneurs should avoid as they dig into the data their startups capture. 1. Assuming the data is clean. Cleaning the data you capture is often most of the work, and the simple act of cleaning it up can often reveal important patterns. “Is an instrumentation bug causing 30% of your numbers to be null?” asks Monica. “Do you really have that many users in the 90210 zip code?” Check your data at the door to be sure it’s valid and useful. 2. Not normalizing. Let’s say you’re making a list of popular wedding destinations. You could count the number of people flying in for a wedding, but unless you consider the total number of air travellers coming to that city as well, you’ll just get a list of cities with busy airports. 3. Excluding outliers. Those 21 people u...