The Lean Startup, Part II (how to make your startup "ship" software that rocks customer's lives)

Is as follows. We add a fourth state to the system, which is, okay, the code is done, but have we validated whether this task was worth doing in the first place or not yet? And then we capacity constrain that bucket too. So what happens is, let's say we have five people, we do five tasks in flight, we complete those five tasks and then we go get the next five. But while we're working on the next five, somebody needs to be figuring out whether customers, for example, actually liked the five features that we previously did. Of course, we don't do them in batches of five. They flow through one at a time. If by the time you're ready to deploy the next feature, you still haven't figured out whether the previous feature was any good, there's no room to move it to the next bucket. So the capacity constraint immediately brings things to a halt. It starts to back things up and focus the team to get together and say, wait a minute, wait a minute. How do we know that that feature we shipped last week was actually any good? Yeah, did we bother to measure? Did we ask anybody? Did we do any kind of testing? We had this realization, oh, no, actually we didn't. And it's like, okay, why did we do that? Why would you go on to the next thing when you haven't learned the previous thing was actually having a positive impact? Because whether you mean to or not, each feature, each thing you do is informed by what you've learned in the past. And so when you decline an opportunity to learn, you, you're actually very likely to be working on stuff that doesn't matter. So teams that apply that very simple discipline, it's a simple technique, but has a very important long term impact, which is it reorients us from thinking of the code being written as our unit of progress to having learned something. Right. When I work with teams that start practicing, say, split testing, that's the usual way, when in doubt, split test.

What does that mean?

Oh, sorry. In an ab test, say 50% of your customers see the new feature, but 50% don't, and we'll see if that changes their behavior in any way. That's like, there are a lot of other ways to do validation of an idea, but that's the easiest one. It's kind of when in doubt, test. Most of the teams I work with that are not doing that kind of testing in product development, they might be doing it in direct marketing or something like that, but in product development, when they start doing it, discover that between 60 and 80% of the features that they're shipping have no impact on customer behavior. None. And that's really demoralizing. A lot of them are like, we gotta stop doing this. Split testing is driving us crazy. It's like, hold on, hold on. Your new features are so great, having such impact that they affect customer behavior. Not at all. So customers are indifferent to whether they're there or not. Why are we doing them again? And then first you get through a wave of excuses. Well, it looks so much better because products that don't have split testing tend to get prettier over time, whereas products with split testing get more effective over time. So it's like, was our goal to have a pretty product or one that actually helped customers accomplish their goals? Oh, right. Our goal is to change how customers behave because that is ultimately what our product is supposed to be about. This sounds like Google. Yeah, they do a lot of split testing.

It drives designers nuts.

It absolutely does.

Doug Bowman left and went to Twitter and now he's being driven to Twitter.

That's right. And I want to. There's one important distinction to be made. Big companies that do split testing generally are using split testing to optimize. Right. The classic 21 colors of blue. It's like, well, we want to squeeze out every tenth of a percent improvement in conversion rates. Okay. That is very useful for certain circumstances. For startups, what I try to do is get them to only measure the macro. That is judge their split test only by the top five to 10 key metrics of customer behavior. So that even if you're testing a minor thing like a button change, we don't measure the click through rate of the button. We don't care. We weren't doing it for that reason. We were changing this page because we wanted customers to be more likely to download our app or pay us money or be retained or invite a friend. And if the micro thing doesn't have an impact on those macro metrics, why are we doing it? So then that's a way of getting people to focus not on the micro optimizations, which most is classic premature optimization. Most startups that haven't achieved product market fit yet are still trying to get every point of conversion out of a page. It's like, no, no, no. We need to learn which of our fundamental hypotheses about customers are wrong. Let's run those experiments. And so it's a way of kind of up leveling people's thinking to, oh, right, Focus on the experiments that actually matter. Not matter to customers, matter to our learning about what our product really needs to become. Wow.

That'S pretty good stuff. Because most companies don't think about testing. Do customers really want this? That's really the last half an hour.

Talking about, kind of would rather not know. It's a very human thing to, you know, it's fun to stay in stealth mode, it's fun to stay in planning mode. That's where you're kind of living in a fantasy. And that's very entertaining. But if our goal is really to change the world, then we have to be really in the world through the whole process of building a new product. And it's very difficult to maintain your vision and also be kind of dealing with feedback. And that's what these techniques are really about, is giving you kind of the routine discipline to be like testing as a reflex rather than as like a special one time thing you do. Like. See, it proves that I'm right.

On the other hand, in the last 10 years doing this kind of testing has gotten dramatically easier.

Dramatically easier. Right.

We can spin up a new. I was talking with, oh God, a guy who builds databases and I said, oh, don't you have to have a database to test, test all this stuff?

He goes, not really.

You just go to Rackspace or go to Amazon, slide your credit card in and start up a server and set up another one and then find some data somewhere and shove it in there and test it out. Yeah.

One of the lean startup success stories. I know you're very familiar with Aardvark, those guys did something that was really smart. They would use Mechanical Turk to fake features that they hadn't built yet. So there are a lot of situations where they expose a test of a new feature to a customer that they planned one day to back it up with a really amazing computational algorithm. But in the meantime, they just have a human being actually do the task and then present the results back to the user just to make sure that customers actually would want that incredible algorithm if they ever had the breakthrough science necessary to build it. Right. And you see a lot of that.

And actually did they just keep that Mechanical Turk? Because that's sort of what we all do on Aardvark. Right? Questions all the time during the day. Would you like to answer this question about Silicon Valley? Sure, why not? They're not paying me and I'm answering their question for free. It's a brilliant system.

That's right. It absolutely is. And they were able to iterate their way into a system that actually made it fun for people to want to engage in those questions and actually worked because they were very focused on having a great design, building great technology, but doing that in the service of actually helping people get the information they needed. And whenever they had to choose between their precious whatever and the effective choice which would be to actually help people, they chose to help people. And that's why they got a lot to celebrate. And that's the general pattern that we see when we're reality based, then it actually frees us.

Now they got bought by Google, right?

Yeah.

How do you think that's going to go? Do you think they're going to continue lean startup methods when they get into Google?

I'm very interested to see. I have high hopes because a number of the founders had worked at Google before and so they really knew what they were getting into. I think some of the other acquisitions, classic thing in a big company is people don't exactly understand the culture of the new place they're going and that integration doesn't always happen seamlessly, as you know. But I'm hopeful we'll see. I think of all the big companies that I've had the chance to work with in my consulting practice, Google is one of the most open to this kind of thinking.

It's interesting talking with engineers who work.

There or work there.

They have infrastructure constraints which make doing this kind of iteration tough. Google Buzz was developed over three years. So I keep joking, it's a copy of Friendfeed. Well, no, it was actually developed two or three years ago. In fact, it was partly developed by the guys who started Friendfeed. They left, did Friendfeed in the same amount of time and shipped it before Google Buzz could ship it.

That's right. I mean that was the. When we were doing imvu, we had a point where we had to decide if we wanted to sell to Google and we decided not to and take venture financing and go on our own path. And then subsequently Google came out with their own 3D environment called Google Lively. But they, you know, and you know, it was a kind of a Microsoft Y kind of thing to do to say, you know, hey, you know, you should really, you should let us buy you or we're going to compete with you and yada yada. And we were kind of like, hey, whoa, Google, really? But you know, it just, that happened to be the dynamic of that situation. Yeah, Google Lively didn't come out then for another two years and it wound up being like, it wound up having copied a lot of elements from Imbu, but in the meantime we had Been learning and iterating over that whole time. And so when they finally came out, they did the classic, they made all the classic big company mistakes. They waited too long to launch, they tried to make the product too elaborate and then they launched it way too publicly. Most startups don't realize that one of the big advantages you have is that you have a pathetically small number of customers. The obscurity that we languish in and we're always like, oh, why can't we get more customers? Is a gift. The gift is you can experiment wildly and nobody cares. You know, use that to perfect the thing in micro scale and then launch. Big companies really forget that.

It's interesting. I've been following this company called Pipio which does a social network, a new kind of social network. I almost don't, I almost didn't want to talk about them because the act of talking about them got them customers, you know, or got them users. And users put demands on them. And I'm scared now that they'll never get to really where they need to be to really build a good system, you know, and Google does is the same way they're hearing feedback. I mean they built a database to handle the feedback that they're getting, right? And so now they have a stack ranked version of the feedback. You know, they could have just shown it to two or three people and we would have given them the same feedback.

I think that's right. It's pretty improving the model in micro scale before you scale it up and prematurely launching is really dangerous. And it's too bad. I know that experience you're having where you have a team, you're like, boy, I don't want to expose them to too much feedback because they might lose their way. But if you think about it, if the company's at all successful the majority of its life, it's going to be spending in public getting feedback instead of being afraid of that phase and trying to delay it. What I try to teach companies to do is to develop a discipline for coping with that feedback early and often. In fact, try to eliminate the stealth period or minimize it as much as is practical so that company doesn't get confused. Like, yeah, you get all kinds of feedback from everywhere. But if the company is systematically testing its assumptions, then it knows what to do with that feedback. It knows how to slot it in, it knows when to test it, when to ignore it and when to say, wait a minute, that's actually an unexpectedly good idea, let's go test it out. If you're in that kind of testing, lightweight, testing mentality. It's okay, you know, nobody can knock you off your game because, you know, hey, if that's an interesting idea, maybe we'll test it. And you don't have to commit to all. It's not all or nothing. You go all in very rarely.

Yeah. Did you work with Aardvark? What did they learn by trying this methodology? Because they probably weren't familiar with it all that much before they started their company. Right?

Yeah, Well, I.

They learned that at Aardvark. They didn't learn it before, did they?

Well, they. I mean, I don't want to take credit for this example. Success they've had by any means. I think the founders there came in with a lot of really good ideas about how to test ideas that I can't take credit for. We work together because we had really compatible ideas, not because I taught them everything they know. Now that they're successful, of course, everything they know, it's so tempting to take credit, but it's totally inappropriate. What I will say is that what I admired about them from the first time I met them was that they had an unswerving commitment to. To finding out the truth, their point of view. They had kind of a philosopher's background, which was, our goal is to build something that really will matter to customers, and that's our highest priority. So even though we have a lot of cool ideas, every cool idea, no matter how cool it seems, is going to have to be subjected to some testing. And so they went, give me an.

Example of a cool idea that they had, and how did they test it?

I don't want to speak out of turn. So they spent a number of months, they incorporated the company without a firm idea of exactly what the product was going to be. And instead of just picking the first good idea they found, they spent months in kind of building prototype after prototype of different kinds of products that could lead to different businesses that were all kind of in their area of interest. It's the thing called social surge. But Aardvark, the thing that we now know and love, was one of the later concepts that they tested because they didn't want to get prematurely committed to something that wasn't right. I thought it was a really smart way to go about founding the company to say, look, we have a vision of an outcome we want to see, but the path is confusing. We want to really try some different routes, and in each one, not just build a concept, not just do a paper prototype, but actually build Some version of semi working software, what we call a minimum viable product, and actually expose some customers to it and kind of see does their behavior actually conform to what we thought. And also does the technology got the perfect idea, but the technology is not right.

So you said, back up 20 seconds just to make sure I got it because.

No, no, that's fine.

I was watching.

Yeah. One of the things that impressed me about Aardvark was they started the company and then were really rigorous about testing a number of different product concepts, all of which were compatible with their overall vision, but none of which they were specifically wedded to until they'd gotten a sense that not only did this thing kind of work for customers, but also that the technology was actually going to be good enough to do what they were promising to customers. And that's why we talk about having this problem team, solution team, parallel process. If you just take customer development, the idea that you should, before you build something, go out and talk to customers and figure out what their problem is to make sure you're solving an important problem. If you don't have that coupled with a kind of solution, minimum viable product process, I always say, you always wind up building teleportation. You start asking customers, what's the most important thing we could solve for you? If you really walk up that hierarchy of pain, you're like, boy, customers would really like infinite money, the ability to teleport, extreme longevity. You start to come up with problems that are too big that you actually don't know how to solve. And so what? The goal of the vision constrains a certain search space. And then the kind of problem team is looking for customer pain in that space. Solution team is what can we do in this space? And it's in that overlap where you start to find, okay, here are the areas where early adopters can be found to buy a product that's not quite ready. The kind of people that Robert Scobles of the world who will use an 80% solution and give you feedback about it instead of saying, this is stupid, or people who will tell their friends about it instead of waiting for their friends to tell them, those are people you have to start with. You take their feedback in and then again, you don't just do what they say. Because Robert Scoville's not exactly the mainstream customer. Right. He's not exactly like other people. He's special.

No, actually that's a big mistake a lot of people make is, oh, Scoble. We can discount Scoville because he's an Outlier. He's a weirdo. I remember hearing that when I had 1000 followers on Twitter. And now a lot of people have a thousand followers on Twitter.

That's exactly right.

So if you had said that his feedback is wrong, you would have said a future group of people are wrong.

And that's, you know, for people, this technology life cycle adoption curve, seriously, early adopters are your customers and also mainstream customers are your customers, just at different times and they have different needs. And so, so we always say feedback is not about you, it's about them. When someone gives you feedback, it doesn't tell you a thing about your product. It tells you about the person speaking. And you have to understand that feedback in context, figure out how it affects your choice of vision. And then going back to the Aardvark story, understand how the information you're getting interacts with the path you're trying to go on. Is it time to double down on this product or to pivot to a different product? And that's a judgment call. There's no spreadsheet that will tell you for sure it's time to pivot or not. But if you make solid predictions and you make data based decisions, my experience is your judgment gets better over time because you're actually using your brain hardware. Basically. We're basically associate of maps. At the end of the day, that's what we're good at. And we start to develop an intuitive feel for what really works with customers and what doesn't. And that allows us to pivot and iterate faster over time, not slower. And that's the essence of the art of entrepreneurship in the era of a lean startup.

Interesting. So what's this conference you're doing?

Oh, well, Aardvark is the perfect segue because they are going to be one of our case studies and people who are interested in hearing the story straight from the horse's mouth. They'll be there. So, April 23rd in San Francisco. It's called Startup Lessons Learned. Sllconf.com, sll conf. And my goal with that conference is to build the most coherent startup conference of all time. Coherent in the sense that we're not just going to have people standing up on stage giving you random advice. That's when I was an entrepreneur. This is what drove me crazy. People want to give you startup advice and it's always at the form, well, I did X and then I made a lot of money. So if you do X, you too will make a lot of money. Yeah. Well, how Do I know that that's going to work for me? How do I know that you really did that? Heck, how do I know you actually made any money? So what we're going to do is.

And even if you did, it was a different time.

It was different times. How do I know it will work now?

It won't almost by definition. Well, yeah, because like, YouTube was built five years ago in a different time. And so if you try to copy YouTube and what they did, you're gonna fail, Right?

I think that's right.

Our expectations are different. And so I love the fact that you test your theories out.

Right.

You know, oh, you want to do a YouTube? Well, build a little prototype and show it to your friends and see if they like it. And they'll probably tell you, why would I use this? I'm on YouTube already, why.

Right. Doesn't solve a problem for me anymore. Yep.

But chat roulette came out, right?

That's right.

So there is opportunity to do another video startup. Right. But you have to, if you want to be in video, you got to try some new things, not copy what the old guys said.

That's right. And so the only enduring advice are the principles of innovation, not the practices and the tactics. That's really. That's what the Lean Startup is all about. So the conference is going to be organized around this Build, measure, learn feedback loop that we talk about in the Lean Startup. Our goal is to optimize total time to that feedback loop. And so we're going to. The conference will be one day divided into three sections around each of those stages of the feedback loop.

Build, measure, learn.

Build, measure, learn. And our goal is not to do any one of those things. Well, our goal is to do the whole loop as fast as possible. So in each of the stages, we're going to have a keynote address. In the build phase, we have Kent Beck, the father of extreme programming and a signatory to the Agile Manifesto. In the measured phase, our keynote is from Randy Comisar, who wrote the book Getting to Plan B and a really great business book called the Monk and the Riddle. He's a partner at Kleiner Perkins and he's going to be talking about pivots and how do you. Why is it that successful startups never actually work with plan A? Why are they always on Plan B? Or more likely, Plan Z?

I think that's a human thing.

Yeah, it's pretty human.

My marriage 2.0 worked a lot better. Marriage 1.0.

Now we're outside my realm of qualifications.

Well, you know, hopefully you never figure that one out. But my best advice for people getting married 2.0 first.

That is a getting to plan B story. And then in the third phase, our closing keynote is going to be from Steve Blank, the creator of customer development. And then in each of those modules we're going to have the keynote address followed by case studies from actual entrepreneurs. So that's going to be entrepreneurs like Aardvark who have already exited. We're going to have entrepreneurs who are bootstrappers, who are relatively early on, who are later on people who came out of big companies to do a startup venture backed, not venture backed, really a diverse range. Aardvark, Grocket, Dropbox, companies that are really grappling with these issues now so that the experience that they're having a is relevant, it's contemporary experience. It's based in the Internet of 2010 and also is informed by this theory. So they're not just going to say randomly here's the things that we did, but here's how it fits into the overall methodology. And then in each module the third part is going to be we're going to focus on a dilemma. That's okay, let's say you buy into the theory, but what about. And these questions are kind of based on the really difficult questions people have been asking me over the past year as I've traveled around the world evangelizing these ideas. So does this idea of rapid deployment, 50 deployments a day, does that scale? Sure, I can see how you do that in a five person team, but in a 75 person team really? So we'll have a case study from entrepreneurs who have actually done that in the measure phase. Who's done that? What's that? Well, we're going to bring IMVU to talk about now that they're a 75 person company, how does it work now? What did they have to change? How did they stay true to those principles over time? And then I've asked every presenter to tell the truth. What didn't work right? What about the model was hard, what was the things, the unexpected problems they had and if they had to throw stuff out, what did they have to throw out? Then the question is what's the role of design? To your question about the famous case of Google, when you're split testing everything, when you're building a minimum viable product, is there still a role for good design? What about the Steve Jobs, the supposed Steve Jobs effect of just design designing. Look at Gowel and I did it right and I think that in that.

I think design is going to be how companies differentiate themselves. And goal is a good example of that. Goal actually, in some ways has fewer features than Foursquare. But I still am attracted to gowala because of its superior design.

And there's this tension between how do you get great designers to work in an environment where their ideas are going to be tested against objective reality? And my experience is that the best designers love that because they are like, wait a minute, I'm not going to be subject to the whim of some random product manager, but if my ideas are better, they're truly going to win out. That's very exciting. But my experience is that for mediocre designers, it's terrifying because they never, like a lot of people, are used to a world where you can win a design award for designing a product that never gets built. That's a relic from the age of waterfall. We're exiting that. As design becomes important, it has to become integrated into product development at the core.

It's interesting because startup, even CEOs don't do this, right? You know, let's do that. Well, I've been inside failed startups. Unfortunately, I've never had that win, you know, but I can see the failures pretty clearly. And one of them is they don't manage by who. Who actually has the better idea. That's right.

You know, we live or die by the extent that. To which we're a meritocracy. I mean, that's the Silicon Valley way. And yet we talk a really good game about that. But in a lot of cases, we do not live up to that value. And, you know, you can see it in the lack of diversity in the people that we hire. Sometimes you can see it in the companies that make decisions by what it says on your business card instead of who's got the data to support it. And that's not about bad people, it's about bad systems. And so part of Lean Startup is learning to see entrepreneurship at a systems level, not getting caught up in the individual personalities and egos of, well, that person says, or this great CEO says. This is not fortune cookie entrepreneurship. We're trying to actually figure out what really works and what doesn't.

Yeah.

So that's the conference. It's April 23rd. It's going to be in San Francisco.

Do you talk about infrastructure at all?

We will, but because, you know, as we're talking the build phase, we'll be really getting into the details of how do you build, like a continuous deployment system how do you actually do that Agile thing I was talking about of constraining what you're working on based on what you learn? And there are really clear infrastructure components. But the conference is designed to be experienced by cross functional teams. So my experience teaching this now for a while is it's really key to have the CEO and the CTO in the same room at the same time. Otherwise when I meet with MBAs and CEOs and business people, they're always like, this sounds great, but my technical guys will never go for it. But when I meet with the technical teams by themselves, they're like, this is great, but my CEO will never go for it. But if they're sitting in the same room at the same time, they kind of go like this and they kind of turn their heads and say, wait, oh, could we actually do this and have a serious conversation? So actually the conference is designed optimally for teams. There's actually a team ticket and we're going to treat the teams like VIPs, their own special seating. We've got a bunch of really quality mentors who've agreed to come to the conference and sit with the teams and actually help them to, to kind of take in what they're hearing and figure out how to apply it to their company. So yeah, we want to have technical people there, we want to have non technical people. We're trying to walk that line to make the content substantive and actionable for both groups, which is a hard thing to do, but I've had now something like a year's practice at doing that.

This is really aimed at startups, right?

Yes.

Are you going to have a separate conference for big company guys who need to learn these techniques? No, no, no.

In fact, the big company entrepreneurs are welcome. And here's what I mean by that. We're used to thinking of startups as two guys in a garage. And if you're two guys in a garage, that you're a startup. But I think that definition is way wrong. First of which, first problem is that a lot of two guys in a garage are actually just building a small business. They're not actually trying to create something new. They're just perfectly happy to find some interesting way to make a little bit of money and they're fine with that. My definition of a startup is a human institution to designed to create something new under conditions of extreme uncertainty. And if you think about the three part definition, human institution, which says our goal is not just to create a great product, but to create a company that builds great products. So it's fundamentally a management challenge, which is very weird based on this kind of two guys in a garage. You don't think of them as managers. And entrepreneurial management is very different from general management. What you learn, you get your mba, and that's fine. I don't mean to say that that kind of management is bad and this is good. They're just two different things. We're trying to create something new, so we're actually trying to change the world through the creation of a new product or service. What Clay Christensen calls disruptive innovation. And under conditions of extreme uncertainty is the most important part. That means we don't even know who the customer is yet. We don't know what the market will achieve. We don't know what the price point should be. Our goal is to be on a search for those answers. And if you think about that definition, human institutions, new product, extreme uncertainty doesn't say anything about size of company. Doesn't matter. I meet what I call involuntary entrepreneurs all the time. People who went to work for big companies. They thought they were taking a general management job, but they discovered, oops, my safe industry is being disrupted. So now what? It's like, listen, you're an entrepreneur whether you meant to be or not. So anybody who's facing those challenges is welcome to the conference. It is for entrepreneurs. So we're asking that service vendors and other such people, unless they want to come and sponsor, not just come and try to sell to the entrepreneurs. But we really want to have a substantive conversation among people who are actually grappling with these problems. My aspiration is to kind of take the movement, lean startup to the next level, because starting to deliberately misunderstand it. You're starting to see this whole lean startup versus fat startup controversy, which is about how much money you raise, which to me is like asking, how tall should you be to be an entrepreneur? It's an orthogonal question. It's secondary to what's the process you use to build companies. So we're starting to get this kind of people confusing it. People are starting to get into flame wars about who really coined the term lean startup and who gets credit for this. And that's like, okay, first of all, congratulations, we've gone. Being misunderstood is way better than being ignored. That's a huge step up. And yet we're still at the very, very, very early phases of this starting to dawn in the awareness of some people. And it's like, now's a chance for us to really, really make sure that we actually have the ideas. Right. So ignore who's getting credit. Ignore all the kind of ephemera around what is starting to coalesce to be a movement and actually get the people who care and who want to get at the truth in the same room at the same time and have a conversation about what is the future of entrepreneurship really going to look like? I think it's going to be really fun.

Very cool. Well, thank you so much.

Oh, thank you.

This is really fun. It's been a great hour. I appreciate it.

I hope you can come in, April. Thank you.

All right, good.