The risks and ethics of running #leanstartup experiments in large organizations.

Land. What was the race dream game? Mario Kart.

Mario Kart.

Mario Kart. I used to like playing Sonic Sonic the Hedgehog on Sega.

I remember those days.

Alex Kidd.

Remember 007? GoldenEye came out for N64.

I used to play dog hunt.

And then 64. I couldn't work out how to use it. I was so used to platform games. It's like going to this leg. It's too complicated. Don't give up.

Although seeing the games now are pretty. Pretty mumbling. Yeah. If you're going to like a GameStop or something and just stand out and watch. Really cool stuff. Just the advancements and stuff.

Oh, yeah, that's cool.

At the car convention here in New York, they had an Oculus Rift hooked up to a driving game inside of a like actual simulator for real drivers. So the. The whole seat itself would shuffle and shake and stuff like that. You know, you're in the Rift.

It was really cool.

People got rid of.

We are live, by the way.

This is recorded.

I don't know. I'm trying to work that out.

I've never used 10. And can I say, Jonathan.

Don'T see the things that most people wouldn't say.

Yeah.

Just imagine your mother's watching. She's still checking up on your.

I think she's accepted me now. Yeah.

Warts and all.

What? Warts and all.

No.

What?

No, it's a euphemism In England.

My over under was seven, so I had to call Boris.

Hello.

This is a giant table.

I know.

We were a little optimistic.

Yeah.

We'Re live. Excellent. No, we got zero viewers. Is it recording? Because.

You are on tv. All right, we'll turn around in a second. All right, awesome. Thanks for coming. Pretty much.

Thank you.

Some sort of a prize, but in a very meta sort of way. That was sort of the point of tonight was you're going to ship it no matter what. We do have a couple goals in mind which even in a small meeting have been met. So a have a meeting, ship it. Do something. Do something where we also brought in some people, not just in the office. So we wanted to start getting out into the community and see who's really doing this stuff and not necessarily just talking about it or meet upping about it or tweeting about it. So yeah, we wanted to ship something in August. We're calling this our summer vacation via. But starting in September, we want to do this every month and really share stories and talk to the community and see what's going on. See who's really Running experiments. The topic for tonight was sort of just an excuse to get people in a room. But I think it actually, even in a small room, maybe some compelling conversations. And it came from something that really happened actually in this room. You know, us getting into a debate with people at Pearson and what's okay to experiment on what's not, what's pushing the envelope, what's the ends justify them in. So just spent a half an hour to an hour talking about that. First of all, around the room, standard icebreakers. Who you are, where you come from, why you're here. And maybe one line sort of like. So mine is like, you know, the journey of a million miles starts with a single step. So you just gotta start getting something even if it's small, even if it's embarrassing. Right. So that would probably be my. My Icebreaker game. So some. Some line like that, like a. At the ship it kind of line. What's your favorite principle? What's your favorite principle or what can.

We say on this?

You can say you're on. Oh, and you are. We're recording this. So this is going out in real time. We are recording this.

So we have two viewers.

Oh, we have two. So epic. Ship it first, we'll. You'll start with you.

My name is Luis, Product manager, Future Tech, Pearson. We look at gaps in our product portfolio and try to accelerate some of the work in combination with some lines of businesses. My favorite principle is time is your most precious resource. Use it deliberately. All right.

Does it have to be a favorite principle?

I was going to try to link it back to shipping Fast experiments into.

The themes in that vein. Then I will link it to that. My name's Lee. I'm also a colleague of Pearson. I work on the learning and development team and I'm going to be working on program creation. So not. Not much the product development side, but wanting to incorporate similar principles into how we develop programs at Pearson. And I guess my principle would be shit or get off the pot.

Nice. Have you ever bleed stuff out?

Yeah.

That's pretty awesome.

Yeah.

So I'm August Gressens. I'm actually in technology at Scholastic, but I have a very great interest in lean product development and product development from a technology side and I'm interested in everything this lean thing. So I've just started coming to some startups. My favorite concept is failing fast.

Fantastic. My name is Verma. We're a company called alphaux, which is a platform for enterprise product teams to provide them with meaningful insight throughout their product development cycle. My Philosophy is just make it real. Make it real enough to get feedback quickly.

All right. Jonathan Burtfield. I'm also at Pearson and I'm a member of the product lifecycle coaching team, helping business units and product teams across.

The company.

Kind of roll out the product life cycle and change the way they operate. Our principle, be a starter and a finisher is maybe something that comes to mind. Some people I know who entrepreneurs are better starters than finishers. And so. But I think it's experimenting is about starting and executing is about finishing. I think they need to go hand in hand.

It's interesting because I was just thinking of the quote that was going around the office, which was stop starting and start finishing. I guess today is actually the opposite of that. I guess I'll go debate myself and.

Go, yeah, you should see their interaction in the office. Emerging podcast square.

I would agree.

I don't know. I'm seeing a. A new thing starting here.

Yeah. Yeah.

I think it's called a bromance.

Yeah. Yeah.

All right, I'll go. I remembered that I did my quote.

But I did not say who I.

Was or introduce myself.

Adam Burke.

I work with this guy and cross from them coaching teams inside Pearson to run experiments. And so this meetup, you know, like I said, we wanted to start doing things in the community, but this specific topic, this came from something I was coaching a team and pushing them really hard, you know, to run an experiment and test it this way. And, you know, got the line.

We.

Could never test that. What if they get pissed off or what if, you know, what when they find out it doesn't, it's not real or, you know, we can never test that. It's not right. We have to build the whole thing. So my philosophy is there are, you know, the ethics of wasting money to build something that nobody necessarily needs or hasn't asked for, might not work, is not necessarily more ethical than running on real people. So that's. Even if we spend 30 solid minutes talking. That's sort of what I wanted to talk about. That's where I came from.

I'm Boris. So I currently work with hima health insurance company and I help connect design experiments to enterprise goals in various ways. And I guess philosophy, I can only think of anti philosophy.

Those who.

Those who can do, teach. I kind of think it's the other. I believe in it, but the other way around. I think teaching is awesome.

So. Cool.

All right, where should we start? Should we start on an actual experiments and actual trial? I think given the. Are you actively Running experiments at Scholastic.

Well, I'm working on a different type of transformation project which is around engineering test driven development. So I'm going through a similar process to you guys are trying to work with teams of engineers to introduce this practice which changes the way they need to think about writing software. And so it's interesting, right? I mean, especially in these big companies where you have these sort of people that have been here a long time, they're used to working the same way. So I'm interested in these kinds of issues of how you approach them and how you. How you can kind of try to change that mindset. Sounds like you've run into something similar here where there's this sort of static notion of the product as being this thing that's completely defined ahead of time. And it's more of a waterfall process of upfront process of defining that big thing and then building it and then shipping it.

Right.

You guys are trying to change the. With the methodology around that to encourage experimentation and questioning all the assumptions that are implicit in defining this big thing. Right. In smaller iterations. Is that right?

In general, I definitely say that that's right. Yeah. You want to answer that question? Then we'll follow up with the yes.

Yeah. I think what we talk about a lot is that it's not necessarily about the size of the thing. It's not necessarily about the prescribing focus. More kind of not falling into the pattern of knowing and more shaping ourselves up to be learners. Whenever we're building something big or small, whether it's about engineering or it's about product side or marketing or sales, it's testing before we execute. And so the nature of those experiments, the nature of that cultural shift is more important than the kind of the specifics of what exactly is being tested.

Starting with something small that is a working thing that you get feedback on and then can iterate on.

Right.

That's.

That's important. Like same thing in software development to this test driven development encourages writing small pieces of code, testing those things, and then refactoring and adding more. But I mean, I think the problem of how you change the culture and the mindset and people's thinking around that in established organizations is really hard. Right.

Why? That's sort of where we. Why, why are people afraid to ship.

Early.

In small pieces?

Why are they afraid to. Because the entire system that I've encountered is you're punished for admitting you don't know something.

This is what I've seen.

Absolutely.

Most definitely.

All the rhetoric says we should test all that. All the behavior that I've seen so far is worked out. Tell me that we're gonna make a lot of money. Don't show me anything that contradicts that conversation.

So failure is something to be avoided.

And not to be learned from. It's a bad thing. Exactly.

Because I think fundamentally, I think most people at companies just want to risk their buffs. Really, that's the main driver. Right. So there was a conversation and I won't go into obviously the specifics since this might be published, but supposedly there.

Was a story in which it is being published.

That's what I'm saying, therefore I'm not going to specific. But for example, in one situation, you're looking at technology that uses facial recognition technologies to actually determine pain points or highlights, emotional highlights of a person. Right? So this company came in for a very particular product we were developing, or not us, but that company, you know, paid, engaged and the results came back really ugly. And they literally kept on the coffer. Okay, thanks, we're done. They went forward anyways and they just kind of shoved it under the rug. Right. And to me, that was saying that there is no willingness to, there's no appetite for failing or at least in the midst of which is human nature.

So being punished for learning even when the truth hurts kind of thing like.

Well, I think there's a short termism around, around impact to the brand.

Right.

People are very afraid that if the brand is associated with anything other than unbridled success that it's somehow going to affect their capacity to continue to operate. So, you know, anything which is publicly clear as a failure.

There'S a lot.

Of, lot of shame around that.

So this is why I'm fascinated with what's his name, Power of Habit, who wrote the book Power of Habit. Do you know what I'm talking about? Basically makes the argument that a culture is merely the summation of habits. And all you're really arguing then is that there's no habits in place that actually support the behavior you're talking about. So unless you actually address and change habits, you're never actually going to address culture.

Yeah, that's good.

But there's, there's something.

I'm sorry, I can't get too excited.

No, it's good. That's right.

Yeah. They got me drunk first and they're.

Like, oh, nothing is censored other than that.

Okay.

No, I think, I think it is a cultural thing. It is a cultural thing from the top down. In large organizations, right. People won't talk about those kind of step up slip ups.

So are you guys familiar with Bridgewater Hezbollah? Do you know their whole culture there? Oh my God. Why isn't that normal? That's a better question.

Explain it. Yeah, yeah.

How would you explore us?

Just completely open send, Everything's discussed. I can't even talk about. You're not allowed to talk about another.

Person unless they're in the room.

You can search anyone's email.

You can, yeah, you can.

You just completely.

I don't know, do they see each other's.

Yeah, yeah. So they call it. So they call something called. There are two things that they like fundamentally advise, like radical transparency and the idea of meritocracy.

Right?

Where there is an expectation there that you speak up, it doesn't matter if you're a junior analyst there. Like if you think your boss is incompetent, you say it publicly and you ground it with evidence and reasons to support them and hash it out so, you know, experience it for a little bit. It kind of puts you on edge at first where you're like, whoa, people just are saying these things on the walls when you show up. There is like people's work on the walls and saying crap, bullshit, like terrible fire, this person, you name it, it's all up in the air. And supposedly you talk to the teams and what they say is that there's a relief that that's actually when you start to create a culture of a willingness to try something and stress test. You know, they follow five things, right? They hire people that work for themselves. At the end of the day, that two can independently muster up the most independent thoughts possible. Three are willing to find the smartest people to stress test everything. Four are weary of being overconfident. And five, reflect often with this belief that pain plus reflection equals progress. Right? Like everything you guys talk about, this experiment, lean culture, I see it there. At least I get that impression. But again, it's such a minuscule example. Like, at what point are we being idealistic? How realistic is everyone actually capable of embracing this type of culture? Or are we being foolish in its pursuit?

Can you retrofit that culture, though?

How so?

Like, I think a culture like that can be created if you have a visionary leader and you organically build people. But if you have an established culture, then trying to embed that any form of change is going to be really painful. Why embedding new things, very difficult is because you get rewarded on doing pretty much what you did yesterday. That's your job is to do what you did yesterday, ideally marginally better. But if you've built a whole career and you've got a salary for doing.

Something.

The most likely thing you're going to do tomorrow at work is that thing rather than try overhaul and do.

So it's I think when you're in large I feel.

Established process.

I think it's realistic to say that we can't seem to single handedly change behaviors.

Right.

So I agree with not having I think a culture is a result of behaviors.

Right.

So there's nothing ephemeral. It just mention behaviors and processes. People have way to describe it. So you go in you I'm going to do new stuff today. Well everything else down screen for years going to break than in just you. So maybe we're asking people to do. You know, ideally you wanted to make a lot of changes but once the organization is established, I'm just thinking out loud. I don't know if there's a way to say you change because you can't.

So my question becomes, do you guys know of any example where this culture has changed?

Where this culture has changed.

You guys are bad producing, operating in a negative way. You should be experiment, learning, failing, driven. Other examples I'm assuming.

Well there's some classic examples into it is a place where the leadership developed that culture.

I think that's the key, right? So the leadership at the very top, it has to come from the top of an organization. You can't be like oh yeah, we're gonna just be smarter than everybody else and better because you're just not going to be accepted. But immediate adjacent people See I've seen that. I've seen a company inside company where they would bring outside help and they would train the entire company. Everybody would get in. I mean but they could do it because then they had location probably it wasn't I think less than people. So they could actually go in and say everybody look, we're doing this now.

How many people work at that what's called Bridgewater.

So Bridgewater is the most successful financial firm in history. They have a thousand employees. So they own more. They like double the assets that Goldman Sachs has with a 40th employee size. Like it considers obviously they predicted every major trend and they capitalize off every downturn, upturn, you name it, they just executed. And the reason when you talk to them are you learned about the culture. They're like we're not willing, we're not unwilling to breach any issue and topic and force each other to stress test the quality of our work.

It's very much. They're built on. Every problem is solvable. If you can strip it apart, then you can rebuild it so they, you know, whilst most hedge funds are making random guesses, they all go, this variable affects this. And try to test it through.

So there, there was a. Yeah, there was one video they showed where it was a conversation that recorded just as the economy had tanked in 2008, 2009, and they saw their reports and they were up like 20% profit. It's like, no one has achieved this, right? Everyone's cheering around. And then Ray Dalio, the CEO, just like shuts everyone up and is like, stop it, we're done. Where are we going to be wrong? It just immediately jumps into the issue of constantly being overweight. They're leery of celebrating anything. That's just a cool. So when you, When I hear experiments from you guys, what I'm really hearing is a willingness to stress us, stress test something. I just don't think that's a human nature. It's built or honed in or how do you. How do you even develop an appetite for it?

I think it is human nature.

I think we tinker or experiment.

Well, experiments can be quite broad. So just say, I'm cooking a meal for strangers. I will first begin to ask questions to what do you guys want to eat? That's like a survey. That's a very elementary form of experimentation. Then I might cook something, whatever it is, some portion of the meal, maybe just the main or the dessert of the starter. Or maybe even just tell people, I'm thinking of baking this with this ingredient. What do you reckon? And they'll be like, I don't know.

Can I ask you a question?

Yeah.

Why do you care in the end? Why do people care about surveying or asking other people? I'm not trying to. What's the intended outcome of trying to survey everyone?

I think surveys, it's a starting point. It's never. You should never build anything off a survey. It's self reported, but you just want to. It's the first. It's like, I've got 20 ideas. How am I going to prioritize which one to even go with? So I think that's where a survey comes in, comes in handy. Like, what are the five most important things? Prioritize these features on the phone. Okay, let's begin testing if your number one actually holds true. It's just a. I think every experiment is just a prioritization thing.

Well, no, kind of the idea in the example you showed about like you're serving your friends or something. My argument is at the end of the day it's to appease social acceptance. I think we're social creating.

I really do.

Fundamentally, I think are huge driver for us just social acceptance, appeasing whatever environment you're in or situation. And I never try to forget that and you know, ask questions or an experiment or how people make purchasing decisions in business events.

Right.

I don't know.

I just. I always wonder at the end of the day, in the back of my mind just like just want to press.

Someone well if you want them to purchase and use your product.

But.

Right. Let's say you are a. All right. Like, let's say you are being asked to. Like someone comes to me like, hey, you should buy 5,000 of these computers for your company because they're XYZ benefits. Right. Like, I feel like fundamentally the person that has a decision to choose say yes or no is where's my boss? I guess what I'm trying to get at is I think the reason culture is so difficult to change is because in many institutions they're built on the idea of appeasement and pleasing some sort of hierarchy system. And why I'm so fascinated by the Bridgewater culture is that they strip that out. That is the one that is frowned upon is willingness to just do peace ones. They try to scrutinize the quality of the work itself as opposed to the social reasons behind it.

Yeah, I think it's great. Yeah, I think that's a great. If you look at the flip side, which is amplified sales. Who do you sell to? You don't sell based on what the company actually needs. Based on what Jobs want.

Yeah.

To appear as or his personal interests are. I think it's ultimately person. And that's why for enterprises buy and what they actually could ideally use. So it's like it's recognized and it's leveraged for billions and billions of dollars in sales.

Because that's how I've been wondering if.

Some of the disconnect is just around the. Where a lot of these established products are in their life cycle. You know, they're in like the exploit phase and they've been in the exploit phase for 30 years or something. You know, especially with a lot of the traditional, you know, publishing products that are made by Pearson or companies like Scholastic. So the people that innovated on those project products and got them to where they are to be profitable are dead. Or, you know, they're. They're they're long gone. And the people that are there are just kind of manning the machinery. So, you know, they don't, they don't necessarily have that mindset.

Right.

That should come with, like, digital product development. I think a lot of this is really about the introduction of digital product development and all the technology and the change that that is bringing in, ushering in.

Right.

Like in a big company like Pearson, like, isn't that, isn't that what the, this is really about with what you guys are doing? Isn't it about digital product development or is it about product development in general?

First, I'll ask you what the exploit phase is, because we saw something similar today.

So just to go back, the exploit phase would just be a mature, proven business model.

Literally from people, like, listening or when we cut up the website, like, is it from someone's book? Is it from, like. Oh, yeah. A lot of time we throw Eric Reese's name around just so people want to follow up on that.

I think I picked this up in Lean Enterprise talks about the product portfolio and the different, different phases of product development.

So what would be the phases in that?

In that.

I don't remember all the quiz. I don't remember. Sort of like, you know, the early phase where you're kind of doing a lot of testing and experimentation and where it actually talks about these ideas of failure and how they're less acceptable and they actually, failure is more risky for an established product than one that hasn't been established.

Right.

So. And there's sort of almost like different cultures that go along with those things. But I'm thinking about these, these really big companies, like publishing companies that have established products that have been around for years. And there's really been no need for any kind of culture of innovation there because their business models have been proven and they should have been business as usual for so long that no one can even remember what it was like to innovate. But now with the introduction of digital, we need to be able to innovate again. Right. And in order to compete in the marketplace.

So I don't think it's certainly from a Pearson perspective, we're not solely focused on the introduction of digital into the universe as a driver for certainly experimentation or cultural shift. It's much more about speed. The market moves much faster, competitors move much faster. A lot of that is powered by digital. But it doesn't mean that necessarily the products that are being delivered are always digital. Some of them are services delivered by humans, and some of them still remain kind of Very primitive digital, even if they are digital, super power charged. But I think the big issue for me around experimentation, from what you're describing, Bridge Warden, from what we do here and from what I see other companies, the focus is really on evidence. And it's on evidence driving decision making as opposed to power or gut instinct or I've been around for 30 years. So therefore it's about abstracting the decision away from that human, human focus. It doesn't mean that that's completely abstract. It doesn't mean that like strategy and the wisdom and the knowledge of the market disappear. They need to be applied in concert with something much more powerful, which is data or which is evidence. And that comes from experimentation and the capacity to open yourself up to say, my wisdom, my knowledge is not quite as important as what the market is saying or our customers are saying what the data is saying. That's the cultural shift that happens. And it sounds like the Bridgewater example is a great way where it's not idiots running a financial company and having success because they challenge each other. It's really super smart people who are able to marry kind of the benefit of testing stuff out, using that data and then interpreting it.

But I think you're leading to something then that I don't know if you guys have seen it. Please let me know if you have. All right, let's say people are, yeah, we should be data driven, right? Look for disconfirming data associating.

Right?

That's what I'm interpreting here, sir. They don't know how. They don't know what good data is or bad data. They see a number and they're like fat. Have you seen this as well? Or what are your thoughts? How do you mitigate that?

I think that or seeing a number at all counts as data. So when you say something like.

I.

Want to think of an example, conducting a review to understand if a program at a company is an efficacious program or an effective program to yield the outcome that it wants to yield. I can say I'm going to go collect quantitative data on retention. But if that's not talking about the impact on the business, then it's just a number. It's still quantitative data, but it doesn't at all affect what I'm trying to test for. I think there's also an emotional attachment to our ideas and that's something that I saw in my last role when I got a glimmer of early stage product development. And that was there is a very deep emotional attachment to people that are very tied to that market and have been tied there for 30 years. And so they are market experts in that, in that discipline. Therefore they definitely know what the market needs at all times. And there is definitely a sense of failure if that idea wasn't pushed through. And I think in order to innovate in, in the way that you all are discussing, you have to detach the emotions from the ideas that you're generating.

So I guess then my question to you too is at what point do you focus your energies on trying to convert everyone or do you just get rid of them and find a ballistic culture? I know.

But.

Right.

Like you wrote a similar where if you don't adapt, you die. Right. Like that's. We're fundamentally facing a time issue. Like at what point do you devote resources to believe in the person and change or just cut.

I think it's like getting them to change. It's quite ironic when you think about it, like, and I think we were having this discussion a few weeks ago.

Is that.

Experimentation appears everywhere in life, but for some reason it doesn't happen in business. They walk through a door in the morning and it stops. They'll be on the golf course and they're content going to the driving range and smashing 100 balls. They're content missing the green. You know, if they were a stand up comedian, they'll go to an open mic and rehearse.

Just talked about that.

Yeah. If, you know they're cooking a meal, if they're learning a new sport, they do that always. If they're going to build a new house, they'll expect their architect to models. But they walk through the door and all bets are off. So I think the culturally, the ability, it's there, but it's something not to do with the individual that's preventing this from happening. It's the incentive structure.

It's the structure.

Right.

So most of the exams you get, you're gonna have control of what you're doing, whereas you walk into an organization and now you, you're just a small part of the structure that's already there.

So if you were going to change. That's kind of what I'm saying. Right.

Sporting team, like try to think of that.

That's another thing you feature in a soccer team.

Team.

Can you experiment alone?

You do in the gym every time.

You can experiment. Maybe like one small thing at a time.

Yeah.

You fairly constrained.

Yeah.

The coach has some bearing on what.

Happens on game day, I think to your phenomenon. Bridgewater calls it impression management. You have organizations that spend more time concerned about impression management than actually producing solid outcomes, Right. Where their focus is on the output as. And they don't know the difference between outputs and outcomes, and they focus on the former.

Yeah, But I think it's also to do with stakes, Right. So, like, again, thinking about the sports example, right, the stakes are kind of a small moment in time, right inside of a game or a match or whatever. And like, let's try a different move. And we've experimented and see whether I can curl the ball around if I'm soccer player or whatever. I think what happens inside of organizations is people feel an existential level of stakes to experimentation because experimentation, if it's done right, can result in failure. And therefore, if I do that, I'm going to lose my job and my livelihood, and therefore, you know, my family, my rent, my mortgage, whatever, is kind of directly connected to, like, that culture of experimentation. It's not accepted.

And the whole solution is not just pushing experimentation, but matching it with time. Right. So in the World Series, you're not going to experiment with a new pitch, but you might increase.

Right.

And as a chef, you're not going to like when the critic comes in. So it's not that experimentation is right all the time at every time for every audience, but it's making them feel comfortable in sort of what we're saying at the beginning and explore, you know, an explore with when we're saying when you're learning that it's okay, but knowing the difference between where the stakes are high and when the stakes are low. And I think that's sort of the job that we're all going to do. Is that preserving the execution side, preserving stuff that works and preserving the stuff where you said there's no reason to change. Is that true? But then somehow adding, you know, safe.

It almost seems like in an organization, it's game day every day.

Right, That's.

But if you look at an. If you look at an athlete, if you count the hours that they're involved in preparing for game day as a fraction of their career, they're probably on the field for, what is it, 20 hours a year? Like 30 hours.

Football player, baseball might be a little higher. Yeah.

Thousands of hours for reworking, Thousands of.

Hours in the gym, in the pool, working on nutrition, trying different things. But suddenly in the organization, it's always game day.

It's like, super interesting.

Yeah.

So let's see your question.

I feel like an underlying assumption.

I think we're out to something.

All right.

He's going to think for a second.

An underlying assumption of this entire conversation. Experiments equals better. Is there any situation where it's detrimental? Yeah.

When the risks are not. Yeah. When the chef. When the critic is coming in that day and you don't have, you don't have the ability to fuck up.

What does that translate into an organization.

When you what I was going to say it's about the stakes. It's like you want to do the maximum amount of experimentation when the stakes are low and the minimum experimentation when the stakes are high. So like when you've got, in an educational context, when you've got a back to school product, right, you don't want to be experimenting with that product in July because you've got to get a lot of ducks in a row so that when all those kids are signing on or those students are in university or whatever is the thing, they've got a solid reliable product. But how do you do that? You don't hold your breath and avoid any testing until you get to July.

And it makes sense mathematically.

Right.

So the amount of risk that's going to affect your cash cow, right. Futures. Now you've got your entire business, right. You might have the upside that you look at the downside of. Dave, I can't afford to take this risk.

Nobody's drawing that distinction though between low stakes season and high stakes season.

Well, here's the thing then, right? This is where I've seen a lot of problem, at least within our organization is that the same person responsible for not taking risks is now being asked to take risks. Like the same individual person responsible both for sustaining and growing as opposed to, well, also be innovative and start building new ideas. I wonder how realistic it is to expect that same individual to code switch between the two mindsets, essentially.

So again, the answer might be not changing the person, but changing the sandbox.

One caveat to throw in is, is sometimes I think when it's extra small, so non issue, I mean it's amazing, but certain things you just want to go and do, right. Someday if you say like okay, it doesn't matter. It doesn't matter what the best put the best, I don't know exact shape of water bottle is. Or just this is good enough for this party. We'll go with this one because it's.

Not going to make you different.

Whereas maybe the shape of water bottle.

And we're going to ship a million of them, then let's, you know, let's test it out and make sure it's the Right.

Market.

I think there's also a culture around if it's not the best product or if it's not the product that's gaining most of the market share, then it's still a failure. So good enough isn't good enough. It always has to be the best. So there's not even a culture around experimenting with. Not products that would fail miserably, but products that will do just enough to get you relevant information. And on top of that, I think something I saw in a previous role was not having the culture around constant feedback and around allocating time to experimentation because you are just maintaining what's always worked because that's what makes you money.

Biggest driver of what that culture has.

Yeah, yeah. I mean, that's one of the biggest dangers, right. Is a legacy business making a statement that we need to continue doing what we've always done because that's our cash cow, without putting the perspective of how do we know that we're actually maximizing our opportunity every single day that our product is out there? They're just making an assumption that what they. The market has not changed, their competition has not changed, their customers have not changed. And if we continue to do it, we'll continue to get the same margins, same revenue.

Alternative means to think. Contrary to popular belief, the human brain is not designed for thinking.

But doesn't the market. Doesn't the market demand? Like if you're a public company, the market is demanding growth. The market is demanding maximizing opportunity. The market isn't saying standstill. Right. And so the question is, well, how do you grow?

Right.

The biggest risk that you are taking, the biggest experiment that you're running, is if we do nothing, we'll just continue to grow because, you know, we've got a brand and we've got a legacy product, and therefore, you know, that's an experiment. We're just not acknowledging it. Right.

It's an implicit experiment.

Right. I was just going to say isn't an experiment.

Well, it's an assumption that you're making. Right. About your product. That's not. It's not tested.

That's tested.

It's almost not an experiment because you're not.

It's more of like an assumption that is actually. That is potentially creating risk. Right. Because you're not. You're not testing that assumption. You're not saying to your customers, you know, or looking at data that would indicate that you do need to evolve the product or change it in certain ways.

You're right. It's almost unfair to call it an experiment. It's only an experiment if you're observing it from the outside and looking at what people are doing and saying, let's see what happens there. Because the market is. Will dictate whether it's a successful approach to what they're doing.

You're making an assumption about what your customers want, which you do when you've got a new product.

Right.

But that's, that's an explicit assumption that you're making. This is sort of implicit, right. You know, it's almost like you could forget you, you wouldn't be aware that you're making that assumption, which perhaps even makes it more dangerous.

Right.

This is the kind of stuff I.

Was hoping the reality of it is you're dealing with then 10 years of senior executive, right. So if you're sitting on cash cow, your job really comes down to holding on to your job.

I promise you 20% growth on GPS.

For a couple of, for a couple of years. So that's like Christian says, capitalist dilemma. Then you focus on efficiency because you're encouraged to demonstrate numbers but through paying.

Cost as opposed to.

Since they treated.

The same in the market.

Right.

You engage to not innovate.

Right. So why is it so hard to disrupt yourself? Is it like Dan, Eric, like, is it a predictably irrational thing? Is it justified because you just, you're just waiting, you're just outweighing it. Or is it an Iraq. Is it rational or irrational to not disrupt yourself if you know, is it literally a matter of time and it's like. And you are not accountable, so you're just trying to wait it out or is it actually irrational?

Well, I think if we come down to two, like one, I don't just resurface discussion, but one, that changes the gu to the top and two, the top wants to sit there for a couple years and get out, you know, with the. Hopefully nothing changes making that assumption.

Right.

But I'm going to assume that nothing's going to change while I'm here.

Do they not change because they don't.

Think, they don't think they're going to be disrupted or do they literally believe.

I mean you're playing numbers, right? What are the chances you're going to be disrupted in the order of 3 or 10 yard?

Right. That's.

I guess that's.

And of course the, the COPP was like, oh, why me?

Right?

Because all he had to do is all, all that mess of the golf had to happen.

So that's.

So is it just a game of musical chairs Are they saying I know when they're just pushing off?

I think that's right. That's what I'm getting at.

Right.

Or and you don't want to be the last one stuck and they're saying there's a 9 out of 10 chance that I'm not going to be stuck holding the mortgage back securities or are they literally saying, no, this is this, this is.

What I'm saying is those two things are true. Then that's like, that's just locked in.

Right.

So change can only come to the top and the top won't change until someone else.

You can only do that if you're not getting any feedback from your customers, if you're not engaging your users and your consumers.

Right.

You could only kind of live in that little bubble of things that, you.

Know, I can just ride it out.

And not make that assumption. I mean, in both cases, you know, building your own product in your own bubble without talking to your customers and doing that, they're kind of the same thing really.

So that isn't just three years of declining revenue is the only thing that will make people change.

No, I find it hard to believe that every, any company that was disrupted didn't see it writing on the wall. Like I just, like I don't think there's a situation where they didn't already know what was happening and where they were losing market share. The question I come back to is like, why isn't something done about it? Constantly defer back to this idea, well, I just need to not get fired for the next three years or something. And maybe that's an oversimplification.

But so going back to the sporting analogy, what would happen if you played sport without an audience?

Very interesting.

What would change is, I think that's what an organization is. It's so removed from its customer. I've worked a lot with exec teams on trying to get them to mindset change. And the only thing that ever works is when you put the customer in the room and it's them telling them because they can say no to a trainer and they do very vocally, but they can't say no to a customer. And sports person can't say, can't yell at a fan. They can yell at everyone else, they can blame anyone, but they won't yell at a fan. And I think organizations exist like a game of football played without an audience and because it's not that two way feedback, because it's like everyone trying to impress everyone. I'm trying to impress you. With my products, it's not the purpose of an organization. Purpose of organization is solve a problem that a customer has is to bring them in so early that they're bought into not only your end product, the whole journey, the process, story, the narrative. But I think it's always like a surprise. Nah, we didn't work. Surprise again. So I think it's, how can you. How can there be a dialogue between customers and leadership and customers at every employee? I used to work in management consulting and we used to build customer strategy. Do you know how many times we talked to a customer?

Never.

Never.

How cool would it be? Yeah. Like even with Kristen, Imagine if like every product owner, manager, whatever, had a relationship with the school where they had to teach a course using that product for like one week in the day or something. How could like. Or the stories about Kayak where the calls go directly to the engineers. Yeah. Just get so tired of calls. It fix the problem right away.

Right.

Or old school.

That's a great. Yeah, that's the kind of stuff I want it like meetup. I know meetup. And the example I always use is meet up. You know, there's supposedly a shift in their best metric, like the day they start, you know, doing regular talking to customers regularly. That's a cool one. I've never heard. So who's got more stories like that?

Well, old school. Have you guys heard of old school?

Do you know what this is?

Yeah.

Oh my God, I love those guys. I mean. All right, so they got like $144 million of funding for like Facebook and few other VC funds. Right. And their whole model is basically progressive education model. Kids come in and they drive their own. What's so scary, I think for Pearson, and I think it's awesome they have a 50 person engineering and development team embedded in the school site. Like the engineers are sitting in the back of the classroom listening to the teacher lecture or whatever. And like the teacher always like go over to the engineers, like, this is not working right now. Within minutes. Wow.

Right?

It's just like it broke down all barriers between engineering and a customer as it's happening in the real time.

Right.

Like you're actually adjusting in the process. That's why I think they're doing one of the coolest platforms that are just ahead, leaps and bounds ahead of their own.

So what else is.

So like Amazon's known for like running millions of experiments every like millisecond. So everyone thinks that they're just good at commerce and logistics. But what other companies are really really good at running experiments and going. I think that was the key too. It's not just that they're listening to customers, but it's not just the learning experiments, but they're also doing it lightning fast. It's those three things together. Customer speed. You were saying speed before, right? Customer speed. Experimentation and data, not guts. We're at the fun part of the night. We got through our threshold of having some really smart stuff. Who's got some more really good examples of companies that are doing.

I think that's an amazing example of bringing engineers. Right. I mean, that's kind of like collapsing the whole.

That's why I'm searching.

Developers in your classroom. You're like, hey, guys, fix this right now. Within minutes, it's fixed.

What I also love about it is.

Those developers are also pushing the code into production.

That's why.

That's what I'm saying.

And passing it to a QA person who's got to do that. There's no.

Right.

There's no time for that.

Right.

Like.

But at the same time, nothing's getting lost in translation. Can you imagine? But can you imagine the stuff that goes through technical support at large companies? And it's not the same issue that the customer was complaining about by the time it gets to the person.

What's the phone? Games? Phone. Game Telephone. Yep.

That's my favorite. Everyone's seen Office Space where he's yelling. He's like, what do you do? Well, I take the specs from the cuz. I'll. I'll play this. Maybe everyone seen the movie.

No.

No, I haven't seen Office Space. Can you do this scene?

Everyone keeps telling this scene.

All right, I'll.

I'll send this. I'll post the scene on the meetup page.

It's really good. Reading the video.

Yeah. I think that one of the cool. I seriously wish, like, we would have $50 million in Pearson. You just open up a school that. That's what we do. Yeah. Like, barcode embedded with rng. I saw. I got so depressed and excited at the same time. I didn't know what to do.

Give me an example.

Put you on the spot, but do.

You have any examples of that?

I'm trying to think of the one that I can back up with some.

Zappos probably does stuff like that, but I can't think of anything off the.

Top of my head.

I know a lot of large organizations confuse customer service and customer centricity. They're like, we treat our customers well, we listen, we do exactly what they want. And that's the total opposite is you're designing for an individual. You're not designing. Designing for a market. And I think there's. I could see organizations misinterpret that and over service instead of change their philosophy to design.

They're definitely two different things. I get scared sometimes when I say over customer service isn't a good thing. They're definitely two different things that do two different jobs.

Yes.

I don't know if you can ever be too good at customer service. But some people might say maybe.

It.

Should not scale if it comes at the expense of other clients. Other customers.

But I think for the old school example. Right. Obviously they're not going to be able to do that with everybody. Should they grow and obviously. But they're very adamant. They're like, we're in the experiment phases of this idea of progressive education platform that if we can get it right here by. No one can beat the rate of their experiments. I can't think of any other model that would actually experiment faster and more accurately than that. But let's say they hit on something. Now you have something that's tried and true. Right. Then you go to. You start weaning off experts. I suppose.

Richard did an interesting thing. Like should experimentation be a phase? Everyone talks about the experimentation phase.

What you experiment on should be phase specific. Experimentation should not be a phase.

Yeah. The behavior should continue throughout. But it's what you're experimenting on.

Yeah. And it's also just that cliche of like, you know, exploration versus optimization. So the stakes should go down. The types of things you're experimenting on. You should have more knowns now than unknowns that you're experimenting on boulders. Not.

What about.

What if you replace the word experimentation with learning.

Learning.

Is it like should they. Should the learning. Should there. Should it be a decline in learning as a product evolves or should it just.

It should probably flatten. Why then you should. Well, because there's some. There's actually. This isn't mine. At some point, you know, when you start learning. What was this one? Attendees talks or someone. It's like you have a. There's a finite thing that you don't know. And as you learn you reach that. That it doesn't. It's not like a lockstep thing where it automatically extends.

Like you're gonna.

Unknown starts to creep up because things change.

Obviously.

Photos Unknown.

Not necessarily. I think you have to. I think you have to design for like a new local maximum. I think you can.

You can.

You can start saturating your local learnings.

Well, I think it's. There's a distinction between. There's a distinction between the amount of learning that you need to do for a particular product that's in the market. And at some point you're going to reach like the. A point where it's not worth investing in additional learnings because you're just like, the returns are going to be diminished. But it doesn't mean that you stop learning about the market and the opportunity because you just shift the focus.

Yeah, but you can use that product too. Then you can use that product in the market as a way to get feedback.

Right.

Not even just general looking at the market specifically. So in a sense, you are experimenting.

By having it there.

You have one more thing.

No, I didn't.

All right, I have one cool thing on your exports thing, and then we'll maybe get into actual experiments that we run together. We can pretty much call it a night after that. So I'm assuming most people at this time, Taylor Freakonomics fans.

Yes.

So there's a really interesting study, and correct me if I'm wrong or if you've heard this before when you said what if fans played in those sports. So supposedly someone did a study where soccer or football. You feel like this. You should know football. Football players for penalty shots. If you aim dead center.

You have.

A higher likelihood of scoring. So if. And this is why people, like, don't. Why don't people just behave on data? So what's the reason? Why wouldn't. Why wouldn't penalty kickers always kick it directly at the center if the odds of scoring are higher?

Impression management. If they kick. If they kick the center and it's.

Blocked, highlight anybody but you.

Right.

So that was sort of why I asked the rational question. So it's like it's completely rational to go with something that has lower reward. Because the risk of. Of hitting could not only be ritual. You could be killed like in a mob scene. Like you can if you aim directly at the goalie. And that's when it gets blocked. Even though the odds are in your favor, the fans don't give a. That that the odds are your favorite.

You.

You look like an idiot. And the kick and the kicker is factoring that in, I would say somewhat rationally. And I think some of that actually happens. It's like there's no reason to rely on the data because if you mess up with something that looks really bad, like kicking it right at the goalie, you might be carried out of your business by, like, a hungry.

Well, we'll see the ultimate experiment in, like, politics next year. It's just daily experiments. It's the way candidates, the way Republican candidate will be pitched and experiments. It's like, let's just put a whole heap of crazy people in John Oliver. Let's see, because it's separate. Let's see who shoots themselves.

They're running experiments. Don't apologize for anything.

See how that works out.

Say outrageous things, but never apologize.

That's an experiment. All right, we'll come full circle. Anything you wanted to experiment on. But we're scared to not scared.

I have. I'd probably get fired at some point. Let's just do it. I don't know. There was feedback I once heard from a mentor that I really live by. It's like find and surround yourself with people that understand the differences between outputs and outcomes and are ruthless about the manner. All they care about is actually performance, and they build a team around that model. So recently we're experimenting on a figure prototype with the supposed confidential partnership stuff. But, yeah, I was just like, all right, let's totally do this. So I had, like, the name up everything. Like, how am I actually going to get some traction behind that? Not even thinking about it. Right before I published, someone else came around like, whoa, whoa, whoa, no, no, no. They would be so mad if they thought it just. I wanted to just go out and just get it done, but ended up wasting. Not wasting. That's not word. Ended up having to go through multiple chains of command in order to get the green light, to somehow put something in the public space to test some hypotheses. Yeah, that page, that's up now. The amount of time it took me to get to that point and permissions, I had to get signed off.

So you don't have to call anything out specifically. So to put up an unbalance or a landing page.

To put up a landing page to test hypotheses about our customer segments or what we thought might be. I had to go through this idea of just like, everyone was concerned about the impression of what would have been.

Right. So that's sort of what I was getting. Obviously, you had no hesitations personally, so. But what were people. What were their actual fear? I mean, I think this is real important stuff. Like, are they scared of the brand? Were they scared of lawsuits? Were they scared of people's feelings? Were they scared of it not working? Did they think that the product just physically had to exist? That, like, even if you did all those things and launch it that the data just wouldn't be reliable because the product doesn't exist.

I think from my understanding there was a fear about what their boss would think and that kept going up until. And the assumption was that the boss would be angry that it could deteriorate potential conversations with tough partner.

I think there's a fine line there. I mean knowing some of the specifics of that situation, there are legitimate fears that are known by some about relationship management and they sometimes translate to legitimate restrictions on what you should experiment with.

Oh that would have just been like publish.

Right. And it's important to know the distinction between those two. You can't be completely, you know, blase about it. But it's often taken to the extreme.

Yeah. And to be fair, I don't. I don't suppose that was an extreme situation. But this is the most recent example of where yeah it just.

Ask for forgiveness or for permission. Pick one. Why? And under 15 seconds we know your answer.

Why? Why?

Yeah, it's fun. It's the last. Last thing of the night.

Excuse me.

The advice that like I heard once be concerned about outcomes every. And look in all the decisions I made, they're never malicious. They're never intended to like go over someone. It's like all the decisions are. I believe this is the best way to achieve this intended outcome. And then I just come across like impression management issues and politics and this will be my lifelong struggle. I suppose.

Something you wanted to test or something you've always wanted to know but were afraid to find out or couldn't even think of a way to test it quickly go off the rails a little bit. It can be in life. I heard you giggling. It can be in life or in work or professional.

What does it feel like if I put my finger in this socket? Always wanted to know that.

See.

Does the experiment take you back to personally tried that?

Of course. Who didn't do that as a kid? Yeah, totally. Nice.

That's a whole other topic like the modern. What's the freaking free range movement about raising kids. Like we've moved away from letting kids make those kinds of mistakes and get hurt.

Right, right, right.

Allowing them to do it.

That answered your question before even I. I'm a crazy experiment. But things that you have to do, you weigh the risk and if death is a real risk I would.

That speaks back to the fallacy of the human brain. We have a total inability.

Well then that is a very good question of how do you actually measure what the true risk is versus what the perceived risk is. And. But, yeah, if the real risk is higher than 1%, death.

But the parenting example is. The parenting example is a good model for how enterprises work that parents overestimate the likelihood of death. They overestimate the likelihood of being kidnapped in a playground because they listen to the news or they do whatever, when the reality is very few kids actually get kidnapped from playgrounds and there are very few molesters.

And they also underestimate the value of letting them play on playground.

Yeah.

And so they underestimate the reward and overestimate the risk, which is why every single experiment doesn't give up.

It's relative, though. Risk is a very relative thing. I remember watching a documentary about a free rock climber who, like, faced death, like, wasn't bound by anything. And they asked him, why do you do this? And he goes, the scariest thing for me is, like, working in an office every day. Like it was. His version of risk was.

Minimum success criteria. 30 people.

Bye bye now.

15 from Pearson. 15 out, 10 from Pearson. 20 not from Pearson.