S1 E14 | Understanding Where You Need Big Data vs Small Data

Most cases, when people want size or sizing of a market or I need a number of samples to be, how confident am I that this is working to do that? My thing is we do need that near the end and as we launch. But when we're very early in the development process, it's very hard to get big data. And so what happens is trying to actually ask people who are really comfortable with big data to move upstream is kind of like I'm asking somebody who's used to playing in a symphony to be a soloist. Right. And so it's this notion of like, how do you actually play with small data to get to an understanding that then lets you look at the big data in a very different way.

Welcome to the Circuit Breaker podcast, where we challenge the status quo of innovation.

And new product development.

We'll talk about tools and skills and methodologies used to build better products and make you a better consumer.

Consumer.

I'm Bob Mesta and I'm the co founder of the Rewired Group and I'm one of your co hosts and we're joined by Greg Engel, who is my co founder and the chief, Bob interpreter. Join us now as we trip the circuit and give you time to reset, reorganize, and recharge your brain to build better products.

Hey, just a little preview to what we're going to talk about on this podcast with Greg is one of my topics. Near and dear to my heart. It's big data. I love big data. I love any data, to be honest. But we start to put it into perspective that big data is not the answer, it's part of the answer, and it's a tool like anything else, and that small data and big data are parallel and there's way more people who know how to use big data, very few people who know how to use small data. And that we talk about kind of the causation and how we connect small data to jobs and then talk about some of the tools that we're, we're working with and things we've done in the past around big data to help us understand how to create better products. Right. And ultimately, you know, the tools that we're developing to do that. Enjoy.

Hey, Bob.

Hey, man. What's up?

Today we're going to talk about something that I think you're going to lead the conversation a lot because I think I'm going to follow, but that's okay.

Near and dear to my heart.

I think the first question of this is how do you think big data has affected the way people look at.

Innovation, what's interesting to me is how big data has actually caused people to be more reliant on correlation than causation.

And what do you mean by that?

And so what happens is that they're trying to find trends in the big data where they have and can see kind of the underlying thing or the one thing that causes something to happen. They're looking for the one trigger, the common element across all these different things. And so what you're finding is that as people get big data, they feel like as long as I have the common thread in the big data, that's the thing I gotta do more of. So, and so it's.

If I said big data is showing us the effect but not the cause.

Correct. Would you. Would you agree with that statement? That's correct. And so in a lot of cases. But they assume that if they know the effect, that they understand the cause, or that the way they've asked the question or the data that they've collected literally infers what the cause is, but it's really not the cause. And because to me, things don't happen. There isn't one cause and one effect in anything. It's sets of causes that cause an effect. And so part of it is to understand what are those sets of things and the sequence of those things that make it happen. And so a lot of times we have data that is just aggregated and then analyzed, as opposed to analyzed how it works and then aggregated. And so to me, there's a big difference between that and so a lot of cases, most people are relying like, well, if a million people used it, then it must be. This might be the answer. And the reality is that what I find is most real, useful information comes from being very narrow and very deep and connecting the causation and then being able to go to the big data and see how it works. But I don't believe that I've been. At least I know I haven't been able to extract causation from big data. I have to do it almost on an n of 1 basis. So clay and I used to talk about this notion of n of 1, where let's do one interview and understand what caused somebody to do this, and then we can connect those dots and then let's do another person and another person, and then once we see that causation, now, can we go to the big data and actually look at that.

For those connections? And so a lot of times people are trying to make connections where they aren't because of correlation. At the same time, the fact is they're not actually measuring sometimes the right things or they're not actually segregating the data into.

Time delay and how the cause happens and the effect happens later. And so they'll look at everything on Tuesday and say, what happens on Tuesday? But I might have the cause on Tuesday, but the effect doesn't happen till Wednesday. So I don't really see it. Right. And so the way they look at data is very, very confounded that way. And they're literally trying to compress everything.

And it seems like, and you can correct me if I'm kind of wrong, but it seems like they're trying to push big data further and further up the pipeline of innovation.

Yes. And that you need more and more data to make decisions. And the fact this is what I would say is in some cases, what you need is better data, not more data or.

Yeah. Or just the size of data sometimes matters and sometimes doesn't. Well, depends on what you're trying to accomplish.

Most cases, when people want size or sizing of a market or I need a number of samples to be. How confident am I that this is working to do that? And my thing is we do need that near the end and as we launch. But when we're very early in the development process, it's very hard to get big data. Like, and so what happens is trying to actually ask people who are really comfortable with big data to move upstream is kind of like, you know, I'm asking somebody who's used to playing in a symphony to be a soloist. Right. And so it's this notion of, like, how do you actually play with small data to get to an understanding that then lets you look at the big data in a very different way.

So I want to make sure we're saying. And we're not saying big data is bad.

No, no, no, no, no. But it's very hard to find what I would call are the insights you need to be able to figure out kind of what is the causation. And in a lot of cases, the anomalies of big data tell me more than the big data or the average of anything. Like, I'm a very big proponent of the book End of Average by Todd Rose. And he just, he, he talks about how this, this very useful tool of average has helped us through the years, but it's also helped distort the way we see the world in so many ways that, that it's, it's actually causing problems.

So it's, it's one of those things where it's. It's still a matter of knowing what tool you need and when to use it and how to use it.

That's right.

A hammer is a great tool.

But, but, but a hammer doesn't work in every, every situation.

But if I'm trying to screw a screw in a hammer is probably not the right thing.

The other part to me though is that big data, an excuse like I don't have enough data to know. Right. And my thing is that they don't. People don't know how to be scrappy enough to actually generate the data they need to figure things out. Right. And so one of the things I learned very early in my career was design of experiments from Dr. Taguchi and R.A. fisher and people like that. And the whole aspect of learning that was helping me be able to do very few things and learn a lot from it. And so this is where, you know, the Taguchi would always say the most important thing for an engineer is to actually figure out how to generate the least amount of information that helps them make the most confident discoveries and direction and that we don't have unlimited dollars and time and money and everything to do something. And so how do I actually use it to generate very efficient technical information to make better decisions? And that the notion is that there's way more unknown than known and that big data almost assumes like, we have everything, it's all in here, what do we got to do? And we don't actually know what we're missing. I think the other part is that the data comes in at different levels. Like they can talk about your heart rate, they can talk about your blood pressure, but then at some point there's a whole other level of what's your enzymes and what's your testosterone levels and all these other things that, that have an impact on all those things. And they almost try to isolate them as individual things as opposed to looking to them as sets of things. Right. I believe that most everything is set theory. But the fact is I don't need a big set to understand how things work. I actually need to know how to.

And what is set theory in your.

Mind set is like, again, there's. People confuse root causes, root cause with root causes. They'll say, one root cause. The reason why I bought that car is it had great gas mileage. And that has, in most cases, it has virtually nothing to do with why you bought the car. It has to do with the fact your car was 170,000 miles on it had two major repairs in the last month. And so people try to narrow it down to one thing that caused you to do this, and then they all like, we got to get gas mileage. And the reality is that's not the case. And so this is where we play with small data to then help us scale to large data. But we don't use large data to help us get to the insights. The insights come from individual understanding individuals and the causal sets of what happened to then help us then see how we see it in the big data.

So when you're coaching teams, or when we're coaching teams, when do we usually reach for big data?

So in a lot of cases, we'll end up using it where we have job, we'll find the jobs. And we can see, like, when people did this and this and that, it's like, okay, that's when they're in this job, but when they're in this other job, they actually do something different. And so, for example, at basecamp, one of the things was we had to think of one of the jobs was help me think it through, which is a project that's in the early stages. And as they started to basically think about the project and what were the tasks and who was going to be involved in everything, what they would do is they would actually build a project. They'd create a bunch of tasks and never assign them. They then invite a bunch of people to comment on it. There'd be a bunch of back and forth on the comments, and then it would basically move into kind of an execution part. But they primarily created it to help them think it through versus somebody else in a different job would basically, they would create all the tasks. They might invite people, but those people only uploaded things. And so all of a sudden, we could see once we understood kind of the underlying causal mechanisms from the. From the jobs, and this was like 10, 12, 13 interviews. But then we could actually go into the data and see the behavior and extract like, oh, these people are in this job, and those people are in that job. And so we use big data in some cases to help us confirm or at least look like how to seek into the big data to actually find these patterns. But my belief is it's very, very hard to find these patterns without doing that qualitative causal things in between. And again, I go back to that. N of 1 and Clay would always say that new theories always come from anomalies. And so if we treat every thing as an anomaly and then we understand how they actually cluster together and how they aggregate up. It's actually easier than trying to segment things down.

And that leads me to kind of a segue, because I think what we try to tell people is know what you're using and why you're using it. So big data is no different. That's right. Right. There's times that you can use Jobs redone. That's right. And times that you can be using Jazzlyn. That might be wrong. Right. But know what you're using and why you're using it. So, for example, with big data, you know, some people would say, well, hearing what you heard, well, I'll never use it in upfront. In the upfront work. Which is not necessarily right, because it depends on what I'm looking for.

Yes.

And for an example, when you were doing the milkshake stuff decades ago now.

You actually looked at data to find the anomaly.

That's right. That's right.

And finding the anomaly in the morning, of using the milkshake machine, which wasn't actually allowed to do, but you found people were doing it, and you're like, why? It wasn't to size the market at that point. It wasn't to do. It was to say, hey, how do we go find out why they're doing this?

And most people. The interesting part to me is most people look at anomalies, and there's a book Thomas Kuhn wrote, kind of this Structure of Scientific Revolutions, and he talks about the fact of how people see data and they see the anomaly, and scientists will literally say, like, oh, that data point's not good because of this and this and this, and they throw the data out and that. What you really learn is when you talk to people who are innovators, they actually look at anomalies as the source of new knowledge. They don't know.

Yeah. And I always have to.

Excuse me.

I always have to know what I'm looking for. So if we're an upfront ideation, we're looking for a place to go investigate. I might use big data to find those things.

Yes.

If I'm in the middle, I might. And I've done maybe Jobsy Don, or I've done segmentation, and I want to figure out how people behave in that. I might use big data then. And at the very end, I might use big data to help size the market.

Sizing is about a prioritization. Where should I go? What should I do?

Well, that's what I want to get to, because I think a lot of times we think of. We. We think of sizing the market differently than other people do.

Yes.

And this obsession with sizing the market is, is somewhat funny to me because we size it in a vacuum and we don't size it in context. Yeah, we, in context is actually the important thing of sizing. Not just, hey, there's a billion people in the world, 75% of them have to eat food. Therefore this business is worth $15 billion.

That's right. I've, I've worked with people who go like, I need to have gen pop people see this concept, tell me what they're going to do with it. And I need people who give it a 7 or 8. I need 40% to be a 7 or 8 for us to launch. I'm like, this is. And to be honest, it always feels like it's an insurance policy done by the church of finance that's put through kind of marketing to kind of say like, is this a good bet or not? And what I would say is that if you look at the history of tools like that, it's actually almost always worse than it, than, than you think. Meaning sometimes you'll launch and it says it's going to be a $20 million business and it's $100 million business and you're still screwed because you don't have any of the infrastructure to do it. Or the other time is you have to get a hundred million dollar business and it's a $10 million business.

And that's where the over reliance on the big data tell us an answer that isn't really possible to give us.

That's right.

Is hard.

I think part of it is, is that they, the excuse we constantly get is like, well, we know it's not accurate, but it's better than anything else we have, so we're going to use it. And so part of this is to realize the amount of money they spend on it, that if they actually understood how to get to the underlying causation and they could actually see the patterns for the big data, they can actually be way more efficient and effective in doing it. Because again, it's not who, it's who, when, where and why. And if we can actually see that part of it, that's what it is. And so I always try to like the first thing people say, like, oh, we've got a lot of data. I'm like, okay, I want to see the anomalies. And like why do you want to see that? I'm like, because anomalies is where it all starts. Right? Like people who are who that like so, like, when we did Basecamp, one of the things we found the anomalies of, like, who are the people who shouldn't be using Basecamp? Right. It's like, well, we have a doctor's practice using it and we have, you know, architects using it and we have a not for profit using it. It's like, okay, they would say we know what agencies use it and why software companies use it, but we have no idea why these other people are using it. And that's how being able to make it grow was coming from the anomalies, not from being better at the core. Right. So I think that's the other thing is that most people think of if I make it better for the best customers, it's going to make it better. And the reality is, a lot of times it actually makes it worse because you actually alienate the people who aren't the best customers because they can't do it.

So again, I want to just point out again, we're not saying Big Data is bad.

No.

We're saying, know what you're using it for.

Yes.

Understand its limitations.

Yes.

Know how to supplement it.

Yes.

And don't over rely on anything. And I would say the same thing about anything we do. You can't over rely on anything.

No.

You need to balance of a lot of different things to help us find the way.

So in my, the thing I always do that drives you crazy is I have my hands and I go back and forth and I almost like have it like I'm rubbing a globe. And it's this notion. You need different perspectives. And Big Data is one of many, many perspectives you have to have on any kind of problem or thing you're looking at. But the reality is we overemphasize because it's a large sample and it's statistically significant. And it's this and this and this that we can actually say that this actually has an impact. And my experience is when you're trying to innovate.

There'S a lot more that goes into it than just having Big Data tell you you have the answer.

And this may or may not make it in the podcast and we'll decide this.

Go ahead.

Are you working on anything to help with Big Data or to help with sizing of markets using some of the stuff we do?

Yeah. So one of the things that I mean, you know, like my happy place is when I have like a, you know, a 20 gigabyte, you know, data file of like, you know, just everything about something and I'm like, okay, I'm going to go, I'll be, I'll be back in a day. And I'm like playing, I usually call it playing with the data. And it's like looking at it from all these different perspectives and just doing it. And to be honest, when I've been able to do that with people, it's actually very fun. But the thing is, what I'm realizing is that what most people don't know is they know how to segment things, which is how do you tear things apart or how do you pull things into its. Reduce it. The reductionism, reduce it into parts. But the reality is that most people don't understand clustering and how you take things and put it together. And so the aspect here of learning how to cluster and taking a set of data and using cluster to it as opposed to segmenting on it, they think it's this. Most people think it's the same thing. But you actually have what we call emergent properties when you put two things together that don't necessarily always go together, but when you put them together, you actually have a new property that didn't exist before. Right. And so what we're using is, I'm working on some data analysis from the qualitative of jobs to then use the variables we find in terms of the causation that's there to then help them understand. One, how to size, but also how to actually understand what are the underlying things we need to do in this market or in that market.

And we're prototyping that on a very small scale with people. Yes, we don't know if we don't know if other people want it or not. No, no. So we're, we're kind of just doing it internally to figure out what we can do with it, maybe working with some people doing it. But that is one of the things we are working on is how do we help size? How do we help put people in context? Because that's actually the big thing is how do you actually put people that are taking these surveys into context?

This is, you know, you and I did this when we built houses, like we would take the mls, we'd actually look at what houses sold in our area that were on the used market. Then what we did is we actually tried to understand we could actually take based on how long it was on the market, what was going on around it, who bought it. The underlying premise when we went to the home building market was anybody who built a new home never looked at used within the first month. We Dispelled all that. Then we actually said, how do we actually compete more with we want to be better than a used home and actually have almost be in the same price point. So we actually used all that data to tell us what to build and where to build in a very, very systematic way. And we were able to follow the market as it went up and down of what we needed to adjust.

Yeah. The builders would often say, why would anybody buy a used toothbrush.

That's right.

When they can have a new one.

That's right. And you're like, okay. And the interesting part is the one thing we did find over time that was actually a big difference was the plate height. Right. The height of the seam ceiling and that all the used homes in the area were eight foot plate. And we made everything ten foot plate. And so when we went to compete and we were roughly in the same price point as a used house, but we had the 10 foot plate, the rooms look so much bigger and it cost us virtually nothing to do it. And so it was just one of those things where we could see what that was. But we also understood by interviewing people say, why did you buy this and not used? There were some underlying things that we ended up adding to what we looked at. It was very, very useful. I mean, we went from what, 150 homes to 300 home. We had a thousand homes in a little over three years. I mean we did great.

Yeah. So again, I think we're prototyping that kind of stuff. It's not anything out in the market now. We don't know what kind of if people want it or not yet. We're just kind of playing with what can we do, what can we do with jobs? And more quantitative, if you will.

Yep. I think the other thing is that I'm wrestling in this space of traits and how people describe traits and like you've got this trait or I've got that trait, but how does it actually connect to jobs? And so my whole thing is, is that there's a lot of people who could tell you what your Myers Briggs is. And if you have this Myers Briggs, you're more tend to do this versus that. And what I'm finding is that context actually has more impact and what context you're in is going to have more impact in terms of how you make that decision of what to do than it is about your Myers Briggs. And so this is where I'm wrestling with what I would call trait theory and being able to understand what are effects, what, what are causes. And ultimately, how do people make decisions? And though I might be this Myers Briggs INTJ or whatever it is, the fact is that doesn't make it like I'm not going to be able to do something else.

Right. And we actually just had this happen to us. Right. Is in a disc. In the disc, I'm going to ask, which is getting along, steady relationship, that kind of stuff. But in the context of trying to do these podcasts, I can't be that way with you because then we're going 15 billion different directions at a time, and I have to be way more direct. So the context of what we're trying to do dictates what I'm going to do, the job I want done, which is have a podcast that is understandable, have a podcast that is resonates with people, that isn't repeating all those different things takes over. And I have to be more directive with you than I would normally be. So the context changes the way I behave in those situations.

That's right. That's right. And so that's where big data would say, no, Greg is an introvert in a steadiness where he wants to make sure everybody, everybody, but everybody gets on. But in some cases, you know, you need to actually, in certain contexts, you actually need to step out of that s view and actually be more D. So it changes.

And that's what we're trying to figure out with the quantitative stuff is how do we get people in that right mindset? Because that's what we want to know. We don't want to know what they would do if everything, if the moon and the sun and the stars all aligned, we don't care. We want to know when the moon's here and the stars are here. What do you do?

That's exactly right. And it's. And it's. To this point of. It's the. And most people have a lot of data on who, or they have a lot of data on when, or they might have a lot of data on where, but they don't put them together. And it's the, it's the kind of who, when, where, and why that actually helps us see the right patterns of what's there. The other, I say one of the other things that is really there is most people assume symmetry, meaning if, like, this is a push, this is a pole, and it's like, not. It's. Nothing is as symmetrical as we think. And that we have to actually realize that there are a lot of assumptions when we assume symmetry that might have gone a Little too far. No, I think it's fine.

I think it's fine. And we get to make the decision whether we publish or not.

It's always cool. I know, but at the same time, it's like, like you, this was a cold podcast. You just like, say, hey, we're going to do this. I'm like, oh, yeah, I like that. So we'll see. We'll see how it turns out. But I feel like this is. This is one of those things where you need big data, but you need little data. And I feel like most people don't know how to deal with little data. And little data is. Is as important, if sometimes not more important than big data.

And I think the real moral to this podcast is know what tool you're pulling out and know why you're doing it and get the right metrics of how this is going to be measured. And that's really the moral of this, of this podcast. So that leads us to our homework. And again, this is a difficult one for homework, so we're going to wing it like we do always. And really, all we want you to do is think about as you're pulling out any tool in the next week, I need you. You need to define it as what it is. What does it tell you? Why are you using it? And that's the big thing, is why are you using it? And how do you measure its success? Because that tells you whether you're using the right thing or the wrong thing in the right situations. So that's really the homework. As always, we hope you enjoyed and you tune in next time.

Thanks.

Thanks for listening to the Circuit Breaker podcast.

If you haven't already, please subscribe so you won't miss an episode. If you know somebody who's stuck on the innovation treadmill, please share it. If you'd like to learn more information, Visit us@therewiredgroup.com to find out how we work, how we can help, some resources, some books, some software. Join us next time as we trip the circuit breaker to help you recharge, re energize and refocus your new product development.