CAM 1 TMRE 10 17 18bc

I know many of you are members of. I recognize in the audience the consortium is a bunch of people from different industries, so consumer goods, technology, advertising and market research with the goal of evolving, refining and ultimately promoting insights on demand. This thing that we've been talking about a lot for the last couple of days, this thing that's becoming more and more critical in our lives. So a bunch of companies and individuals getting together to try to work out what that is and how we do it better, more often and for more companies. Please reach out to any of us if you want to find out more about that consortium as well. The panel's going to be quite a lot of a conversation, so there's a bunch of questions I've got prepared for us. Based on the calls we've done and the kind of conversations we've had in this room the last couple of days, we'd love to questions from the floor as well, although we might not get through more than about four of them. So I think it's going to be quite a rich conversation. So please load up. Slido, please start to enter and vote on questions. We will get to the biggest ones, I was going to say as well on Slido. All the questions we don't get to are incredibly valuable. The same for all the sessions, so feel free to add them even if we don't get to them. They'll help us to program the future conferences. They'll help us to know what to do in the consortium. That'd be really valuable. So don't be upset if we don't get to it. It's still super valuable, but we'll certainly get to some of them. We're going to try and make the panel rich with both specific examples of this speedy insights that we're talking about. We're going to talk about specific challenges that get in the way of doing that, and we're going to try and talk about failures as well. Times when we've tried this, maybe it's not worked out the way we planned. So please look out for all of those. The context, of course, then, as we've talked about for the last few days, is this rapidly changing landscape, this complexity in the world around us and the need, with limited or maybe declining budgets, to research so many more things and get answers so much more quickly than we used to. How do we do that? How do we supply that need? How do we satisfy that need in companies? It's exactly what this panel's all about. The answer being, of course, move very quickly, try new things and Fail and fail very fast. So let me introduce our panelists then, in order of the photo on the side there. So if you forget names during the session, you know where to look. On our left, we've got Phil, head of Product and Strategy at Taluna. Next, we've got Michael, director of Consumer Fundamentals and Insights at Procter and Gamble. Hopefully, you know me, David Boyle from Harrods. You've seen enough of me this week, I think. Next, AJ, VP of Ad Sales Research at cnn. And last, but by no means least, Laurie, head of Consumer research and Market Intelligence at Converse, part of the Naik family. So welcome, everybody.

Thank you.

Thank you.

So, first of all, fast fail. Failing fast.

What does that mean to us from my standpoint? I know a lot of people were asking me about what this panel was about, and I said that, well, I don't know if I really even agree with the title of the panel. And the reason I say that is just because I think when we're talking about fast fail approaches, I don't think that the idea is that we really just want to fail. I think if I really wanted to fail, I would have drank a lot before coming up to the stage here. But, you know, I think that the idea is that we really want to learn quickly. And I think, you know, giving yourself permission to fail is, of course, an important part of that, and I think that that is an important part of learning. But, you know, again, like, you know, it's not that you're, like, you know, designing the approaches so that they intentionally fail. You're designing the approaches, hopefully, so that you're intentionally learning and learning very quickly.

Yeah, I agree. I mean, we talked about this a little bit earlier. The notion isn't necessarily to fail. It's to realize something's going wrong very quickly, being agile enough to move fast, react and then make changes and go back into market. So for us, we don't call it really fast fail. We call it, like fast success.

Do we notice that we have more failures along the way, though, by going at such pace?

Well, again, I think from our standpoint, what I always tell my team is that if we're succeeding 100% of the time, we're really not trying hard enough. So I think that the real idea here is that you are really stretching what's possible. You are really trying things that you're really not sure are going to work, and you're intentionally trying to, I think, be on the leading edge of something. And in doing that, I think if you're succeeding all the time, I would say that you're probably not on the leading edge of something. You're probably not really stretching yourself enough. And so I think a little bit of what we're talking about here is just making sure that you are doing those kind of things that are making yourself a little bit uncomfortable, that are things you don't necessarily know how to do, and that you're using those to learn faster, hopefully, than your competition.

And do we think that this need to do research ever faster and to turn research around more quickly and to research more things? Is this a fad that we're going through, or is this a real trend that's going to continue?

I'll take that one. I think this is here to stay. And it's really being driven in my organization by the pace at which marketing is moving. And when I have a marketing team that can field a new campaign on Instagram, and then they'll go on their phone the same day, and within the next hour they'll get 25 likes and they'll declare it a victory. I know that I need to step up and be faster with providing the real facts and the real objectivity on whether or not what we're doing is working, because if we don't do it, then they'll conclude, perhaps inaccurately, that what they're doing is working.

Yeah, it's not like access to data or consumers are going to go slower. They're going to continually go faster. And a lot of time, especially with social media, most consumers will know if they're going to like something or the sentiment's high or not, even before research. So we're going to have to continuously move faster.

I mean, I think it's not a. I think it's a fact, not a fad. And the fact is essentially that, you know, like, even things that we used to do, like survey research, you know, might have taken six weeks before. You can typically do survey research in an hour or a day now. And so, you know, it is just sort of a fact of the matter that there's an expectation because of technology that we do everything much quicker. I think it is changing the dynamic of every industry we're in, and certainly in the ability, I think, you know, for us to, you know, innovate quicker, get to market quicker, and hopefully do things that are going to be more meaningful to consumers in a much better, faster way.

I mean, that's kind of where fast fail came from anyway. I mean, most companies either didn't have time or didn't have budget to do the research so they'll put it into market and hopefully they'll react quickly and adjust. So I don't see that changing at all.

There's a question from the audience then. Ten years ago, I didn't feel I had organizations permission to fail to converse. Nike and CNN give you that permission now.

I mean, if you're not expecting failure, you're not really learning. So I mean, you know, from a product development standpoint, especially at Toluna, I mean, my executive team probably doesn't want to hear this, but if we're hitting, you know, 30% of the time, that's pretty successful. So.

But I think it's a reflection of what research actually is. You know, we can't always expect a positive outcome. And the biggest foundation of research is learning is that whole process of getting to that end, whether it's a positive or a negative result. And I think that's something that as an industry, everyone has to understand and appreciate.

I think it's also a little bit of a matter of when and how you're failing. And what I mean by that is that clearly our intention, at least at P and G, and I'm assuming this is true everywhere else, is not. Not to fail in market. We don't really want to launch something and spend a lot of money advertising it and having it be a huge failure in market. When we talk about failure, what we would like to see, I guess, is to try something in a very small way, in a very small kind of test, if that fails, be able to learn from what didn't work about that pivot and make it into something that would be a bigger, better idea. So I think a little bit of it from a. Permission to fail, I think it's a little bit of like, well, yes, I think it's permission to fail, but permission to fail fast and fail small versus failing after you've invested a lot of money and gone to market. Essentially, I think that's really what the idea is all about.

I think converse answer the question about converse is they're very open to the idea that not everything is going to work. I mean, we create a couple thousand products every six months and obviously not all of them land. As far as my role in all of this, it's more about de risking when there are big bets that we're thinking about making and trying to mitigate that risk of a big failure. So I think. I guess I agree with Michael on that. One is fail in test as opposed to failing in market.

I think one of the other Things is that our R and D organization actually did a big analysis on this and figured that when they took all the technology innovations they developed and they looked at the ones where we basically just went a normal path and went to market, our success rate actually wasn't that great. In places where we did a lot of early experimentation and pivoted a couple of times, changed the technology, made sure it was really consumer relevant. In every one of those cases we had 100% success. I do think that it speaks to the need to do this type of thing because I think otherwise you sort of start with what you consider to be like a great idea. But I think if you're not willing to really pivot, learn and really take that feedback, I think the chance you're going to be successful as an organization is very low.

But going through those processes of testing and going back to the program and going back to testing is time. And a lot of organizations don't have the luxury of that time or don't have the luxury of that budget. So a fast fail approach is, you know, all they have to do. So how to react once you know something is not going the right way and then going back and being agile is going to be critical to their business.

I think as well though, it's a little bit, you know, like I know Beth had mentioned this as well, but this whole idea of like creativity and the need for a little creativity as you're doing it as well, I think if you're just basically like always doing the same test or kind of the same thing, I don't know that you're really learning a lot. And again, I think that, you know, when we really kind of understand what problem are we trying to solve and go try to learn about that in a novel way, we very often come up with a lot better answers. And I think one of the things with kind of the, like I was saying, like the new expectations around time, the new expectations around money and how much you should spend on research and some of those types of things. I think that, you know, like necessity is the mother of invention and it really pushes you to come up with really interesting, unique, novel ways of doing things like you know, small based web tests or using Google AdWords or things like that to test some of your ideas. And I think that some of those things are really fantastic because they are very behavioral based, but they also give you the opportunity to very quickly kind of learn and iterate on something in a relatively safe environment.

I love those examples, like super scrappy ways of getting really quick information.

Right.

What other good examples are there of testing quickly and being scrappy?

Yeah, I mean, from my perspective, I think everyone will agree with me here. I don't think anyone should go to market or develop a product without doing some sort of research. I understand the need of getting fast insights quick that you can react and rely on. Where there's an abundance of tools out there in the marketplace today that allow you to do real time testing, they'll allow you to do in the moment testing quickly and more cost effective. And there's a ton of those tools out there. There's a ton of companies out there presenting them here today. So, you know, that's a great example. But also you can still do really good research very fast.

I guess I have two examples. Leveraging an online community is a terrific way to really quickly and also to iterate as you go. So that's a tool that I think is one that you ought to definitely think about having in your, in your toolbox. And then more recently finding, you know, there's so many new technologies and suppliers out there with new types of techniques that are specialized to do one thing really, really well and very cost effectively. And we recently found one that could kind of help us to screen through a lot of those 2,000 products that I mentioned to identify which ones are most likely to be winners and which ones aren't most likely to be dogs in market. And it's not going to be perfect, it's not going to be a one for one. But by having that specialized tool, really focused, automated, within a few hours, we can get an indication of whether or not consumers think it's worth pursuing in.

The media and tech space. It's so fast paced that often you don't have time for research. And we go by leadership's hunch or a product person's hunch that hey, this is going to work. And unfortunately we don't all have that gift that Steve Jobs would have. And oftentimes we put something out to market without testing it first and we're almost setting ourselves up for failure. We're putting all these resources behind it, from engineering to product to marketing and sales, all behind it, not even knowing if it's going to be effective, if it's going to succeed or not. That need for faster research is of course there, but at the same time there's the flip side of it where we need that research fast. But what sacrifices are we willing to make for that research?

That's a good question. Do we feel like sacrifices are being Made in order to get research done so quickly these days?

I believe so. I think that if you take a race, any time you're trying to finish that race faster, you can always finish that faster by cutting corners. And you're only going to get caught by people who actually know what they're talking about and who actually notice if you think about a race. And I think that applies in our world and that one philosophy, by cutting corners, if that's used by, in the wrong hands or by people with a lack of experience in this, we're all setting ourselves up for failure. And I think this challenge for speed in our industry has opened the doors and opened up this opportunity for. I have no problem with smaller research firms or startups, there's no problems there. But I think it's opened up the opportunity for people who aren't well versed or who aren't knowledgeable or have that strong foundation in research to offer these research products and the speed at which they're doing. Maybe it's a technology company that thinks about the technology first and research second that's really setting ourselves up for failure. And you know, I can't tell you how many times I do get pitched by companies that come and offer their research products to me or their data products to me. And as soon as I start grilling them about some of their best practices and the academics behind their data, all of a sudden it's like a deer in headlights. And in the media industry, the whole ecosystem we're built, you know, upon, we got the publisher side, we got the agency side, we got got the clients like Laurie and Michael here, and there's so many players here. Sometimes these new research companies, quote, unquote, research companies, they won't approach the research side of these businesses. They'll approach the people without that research function. Maybe it's a planner or somebody who doesn't have that foundation that we do and that's where it can actually slip through the cracks. And they'll start using the data without really fully vetting it.

Yeah, I mean, I agree with some of the things you mentioned. You did also mention a lot of times you are going to market without any research. And I'd rather cut some corners and be 50% or 70% confident on data by cutting corners than absolutely going blind. So there's got to be like some sort of give and take and there's tools and there's products and there's services in the marketplace that can give you some directional data within a few hours. And surely someone could Sacrifice a few hours to be a little bit more confident going into market.

I think a little bit of it, though, comes down to, like, as experts in market research, really being experts in market research. So, like, really understanding on some of these tools, if you're using something which is giving you research results in an hour, is it really going to be that representative? I mean, I think you need to know that for the type of decision that you're making. I think you need to know if you're using something like a survey monkey, that they get a D minus on 538 for research quality. Right. These are things that I think are important that you as a researcher, you know, kind of really understand. I think as well, like if you're doing some, like a lean approach, if you're just iterating and iterating and iterating, but you're actually not ever, like, going and understanding deeply, like, what is the consumer need you're solving? What's your point of difference, which you're never going to get from just throwing out minimally viable products over and over again? I don't think you're going to get anywhere. So, you know, I think that that's why within kind of the modern market research toolkit that we really need to, you know, maintain a lot of the expertise and standards that we have while we're evolving, you know, to some of these kind of new tools and techniques as we go forward.

So are we saying, you know, there's an abundance of data out there, there's no shortage of data. Right. Like, we can always get some sort of indication on anything that we're trying to do. It almost sounds like in some cases, if I'm not 99% confident, I'm not gonna do any research. Is that what we're saying?

No, I don't think we're saying that at all. I think there is a certain level, there's a certain threshold, threshold where we want to minimize that probability. You know, bringing something close to 50 or 60%, we might as well make a guess because it's a 50, 50 chance that the results, it was a reflection of chance versus the actual stimuli.

Right.

I'm not saying that we should always go for 99%, but I think it also depends on what we are testing. Is that at a product level? What's the investment here? Is it. Are we doing something where we're testing advertising or media, or is it something where we're actually going to build a product out of this? Is it a new car? And I think we have as a company or as departments, we have to determine what is our tolerance there.

It sounds like a huge challenge for researchers then. So you've got more and more tools and techniques coming into your portfolio. Increasing pressure on speed, reliance on your skills and expertise to actually vet all these new vendors and really understand what role they can play in your portfolio. That's a hell of a challenging role compared to the role running a big tracker like we used to have.

Yeah, I mean, I think for sure. And I, you know, like, we talk about the pace of change. I mean, the pace of change is moving very quickly. And so even kind of keeping up with the technology, I think, is, you know, kind of difficult, you know, from that particular standpoint. But, you know, I do think that the opportunities that we have to learn today are also much, much better than what we had in the past. And I think, you know, from an industry standpoint, one of the things we really have to think about is rethinking kind of how we do our work. Like, an example for me would be that we used to do a lot of habits and practices studies, so big surveys, very often diary kind of style type of things. But why do a laundry diary study when you can put a device on somebody's washer and dryer that tells you exactly when they're doing laundry, how many loads, what type of loads, et cetera kind of thing? Because I can guarantee that that data is wildly better than anything that they were telling us in any survey that we were doing before. Now, the issue with that from a skill set standpoint is like, okay, we did a survey before. We might have used some basic statistical analysis on that. Suddenly you're getting a big data stream from a whole bunch of washer and dryers, and you need to figure out how to get that into some type of a database. You actually need to figure out probably how to get it up into Amazon Web Services. You probably need to know a little Python so that you can do some kind of programming to make sense of that particular data. Like, I do think that there, you know, is a bit of that that we really need to be thinking through, because I think that that's the future, like, core skill set. That, at least in the analytics part of the work that we do is going to be critically important.

Yeah, and we talked about this prior to this session, is that it's not really data. You really need to get insights fast. How do you cut through all that crap through the data to get insights that you can react on quickly through multiple different sources? Of data?

Yeah, I mean, I don't think anyone can disagree that there's an abundance of data. We have all the data, data's all out there and there's multiple streams of data. As Michael mentioned, you know, there's a need to be able to bridge that, bridge all the gaps between the data and being able to tell a story. So like I said, there's a lot of data out there. I think what we're lacking is insights and that's what we really need here. But Michael mentioned something before about the whole washing machine and the behavioral aspect as compared to survey self reported type of research. And I think that's something that we need to pay extra focus on because when we think about some traditional methods, you know, whether it be focus groups or interviews or any type of self reported type of metric, these are rational measures and they're oftentimes, you know, it's a very useful tool. However, I think we have to be weary of over utilizing those tools because we're in an environment right now. We're in a world where, where things like behavior and emotional response is going to be as important or more important in certain aspects than rational responses. And what I mean by rational responses, when you think about self reported metrics, these are questions, these are answers that a person or respondent has a chance to think about their answer before they answer it. So there's of course an inherently, there's an inherent bias within that. But when you, when you look at somebody's actual behavior or look at somebody's emotional response through things like neuroscience, these are things that they don't have a chance to think about. They either do it or they feel it. And in this world where things are getting more and more personalized, from the data on the washing machines to everything that we're doing on our phones, these are areas that are so personalized and extensions of ourselves that we are going to have to rely on on these types of measures rather than the spoken word.

I agree 100% with what you're saying. I mean, I think in this day and age having to ask behavioral questions or recall questions to panelists is ridiculous. With all the tracking tools that we have today, and it's obviously, we all know this, it's getting harder and harder to get panelists to complete longer surveys. So you know, what we're trying to do, what we're trying to focus on is the behavioral information, the observed information that we can track. We don't ask and then we're only focused on the why behind those Behaviors.

Laurie, have you noticed this trend towards behavioral data as well as research data?

Yeah, absolutely. Although I would say that our business might differ from yours in that we are only a very small portion of our business is direct to consumer, over which we know what their actual behaviors are, who's buying what, most of it is through wholesalers for which we have very little behavioral data. We don't even have. We don't have very good in our industry, in the footwear and apparel industry, we don't have very good scanner data. So we don't even have great market share. So I would say that we don't have great data all the time. So we're having to stitch together a bunch of suboptimal sorts of data. And I think for us a challenge is there is this perception out there that behavioral data is the holy grail. And what I struggle with is how am I going to get it, how am I going to integrate it with other forms of data, who's going to analyze it and interpret it and generate insights. So it is exciting to think about that there is more behavioral data becoming available. And as we move to more of a DTC or direct to consumer environment, we will have more of that. But we're still at the early stages of knowing how to get the right data and how to analyze it and then how to synthesize it together with other forms of qualitative and quantitative data that we're more familiar with.

That's hearing more and more challenges to the role of a researcher here. My God.

Well, I have sort of an opinion about first party data versus third party data. I certainly think that if you're not getting first party data, you're late to the party. And specifically because of the challenges that you end up having with trying to integrate all these different data sources, I think the different walled cities that you have, et cetera. So I think one of the big challenges from a research standpoint is really trying to figure out how you get more of that direct data right from the consumer that gives you the ability to really better understand, I think, more directly what their behavior is.

Here's one more challenge for the role of the researcher. Then why don't we focus on some solutions maybe. But you've got all this new data, all these new data sources and you've got these quick and easy research techniques, some of which might need a little bit of guidance before you let them loose. Everyone in the organization has now got access to all of this. How do we control that? How do we make sure that Smart decisions are being made with it.

That's right. That is another thing that you could add to the skill set that we need is so everything is now on a platform or a database. And yes, you're right that the organization has more access to more information.

But.

But another part of my role is now training people on how to interpret the data, how to access the database, how to use the dashboard that they can read the data on. So while it opens up wonderful opportunities and it speeds things up and democratizes the information, there is the challenge of making sure that the consumer of that data knows how to use it responsibly.

I think Laurie's right on point there. Democratizing that data is great, but in the wrong hands it can be widely misinterpreted. And that's definitely a challenge I think a lot of us face across our organizations. And it's great to have these dashboards and platforms and interfaces where we can access this data and access the research. If they're speaking to it incorrectly or interpreting it incorrectly, it can be, you know, it can be very detrimental. And building protocols around that and probably the opposite now of democratizing it is one solution, but I think we have to be. We'd have a certain level of leniency there, but we still have to keep a close eye on it as researchers.

See, I have a little different point of view there where I do think one of our big goals in data is to try to make it as transparent and ubiquitous and democratized as possible. I think not only inside of our companies, so that, you know, everybody's working off of the same kind of data page and has the same access to knowledge and we're not using it as some kind of weird political currency, but we really are opening it up to everybody so everybody can benefit from it, even to the degree that the consumer can benefit from it. So for example, we have oral B toothbrush that basically you can hook up to an app and it tells you how you're brushing if you missed a spot, all that kind of stuff. To me that's ideal. The more that we can cut any middleman out, give the data back to the consumer as an added value. I think that that is even kind of the best scenario of all, if you will.

I think you're right in terms of if it's a one off type of metric. I'm thinking about it in terms of, I guess a greater deal of data where as a non researcher they can misinterpret comparing, let's say an A B test. I Think I was looking at it through that lens. So how would you. Do you ever deal with people who do misinterpret that data or is it streamlined enough that they're able to understand it?

Oh, no. I mean, I think you can give 10 people 10 sets of data and they'll have 10 different opinions on what that data says. So I do think that certainly giving people training in terms of how they should think about interpreting some of that data, I would say say is useful from that standpoint. But I also think that the more that you can help put it together in usable dashboards, visualizations, things that make it really clear what the conclusions are. Again, that's what as great research is a big part of what we do from that standpoint. But I do think that you're never gonna solve the problem of stupidity, regardless of how hard you try. And so I think at some point in time, I think you've gotta kind of balance it. And I think at the end of the day, giving somebody access to that information probably outweighs the chance of the stupidity that inevitably is going to happen at some point in time.

In the story arc of this panel, we've gone through all the challenges and the woe is me. Let's have some uplifting stories now. Who can tell a story of an organization that's gone through some change and is now faster and quicker as a result?

I can start, I mean, you know, our holding company, itwp, we purchased Harris Interactive almost five years ago. Now primarily a full service research firm who didn't leverage many technologies or tools. And one of the first things we did at Toluna was go to every single one of those offices and pitch our entire technology stack. Obviously I wasn't the most popular person doing that, but we did it. And now almost 50% of their revenue does funnel through some sort of automation or some sort of Toluna digital technology platform. So we've been able to be more efficient, we're returning research and results much faster and we're delivering insights much faster.

That's great examples from the client side.

Okay, so I agree with one of the other things that Michael said was from adversity comes innovation. And we had a situation in which we had some budget cuts and we needed to do things much faster and much more efficiently. And as a result, we started exploring new sources of information that could give us an indication of how we were performing, what consumers liked about the brand and what they didn't like. And that led us to really innovate and start using all new sorts of data sets. And so I would say that that resulted in pretty much of a win win because we were faster and we were more efficient as a result.

I mean, I think, you know, at P and G, I think we're seeing that across our businesses. If you look at our time to market, you know, from like idea to market, it's dropping dramatically, you know, probably like, you know, like already half the time of what it used to be. And you know, I think on the way to being much, much faster than that as well. I think in addition to that, we've also really started, you know, giving ourselves perspective, permission to explore different types of business models that we weren't necessarily doing before. So looking at kind of service models in some of our laundry business and I think again, using a lot of small scrappy, startup kind of methods of doing that type of thing as well as. Even if you look at a lot of our marketing budgets and how we've kind of changed our marketing budgets and what type of advertising we're doing, you know, we've moved an incredible amount of media dollars out of traditional media very, very quickly, are doing a lot of very, very effective, what I would consider to be kind of poll marketing and including like on some of our most successful businesses like SK ii, where they're doing that almost exclusively. So, you know, I think when we're talking about these approaches, I mean, I think that they definitely are paying immediate and huge benefits, at least to our business.

How about empowering your staff to do either, you know, having that option to DIY certain aspects of the things so that it can be more agile and faster to market versus always maybe relying on a full service firm.

I think like in our particular case, it's been a huge push for us and I think for a couple of reasons. You know, I think one, you know, the technology today makes it so that, you know, like, I think that I'm not sure why you would really think pick up the phone and call a supplier when if you have something right in your desktop, I mean, you can hit a few buttons, launch the survey, get the results back. A lot of the analysis is automated, et cetera. I mean, I'm not sure what the value that you get from kind of going with a full service supplier is. It's obviously more cost effective, but it's also something because of the fact that you're doing it yourself. You're doing, I think, a much, much better job of thinking through, like what questions am I asking? What am I going to do with this person particular data, what's the analysis as well as having like a more tangible feel for, you know, what the consumer is actually saying. So I think that there's multiple benefits there and I think that, you know, I think we're going to continue to see that be a direction in the industry. I can't understand like, you know, like why almost anything that we would do going forward wouldn't be something that, you know, wouldn't be almost completely, you know, kind of automated to the place where it could be diy.

I think what I'm hearing is that the fast fail approaches live alongside traditional research for most of us still. In which case, let me take the top question. How do you decide which products or initiatives get the fast fail treatment and which get the traditional research rigor path?

Yeah, that's all about the level of risk and the level of investment that's required. I mean if it's a seasonal color that is in and out, you can go with a, with a fail fast kind of approach. And DIY is something that we're definitely experimenting with and actually giving the product teams the tools to do the tests themselves. But if you're talking about a more major change in one of your iconic products, then you're going to want to give it the full industrial grade rigor that you always would have.

Do you disagree with your stakeholders sometimes about which treatment to give a business question?

Yes.

I think it also depends a lot in terms of like where you are in the process. When we're talking about a lot of these kind of quick approaches. I think when you're earlier on in any kind of innovation cycle, a lot of that type of thing makes a lot of sense. I think if you're getting ready to launch something in market and you want to super robust volume forecast and you're trying to dot every I and cross every T and you know, make sure like you understand like how many products you need to produce and you know, how big of a factory you're building, like some of those kind of things. I mean that obviously, you know, I think is a much riskier situation. So you know, I think a little bit of it also depends on like where you are in the innovation cycle.

On the media side, there's a lot of off the shelf solutions that we can always use for some of the more regular tentpole type of research. Given the current climate that we're in and different questions that are coming up within that ecosystem of media platforms and advertising, we do have to look for some new innovative and out of the box type of research, for example, with things that are so highly politically driven now, how is that affecting advertising? And advertising alongside some of that content? And these are things that off the shelf solutions don't always answer for us. So the more out of the ordinary some of these topics are and what our goals are, that's probably what helps us determine if we're going to go the traditional route or something a little bit more innovative.

Let's take the next question, but can I ask for one word answers to this question, please? What is more important, quality or speed?

Quality.

That was quick. Anyone going to disagree? Both. That was a one way you got to buy one.

I would be in the bolt camp as well. I think we're in an era where you don't need to compromise. So I think that the big thing, I think that they used to always say about market research was you could have cost, you could have speed, or you could have quality, but you could only pick two. And I just don't think that that's true anymore. I think with technology that you can actually in many cases have all of those things. I think as well, it depends as well like what you would really define as quality. Because I think sometimes we underestimate. For example, I remember years ago Harvard did kind of a study on qualitative research and they basically said, well, you can do 10 focus groups and you're going to get about 60% of the insights that you would generally get in that particular subject. Or you could do six really good in depth, one on one interviews and you'll get 90% of the insights. Right now, I don't know, you know, like if you actually ask people from a kind of more traditional research background, they'd be like, oh, but you know, like those, you know, whatever dozen focus groups of 12 people, that's almost a quantitative base size. But like, I would argue that the quality of data that you're getting is a lot worse. I think, you know, polling data is another great example. I mean, if you listen to polling data, you know, in the state of Florida at least, you know, for the Democratic governor race, you would have been way, way off. And I think we continue to see that. So, you know, I think, you know, quality is a little bit of also like, you know, how you define it. But I think sometimes we're underestimating the power of, you know, like different types of data. I think behavioral data certainly means a lot more to me than claim data. I do think that, you know, like qualitative data very often is a lot richer source than quantitative data. But I do think, you know, like when you get closer to, you know, kind of your end goal or end market, you probably, probably do want to have some broader quantification of size, of prize, et cetera kind of thing. So it's a little bit of making sure that you're really also defining what quality means.

Nice one word, Laurie.

Oh, you're not going out of this one.

You thought you did a person over there.

No, I guess they're both important and I would say it depends. There are certainly times when you have to make some. I think that in my organization we have to make trade offs. I would like to believe that we could have both all the time. But sometimes I do think there's trade offs and we sometimes opt for speed, we sometimes opt for quality.

I'll pick up your point about behavioral data. One of the questions further down the list is, is behavioral data just rear view mirror data? Is there a problem with behavioral data because it's just looking backwards?

That's kind of interesting. You know, I think what it, you know, if you actually look at this, the. There's a really good kind of body of evidence that basically says the best way to predict future behavior is to look at actual past behavior. And if you actually look at the kind of predictive nature of that, it's actually quite compelling, particularly if you're looking at what people are actually doing. If you actually ask people what you think that they would do, it actually has a very, very low correlation with what they actually do or predicting the future from that standpoint. So I do think that, you know, behavioral data is to some degree a bit more predictive of what people are actually going to do because it is what they're kind of already doing. You know. Now, that being said, I don't think that any type of research that you would do can necessarily predict things about people that they don't know. So when you get into the whole Steve Jobs thing about, you know, like, oh, well, you know, consumers don't know, you know, kind of thing, or, you know, the Henry Ford. If I would have asked people, they would have told me they wanted a faster horse. I think that all of that's very, very true because people don't know what's possible. And one of our big things that we're doing as researchers is trying to bring together what's needed with what's possible. And I do think the behavioral data gives you a much better understanding of truly what's needed from people, really what the true jobs to be done are then we need to connect with what's possible to create sort of what the new potential ideas for people to get.

A full understanding of the consumer though you just can't rely on one single source of data. It's not just quant or qual or behavioral data. You really need to understand the full 360 degree picture. And I think what the biggest need in the market today is platforms and technology that allow you to combine all these multiple sources of data. We're not talking about data anymore. Now we're talking about actionable insights that you can cut through and react on.

Great to wrap up then maybe one piece of advice. If you had to give one piece of advice to people in the this room based on your experience of trying to embrace fast fail, what would it be?

Keep learning. I think of myself as a lifelong learner and you know, talk to people out there, talk to the vendors here today and at this conference and learn as much as you can. I learned so much from my vendors and I think they're a really valuable resource.

I think it's important as researchers to be very well rounded in multiple areas of research. There's a lot of people who focus or specialize in certain areas. But I think it's important in this day and age to have vision and to have transparency as well as knowledge of different areas, different methodologies. As fast as different industries are changing with the impact of technology, we have to move just as fast with it. And I can't remember where I read this, but it stuck with me. And it said that we can't use yesterday's research methodologies for today's, for today's businesses and expect to still be in business tomorrow. And I think that was. It really stuck with me and it really was a reflection on how fast we're moving. And I don't think we should be afraid to not have the answer or not know how to do something. All it means to me is that it hasn't been done yet. And that's something we need to look forward to.

I think that the great insights come from creative problem solving. So I think you should always be trying to figure out how to make yourself a better creative problem solver. And I think one way to do that is to really constantly challenge yourself to do something that's a little bit out of your comfort zone, to learn something that's a little bit more difficult for you really, you know, versus, you know, continuing to just try to do the same thing over and over again.

Yeah, I'll be quick. I think it's just to be flexible. And AJ Mentioned methodologies. You know, I agree. I think just because you've done something a certain way for a certain amount of time doesn't necessarily mean it's good for today and tomorrow. And when we talk about this whole quality versus speed thing, do we really need like to use a methodology on sample composition and quotas on, you know, brackets of 10,000 for income every single time. Those are things where you can be a little bit flexible and get faster speed and hopefully not release much quality. Great.

Thank you so much. Fail fast, but not because you're trying to, but because you're trying to move fast. And it will come as a byproduct. Fail early so it doesn't have big implications in market. Integrate lots of different data sources. Don't compromise on quality. You can do quality and speed. Keep learning. Keep, keep, keep learning. It's one of the things I think I picked up from this. Thank you very, very much. I think it's time for Mexican night, isn't it, in the exhibition hall. So I'm really looking forward to that. I know you'll all be joining us first thing in the morning for Peyton Manning, let me tell you, had a couple of calls with his team and exchanged many emails. He's going to be the most thoughtful, the most considerate, the most focused speaker on the topics that we're talking about this week of the whole week. He's appropriately closing the week. It's going to be the highlight of it. I'll see you in the morning.

Sa.

Around. Another promise, another scene, another a package like.