Webinar: Synthetic vs Human Validation

So welcome to this webinar on the road to autonomous validation. So today we're going to talk about what the current state of synthetic validation is and we're going to get quite practical. So we want to walk you through five of the practical use cases on that we are currently seeing and how we are using synthetic validation. We'll be sharing some of the insights from our client projects, obviously without going to too much detail or sharing any confidential insights. And at the end we'll have a, we'll halfway through we'll give a brief pause, we'll have some room for questions and hopefully a lively discussion. And towards the end we'll share our overall conclusion and then also open up the floor for any questions before we dive into an introduction towards synthetic validation. Just as a very brief introduction so you know who you are talking to. Amnick of the partners and CEO at Board of Innovation. And I'm joined today by Linda, who's one of our senior business designers and is leading our internal AI efforts. And Board of Evation is a global innovation firm. We work with the world's sort of most ambitious businesses to imagine new products, services and businesses and create them today. And over the last 11 months now, we've been diving really deep into the world of generative AI and specifically what it means for innovation use cases and where we see that evolving. And that's actually where I would like to start today. I want to start by sort of setting the stage and giving you a point of view in terms of where synthetic validation so specifically the testing of new products, services and businesses and gaining insights based on generative AI, where that's currently at and where we believe it's going. So let's start as always with a bit of a definition. So to us, synthetic validation is an experimental approach to test new products or services against typically consumer data. It can also be expert data that is generated by AI. And I think it's important to note that this isn't something that's completely new. This isn't something that's brought to the forefront by. Well, it's maybe it's brought to the forefront by generative AI, but it has been happening before ChatGPT was launched. But I think what we are seeing is that it's increasingly being used in that because of generative AI, there's just a lot more use cases that are becoming available to us all and I think we can already see it playing a role. So I just want to share three examples that I recently came across. Gaming typically interesting use case to look at what you're seeing here is, is somebody, and I'll share the YouTube link in the chat afterwards, who trained through reinforcement learning, trained an AI model to complete Pokemon RIT. So if you are from the 80s like myself, you might have spent hours in the car on the way to holiday playing on your Game Boy. This will look familiar. If you aren't then, well I guess you missed out. But basically can you leverage AI to learn how to play a game and is that actually something that you can then use to debug games in general to improve gameplay? And I think there's a whole range of these simulated environments that you can build to also start testing what human behavior, or simulate rather what human behavior could look like in the real world. If we're looking at more broadly the use of synthetic data, it's also something that's being used to generate a lot of additional edge cases that can be used to train self driving algorithms. What you're seeing here is real world footage of potential red objects from a red sign on a building, towards tail lights, towards even people with red hair that are being falsely triggered as a red light so that it stops the car. There's a lot of these edge cases in autonomous driving. Can we generate more of those? And that's currently happening through generative AI to then train the algorithms that are used for self driving. And then finally I think when GPT Vision was, or the vision feature in GPT4 was launched, it's really interesting to see how you can combine that with auto agents basically having two large language models talking to each other, asking to describe the picture that they're seeing and then ranking it, rating it and giving feedback which then that feedback is used to improve the prompt. So in this case you see a llama driving a car. The initial image is just a llama sitting in a car. Then that's being interpreted by a model. The model basically says like well it should be sitting behind the steering wheel. It's behind the steering wheel and in the final picture you see it putting its hand or paw or in this case I guess on the steering wheel. So we are starting to see this more autonomous loop that leads to better design. Basically I think overall what we're seeing, and that's maybe an important note, this is fields evolving very quickly. So a lot of the examples that we'll be talking about today will feel very outdated in three or six months. So I think that's always an important note when we're talking about this new and fastly evolving technology AI is at the worst it's ever going to be at the moment. And if you look back to where it was and what was amazing is collectively in November of 2022, yeah I think you can all recognize the speed at which that is changing, changing and I don't see any reasons why that speed is going to. Why that is basically why that would be slowing down. So ultimately what we see or we think is happening is that innovation is on a clear, clear roadmap towards higher levels of autonomy. And just to make the, to draw the analogy to self driving cars, we also have different levels of autonomy. I think the current state is probably that we, that where we come from is more of a human led or human only innovation process. And what we are seeing our clients do and what we see other agencies doing is starting to use AI driven tools like GPT and a lot of specialized tools to assist or to power some of the process. And we'll be sharing one or two of those examples later on in the webinar and that is basically just generating efficiencies across the innovation process. I think where it gets interesting is if you actually start to look at more autonomous models and where we start to think about how can you actually reimagine the entire innovation process. So by ways of example, where instead of using GPT to run interviews and generate interviews to gain insights, would you actually need insights in a world where you can generate infinite IDs and validate those IDs or solutions against synthetic consumers and basically prioritize down from there? And ultimately where we believe that is going towards more of an autonomous innovation engine, where if you're the CEO of a company, you would basically be being presented with new products, new services that fit within your broader portfolio, that you have an increasingly high hit rate continuously and that a lot of that work is happening autonomously. So what we want to talk about today, and we're just going to start with different degrees of autonomy, we're going to start with AI assistance. So what does an AI assisted video survey look like to get insights then? If we same use case but AI powered and we'll explain the difference in a minute. What do AI powered semi structured interviews look like? A tool that we've been using on our projects recently called outset AI to then move. We'll take a break afterwards and my colleague Leonard will dive a little deeper into the semi autonomous element. So if you want to get multi market consumer insights, some learnings from a project that we've recently done with Nespresso, one of our global clients, we're going to go towards the more complex B2B sites. How can you get synthetic stakeholder feedback in very niche markets from experts? And can you use that to prioritize B2B concepts? This is a tool that we've built using the OpenAI API that helped us to do that in Google Slides to then finally give you a very early preview into what autonomous pack design for consumer brands could look like. And this is a version 0.1. This is something that we are experimenting with internally with some of our clients that we just want to give you a sneak peek into. So with that I'm going to dive in. I'm going to start with the first two cases that are probably more the current states, moving beyond the human only part of validation and starting to leverage some of, yeah, more tech tools in AI powered tools. So first one, AI powered semi structured interviews. This is a project that we did at the time for Levi's and in general we wanted to just get a quick understanding of what our users contexts look like across different markets. So a typical approach would be well, get a survey done or go talk to a lot of people and run user interviews. But limited time, limited budgets, want to look for a more I think innovative or scrappy approach in a way. So turning towards bringing different tools together. So in terms of preparation, you can easily give that research context to large language model, generate discussion guides or interview guides. So you're taking away some of the manual or physical work that typical consultants or somebody in your team would be doing. Then we record those questions into Video Ask, which is basically a tool to do video surveys so that we can actually get, we can actually get real consumer insights quality, we can see their context, but we don't need to go through the effort of interviewing dozens of people. Then we run the interviews and I think that's the nice part. You can basically just sit back and wait while your responses come in. And the only driver to increase your scale is recruitment costs. And another part where I think is getting interesting now with generative AI is there's a whole range of tools out there. I think for this one we used Rusk AI that then allow you to scale that towards different languages. So this is Kevin. Who was testing the tool, asking Julian, our colleague, who is Romanian, to check this AI generated translation. Now I have no idea, I don't speak Romanian, but apparently the translation was quite good. This is again three months old. In the meantime we have features like lip sync. You're actually starting to see with GPT4, the latest released features that it's starting to really imitate properly the tone of voice, the timbre of your own voice, so it doesn't feel like a translation anymore. So a lot of the previous barriers that were out there in qualitative research methods around translation so forth are just disappearing, which I think is just opening up a real range of new opportunities to then move towards analysis where you can use a whole range of different tools. And all of these tools are starting to build in genai features to do automated labeling, tagging, insights generation and so forth. So this is really, I think, sort of like the ground level of validation using and combining different tools to yeah, basically be able to prioritize your time so that you can actually get to deeper insights. Another example which I want to call AI powered. And the difference here being that while the previous was AI assisted assistant, meaning it's taking over parts of humans that are doing the work here, we are actually replacing the human elements of one side of the conversation. So context working for. We are currently working for one of the top three food and beverage companies and want to explore new premium products and activation methods. And we want to get some quick consumer insights to develop new food concepts. And we want to do that at scale across one market. And what we ended up doing was running an AI powered survey or yeah, survey. I'll go into detail in minutes with over 220 people. And it was nice thing about this, like basically launch it on a Friday, come back to work on Monday and you have some initial insights that are waiting for you. Doing this the traditional way would take two weeks. A lot of recruitment costs, a lot of probably doing physical analysis and so forth. So what we ended up doing, we're using this tool called Outset AI which basically allows you to design or train an interviewer. So this is basically a chatbot that is going to talk to consumers, but that doesn't follow a script that has the ability to go off script depending on the settings that you give it. And also the ultimate learning objectives that you're trying to get out of the interview, but also the respondents of each individual, individual interviewee. So build a discussion guide, then run the interviews. So here you see some screenshots from, well, the interviewer, which is a chatbot asking questions. And then the question is basically referring back to what the interviewee, which in this case is called you, is sharing. So to blur some of the elements. But this is basically happening at scale, but it's happening with real consumers or real humans on the other sides that are being recruited through the platform. So got 220 of those done in two days. And then this is what you wake up to on the Monday morning. So basically an automatic synthesis which then has quotes that you can click through to the transcripts to understand more detail. Also happening in multiple languages, which is interesting. So this was happening in English but also in Dutch for example. And we get some responses to the specific questions that we're asking here and an early analysis. Is this good enough to be driving things? No. Is this a great starting point and a good way to start your week? Yes. So it's something that we are increasingly using across. Across. Across the work that we are doing now with that before I hand things over to Leonard and I'll stop sharing my screen in a minute, wondering are there any questions before on the sort of like AI assistants or AI pirates move before we move on to the more autonomous side of things? And I'm just going to. You can either just drop them in the Q A and you couldn't hear my computer audio apparently. So that's a shame. I'll fix that for next time around. Well, in summary, you could hear my colleagues speaking English and they're Romanian.

Any questions of Rosc AI in the chat as well. So they have some videos on their website as well.

Cool, thanks. Any questions from the audience before I hand things over? How does the platform recruits human interviewees? Well, I guess it's. It's actually a good question. My guess is that they have agreement or an API link to any of those user interviews platforms or we use a whole range of those tools ourselves actually. So there are a lot of those tools where you can basically get recruits specifically if you're talking about sort of general public within a day or so. How can we learn about these tools? Well, I think one of the ways is just experimenting with them. Another way is you could start following us because we are sort of continuously updating our tool stack and that's also available just open source on our websites. How are participants reacting to AI tools? Would you say there is different things? That's a great question, Denise, Thanks. I think actually it is a really good question. I think in a way, I don't know. So we haven't really explored the difference or noticed a big difference. But also I don't think we've been paying that much attention to it since we really use it as sort of like an early scan and then we always still talk to human consumers. I think the main point here is that over time people will not know that they are talking to an AI unless they're being told. So I was at a summit a couple of weeks ago where we were where they were presenting a new AI built avatar of Lionel Messi that now is being used to actually record all of his commercials. And you couldn't say the difference. So I think that's ultimately where we are going. And final question then I'll move on. We'll be discussing the bias at the end of the presentation. So we'll get there and we'll be dropping in a list of the tools as well in the chat throughout any of the tools that we're using. So I'll get those links to you in a second. And then Linux, do you want to start sharing your screen and take us through the next three tools?

Sure, sure, sure, sure. Okay. So as Nick already mentioned, we did a multi market exploration. The project at hand was with Nespresso. We're their agency of record globally and one of those projects that we ran with them was sense checking new coffee machine free features in especially in foreign markets. So context is that whenever they launch new machines, of course you do need some, some synergy. You do want to send a check that a feature that you launch in market A, you can also launch market B. And in order to really do that at scale, we actually use the platform called syntheticusers.com which is our go to for search exploration. What we did specifically is we prompted for more diverse audiences from foreign countries. So how does it work actually? So here you see for example a list of audiences that we described. The ideas are fictional and the audiences are as well. But just for demo purposes. We described the same audience basically across the globe in US, Korea, Germany, but can be any country. And then we also made a summary of their pains, their gains and their jobs to be done, their goals. And this can also be of course validated with real users or with LLMs. But so this context we already had, we prompted it and then we gave our solution which is here as set a fictional example, a machine that can help you to create milk foam in your coffee at home. So how would a person in Germany react to the same idea compared to a person in the same demographic in Korea? And then we use this really to sense check. One of the cool features of this platform specifically is that you can also chat with individual users and it also makes summaries. So basically we wouldn't be launching a feature that didn't make sense for a specific country and we would also not do something without doing that extra sense Check. So we will also double check. Does it actually make sense for them moving up in the ladder of autonomy? Let's say we also did synthetic stakeholder feedback and concept prioritization. It's something we hear often from B2K B2B companies like is this also relevant for us? And I would say definitely. We worked with a leading global company in construction to try to prioritize new ideas with regards to building up their revenue stream. What we did here specifically is and I'm going to go through it step by step, we we basically hacked a way together in Google sheets. We installed the GPT plugin which is an easy thing to do. You just need like a GPT API key. And then we described our ideas here blurred out. And those ideas needed to be validated with the complex ecosystem consisting of of course architects, but also engineers, designers, constructors and so on and so on. I think we had over 10 stakeholders here and eventually we mapped. We mapped them out and we made them score or different ideas. We also asked there's no two identical architects let's say. So we asked ChatGPT to come up with a list of 25 Personas of Australian was the context here architects based on the following information provided and then we had the information written out based on our Persona and then we asked the same query to actually rate these ideas on a scale from 1 to 5. And then we would take averages and we did that for architects but so as well for all the Personas mentioned and then you end up with averages. So we basically try to summarize it in this graph. How to read this. So this is how different key stakeholders. We only focused on the key stakeholders rated different concepts. Two things, two takeaways. Main takeaways from this is for example it allows you to already like do deprioritize a couple of ideas just because they might be hard to make a reality because some stakeholders are not on board. So we could deprioritize these for now and something else is like on an ID or on a concept level we could basically see who was least on board and who we should focus our further efforts on. So we took actually the synthetic feedback from in this case developers. We asked GPT to generate like an interview script and then we went ahead, we interviewed real developers based on that interview script and we iterated on the concept actually to make it better. Moving on. There's also autonomous pack design and as said by Nick, we're still experimenting with this. This is super recent as you can imagine. Like GPT4 Vision 4V only launched, what was it, one or two weeks ago. And what we see is that it actually unlocks a lot of new possibilities, and we really see the space moving here. So this challenge that we're experimenting with some of our clients is how might we get synthetic consumers to fuel an iterative pack design process where, if you remember the example of the llama, Nick covered it very briefly. There's also an example of a parrot, where this first image of a parrot supposedly driving a car was a query in Dolly and it actually conversated with another AI agents that served as, like, an AI image analyzer that would basically give it a rating from 1 to 10. Like, how accurate is this parrot compared to the initial prompt? It will give a score. Let's say this is a 4 out of 10 because, yeah, well, the parrot should be in the car, and it should be actually like, having basically close to the steering wheel to show that it's driving the car. What happened here is it would actually fuel back the feedback it would take, generate a new image, it would analyze this again, and it would basically make these iterations nearly autonomous, basically, to get to the best results. Well, knowing how this works, you should also be able to imagine how this can work for synthetic consumers that actually talk to your image generator and that can basically give feedback back for their specific context. So let's say the fictive example of let's design the best chips package for Gen Z and Laos. If you start from the left as, like, your first output, and you get feedback from this specific audience in Laos, and I wouldn't know what the feedback would be, but the LLM is actually getting better and better and better in giving constructive feedback and would basically come up with this V2, and this would fuel back in new feedback, and it would go on with a V3, V4, V5. And, yeah, this is what it looks like today. It's definitely not yet at the quality where human interviewers would be, but I don't think it's hard to imagine how human interviewers can also play a role in this iterative loop, giving constructive feedback and making the outputs more and more in line with what you would expect. So that was an example of where we see it going with a fully autonomous loop where basically AI serves the role of an interviewer and the role of an interviewee. But there's still, like, a lot that can be done before that, since, like, not like, at both sides. There's still something and there's still a lot to say actually on the quality of human interfering in the process. Still, basically, to summarize, synthetic validation will not replace human validation. Well, let's say yet, but human only validation will be replaced by AI powered and semi autonomous. So there's a lot of tools already in the process that can support you as an interviewer, as an interviewee, and we also see it becoming more and more autonomous and it enables you to validate sooner. So before you do that manual round of interviews, you can already get some first feedback that also helps you to conduct or construct an interview guide. Then you can also diversify your feedback. It's a lot easier to become purposely more inclusive in recruiting and in talking to a diverse set of users. And lastly, you definitely can increase your hit rate because it's also able to basically identify the least performing ideas and it can help you move the human selection bias and make you focus on the right ideas to move forward. That being said, I think there's still some time left for Q and A, so giving it back to you, Nick.

Unmute myself, thanks. There's a lot of questions coming into the chat, so please do upvote the ones that you would like us to address in the next, I think five or ten minutes or so. I'm just going to take some at random. From Sandro. You made a lot of examples where you wanted feedback for user groups in areas you have no knowledge of, example leos. Do you think in areas where you are familiar, familiar humans are still as good or better? I think it's a good question. I think it comes back in a couple of the other questions that I saw in the chat, which is basically how do we know the models are actually providing valid insights or validation versus they're just hallucinating or making things up? Which is in essence the core of a large language model is predicting what the next word in a sentence is going to be. We actually do always make sure that people in our team have an understanding of what the local context could look like. So that might mean that we actually understand the construction ecosystem or our clients does and that we then validate across the different phases of the process. Is the feedback that we're getting, does that make sense to the Personas make sense and so forth. In some of our bigger projects we're actually being more rigorous about that and we are doing sort of a B testing for every separate experiment where we do things the quote unquote traditional way and we're using a more synthetic way and then we sort of measure correlation, for example. And in that way I think you Gain a measure of certainty or a measure of validity for specific use case or for specific concepts. I do think even in those cases where it might not be right, it's still a good starting point to then explore further and validate more in the traditional or in the human ways. So I think that's bit where the art comes in, where we also see things evolving very, very rapidly. But I don't think we're at a place yet unless we're talking about very incremental innovation for a very general purpose audience on, for example some food concepts and so forth, where we can really say it's going to be autonomous. That being said, a year ago we couldn't imagine the idea of autonomous innovation at all. So I do think that those things are expanding quite rapidly and there's some just technological progress not needed in terms of being able to measure the validity of a response.

Then scanning maybe one on my side. And I think I'm summarizing a couple of questions with that, but I'm reading between the lines. What makes you so sure that you can trust the results of testing with synthetic users? Well, good question. We did, especially in the early phases, but still, still we're running a lot in parallel. And yeah, what we presented today, we're actually sure it can get to better results. I have to say, for example, the Nespresso case, it's still evolving. Some sensitivities locally are not there yet. So always be cautious, always try to a B test when you're using this. I don't think you have to go all the way on both ends, just try to run it as much in parallel and explore the boundaries. But so what we presented, everything has been a B tested and we actually see promising results, only promise to become better, of course, in comparing both real user insights and synthetic insights.

And then Ishmael has a question around the B2B case. How did we describe the Personas one and then seconds? Did we repeat the prioritization and were the results consistent? So in terms of repeating in general, we do try to run things a couple of times because as you know, answers might different and sort of see like do we see a trend line in terms of what variations look like? In terms of. That's actually an interesting use case. It's not a matter of describing them as such, it's just a matter of how many do you ask the models to generate and that you then use to score against different concepts. In this case we use 25 poissons, which is something that you would never do In a real life setting. And I think the interesting thing here is that we against six different so against developers, architects and so forth, we did a virtual survey if you will, or synthetic survey of over 150 synthetic against data points. And then the only use case for that is just to do quick deprioritization of some concepts and to prioritize some others and then those. And then within a prioritized concept understand who is going to be the most critical. Who do we assume based on this model that is going to be the most critical and then if conversations with those real people first. So it's in a way think about it as an additional in between step that actually once you've set up a tool like that shouldn't take you more than a couple of hours, that gives much more of a direction in terms of the future validation that needs to happen. And that then by doing that future validation you can start to understand is this actually valid feedback. So that's sort of the learning loop that you start to build in specific to your context, your industry, your geographical focus, etc. And I think ultimately going through that loop multiple times you will gain a sort of tactical understanding in terms of like right, I can proceed for these things, I can't really trust it for those. And that ultimately allows to find more, more shortcuts or rather to focus your attention elsewhere. I think we've time for one or two more questions.

Gabriela is asking how valid are these tools with us, especially small markets, small niche products. I think it's an interesting question. We'd love to hear like how small you would define it if it's only a couple of people. If you're like in heavy B2B and you're only working with five clients, then then maybe, well I would say always try to ab test it, but maybe it's not worth it. But yeah, I would say as soon as you go a little bigger, it can start or you can start to see results.

Nick, I think from my perspective we do often operate in very like any consultancy or any consultants operating context that you're just so niche that you don't have any priority knowledge about it. I think it's recognizing if you don't have the knowledge, it's just an interesting research and starting tool to get an understanding of what could potentially be feedback from those users and then being very aware that those concepts or those ideas or feedback was generated. But you're going to still be better off than going completely blind would be my point of view. And then Again you sort of building that natural learning loop. I think we had some questions around very practically how the Excel sheet or tool was made. It's just a matter of having the plugin. So that basically means that you can link specific cells to each other using basic formulas and then linking to the API of OpenAI in generating parts of the answer. So that, yeah, it's a bit of a hacked together solution, but that was built specifically for that use case. And then maybe finally let's look at upvoted questions.

Yeah, there's one from Tristan who is asking, yeah, you, a human is good at uncovering unknowns and knowns, but how does an AI do that? I would say the unknowns unknowns most often they fuel into like ideation and getting to new ideas for unknown problems. And large language models are really good at coming up with a lot of going wild on quantity ideas and then some of them will be quality and then I guess the role of unknowns unknowns will. Because you don't really need them for ID generation. That much will come more in the validation. So there it's, it's less of an unknown unknown story if you, if you get what I mean. So yeah, since, since LLMs will take over part of the ideation they become less. Or it's just a means to an end that might become less relevant.

All right then maybe as a, as a final message. Since we're coming up to the end of today's session, we are hosting a virtual summit, the world's largest gathering around autonomous innovation in the first week of December. We'll share the link in the follow up in an invite to join, but we'll basically have a lot of the leading organizations that are using this in how they do their, for example, new product development. So the people from Walmart, Pepsi, Nestle and so forth speaking about what they're doing within their organizations, but also startups and founders of some of the tools mentioned here. So two days full of talks like this one, so I'm diving really deep. So if you are interested in this topic and just based on the question, it feels that there is definitely some, some interest in the audience. Yeah, we definitely encourage you to join that one and participate in the conversation there. And with that, yeah, I want to thank everybody for joining on this Tuesday and we'll, we'll follow up with the recording some of the slides and a list of the of the different tools and hope to see you at the Autonomous Innovation Summit in December.

Thanks to keep the conversation going. So feel free to connect on LinkedIn. Bye, bye, bye.