Usign AI to Improve Innovation Experiments

Today, I'm really excited to welcome our guest, Klaus, to Future Foundry Live. Klaus from Telia. We're gonna talk about how the team at Telia, how other sort of partners are using artificial intelligence to improve the way that they methodically sort of test, de-risk, and validate new ideas. And I'm really, really looking forward to this conversation. So Klaus, great to see you, and thanks so much for joining us.

Thank you.

Where in the world do we find you today?

Thank you, Jacob, and thank you for inviting me. I'm in rainy Copenhagen today and all, so it's summer, but it's not really summer-like weather here in Copenhagen yet. So I know that you guys probably have better weather than us over there in the UK at the moment.

Only marginally, only very marginally. Claus, before we get into today's topic, just for those that you won't have met previously, can you tell us just a bit more about your journey in corporate innovation so far, what's brought you to today, and maybe expand a little bit on your role at Telia and what that looks like.

Sure, sure thing. So I'll actually track back, I think it's 12, 13 years ago where all of this actually started. And it started when I was with another telco in a Danish organization, and I was heading a new launch of a new product and service. And with the team, we did everything by the book. We interviewed customers, we did focus groups, we did everything that we should do, and I really felt proud about it. And we launched with a big TV commercial and everybody was applauding on stage and all stuff and, you know, the whole shebang. And then we launched and we looked at the numbers and a week went by, 2 weeks went by, it didn't really catch up. And I was growing so frustrated because we did, we did a good job basically. And after a month we saw the numbers was not exactly what we hoped it would be. It was actually really, really far away from what we hoped it should be. And within the first 3 months we were discussing whether we should draw back and pull the product out of the marketplace. And we did so after 6 months, basically. So that was my first encounter with understanding that there's something wrong with the process we had, even though we actually talked to customers, we interviewed them, we discussed things with them, we tested things on a light basis, but what went wrong? So I was growing this frustration, which led into a curiosity around what was wrong when we actually talked to customers. And I stumbled upon some different things, and the design thinking process was one of them. Lean Startup methodology was another one. But what really hit the nail on the hammer here was the Stanford University video from Alberto Savoia on prototyping, where he talked about how to test things when you don't have tests or don't have built anything. And I delved into all of that knowledge base to understand how could we apply this into the procedures of running products and business development in the telco I was running with, and immediately saw results because what people say and what they do is 2 different things, I learned basically. And this is the say-do gap that is very apparent in most of the worlds. where, as humans, we try to say something of beneficiary of the receiver or anything like that, or include it into some biases and stuff like that, where what we actually do in the marketplace is different. And I understood that we should actually test for the behaviors and not ask for their opinions. And I started to delve into that and try to do that both on business development sides and also innovation cases that we ran in the different industries. And I started a few businesses myself with a really, really successful failure rate on that one. But that's a good learnings and did the same thing, trial and testing. So over the years, I've been testing thousands of ideas and concepts into this process. And together with my company, now Intelia, I've been in 8 years. We started that in a Danish organization, just trying to understand if we, instead of buying conjoins and focus groups, actually went out and built some fake door testing and stuff like that in the marketplace, could we learn about the customers in a better and cheaper and simpler way? And yes, we could. So that was the starting point of that. And around 3 and a half years ago, a new management team came into Telia Group and was looking at the innovation efforts and was looking at saying, we need to have more action, more customer insights into the understanding of how we drive innovation forward. And people coming from the e-commerce side was all looking into the prototyping elements of that. So they established the Telia Prototype Unit. which I was positioned into being the leader of. And I immediately took over there and said, the prototype name might not be the right one, but let's stick with that because what we mostly do is not prototype, but all the stuff before you actually build a prototype, testing things in the market. So I basically, for the last 3 and a half years, been doing that in Telecompany Group. We had 6 countries up until last year. Now we have 5 countries that we're servicing in all markets, B2B, B2C, and wholesale. And we've been running hundreds of experiments every year for different countries and different development teams on commercial assets innovation. And I would say 90% of what we do is more business development than innovation because that's just the name of the game of what kind of organization we are. So, and yeah, so that's where we are at the moment. I have a background of university in strategy and commercial excellence and all that. But really what ticks the box for me is really the curiosity and the experimentation of trying to remove all the layers of biases and try to get the real data on the table to take decisions upon that and design experiments and actually gather those data points in the right way. So that's where I'm coming from, bringing the more scientific approach into business because I kind of think that's a forgotten trait and we kind of in business forgot that this is existing in the world. So that's, I'm trying to push that into the business world again. So that's my own little mission.

And I love that journey because actually I feel like the story that I hear most often is, oh, you know, we spent, you know, 3 to 5 years, a few million dollars developing a product. It was super painful. We spent a huge amount of time, emotion, money on developing the product. We launched it and nobody wanted it. And then we learned about how important, you know, rigorous sort of testing, experimentation, and validation work is. But what I love about your story is that you kind of felt you were doing that anyway, right? But actually what you were probably doing was almost, almost as damaging, right? Because you're moving forward with the courage of the convictions of what you've been told from customers, and, um, that almost gives you, yeah, even more courage to, to overspend and to, um, and to overcommit and to overinvest, right?

Yeah, exactly. And I think this is so true. Everybody that I talk to about our journey and what my journey is, is they all have these kind of examples of from their own lives that they done something similar. They invested heavily in something they thought was right, and on the basis of data, on opinion-based data, and said, we were so convinced that this was the right thing to do. And looking into many industries, the decision data on moving investments into new products and new innovations primarily comes from opinion-based data, which is bananas, basically, when you think about it. So it's really interesting, and I think too few organizations are working in this industry around creating behavioral science-based data points.

Yeah, it's genuinely frightening to hear corporates talking about focus groups. It genuinely sends shivers down my spine. But look, I mean, we've had a lot of conversations previously. We've talked together in depth about scientific method and how we can apply that in, I guess, not just a corporate context, right, but also in sort of startup context. And what's always stood out to me, Klaus, is the real depth of expertise that you've got in sort of testing and experimentation. And I guess not a huge amount has changed in terms of the testing and experimentation stack, probably over the last sort of 10 to 15 years. Artificial intelligence changes that. Why are you so excited about the contribution that AI can make to the experimentation process? And at a really high level, we'll dig into this at a really high level, what do you see as the major benefits?

It's a very good question. And let me start at a point where our mission and what my team is doing is to run 100,000 experiments per year in Telia Company. We are a 2-man plus me team, so we cannot do— we don't have the capacity to do that. But we need to nurture a culture to do that. And without having artificial intelligence tools at hand, this will never be something that we can do. So, so I think basically, putting experimentation together with AI, it will drive a democratization of innovation in corporates in a way we haven't seen before, because we will bypass a lot of links in the structural settings of an organization. We'll open up the, the, the idea space for a lot of different actors in the organization which haven't been, been placed in that situation before. So there's a lot of, lot of gains to be held there. And then we will In the end of the cycle of the next 5, 4, 4 to 5 years, we'll probably see AI agents that will be doing most of the work, will even open up more precision, low-cost speed, and acceleration of growth in most companies. So I see, I see the introduction of AI as a, as a real deal breaker for how to run corporate innovation. And I think it's a, it's a new opening for corporates to run innovation in a new way. if you dare to step into it, basically. So I think it's brilliant, and there's a lot of caveats of risks in there that we, of course, will talk about. But on the outskirts, I'm an optimist around how AI will transform innovation in corporates.

And what do you think is kind of the single biggest problem that artificial intelligence in an experimentation context can solve for corporates? I mean, some of the problems that we encounter Quite often, it's difficult to, in the early stages, even if you're doing discovery interviews, and I was saying customer say is bad data, but customer discovery interviews work. They would say quite hard to recruit customers, quite difficult to build experiments within the confines of their existing technology. stack. And I think the third thing that I would say is, you know, we tend to have kind of like a risk and compliance problem around, you know, how can we sort of test and experiment with things that don't actually exist? But in your mind, Klaus, what's the single sort of biggest problem that leveraging AI technology and experimentation can help with?

I think there's a point in the very near future where we will utilize synthetic databases to run experiments towards in the first instance, meaning that we don't have to confront our brand with the real world. We will run thousands of loops of variants of experiments towards a synthetic database where it's basically a digital copy of human, of a human marketplace. So, I mean, and we built a prototype on that on 1,000 different human beings in a market to run as a synthetic database where we train an AI to be, give us behavioral answers back on the questions we give it. Meaning that you can run, let's say, 100,000 ideas towards the market to understand which one actually performs the best. And then you get a shortlist of, let's say, 5 to 10 that is really valuable, and then you continue on that. And only when you have a core value set where you have validated a few times to say, right now we introduce it to the marketplace, now we actually go into the real faces of the real people because we are so certain that this is not damaging to our brand anymore. And so I think in, in that, but that is a few years in front of us. So, but I think that will be the situation that will be changing most of the innovation structures in corporates, because then all of those fears, as you just talked about, will disappear. Basically, yeah, yeah.

And also, I guess it makes those sort of lofty goals around 100,000 experiments actually, like, much, much more attainable. I guess, to your point, those synthetic databases are sort of a few years away. Thinking about sort of right now, what are some of the key ways that AI is being used to improve the way that you do testing and experimentation at Telia?

Yeah, we do it in a few different ways. So just constructing and designing and design, we utilize AI to do that because most of the designs we do is very much digital. So we test very much with Meta ads or LinkedIn ads and something like that. And just setting up a copywriting script for an AI to continue to do or a creative picture to run from Midjourney or something like that, that we've been trying out. And let me tell you, the difference was that before we had AI in our pockets to help us out, it took us 2 to 3 days to actually design an experiment and be ready for execution. With AI, it takes 30 minutes. So it's a huge difference. So just, that's just the optimization of efficiency to do the task that we do today. And then you can put on top of that, then you can actually automate the process of actually putting it through a process. And when the ad is ready, then it will launch itself into Meta or into LinkedIn to do that. And the only thing we have to do is basically just approve the budget. And so we, we've tried to automate the full process from the start to the end of that. And then you can run experiments. You can run 10 experiments per day if you want to do that, because the time of just developing an experiment is so short. That's one part of what we, what we're trying to do and work with. The other part is more on the creative side of that, because we try to play around with some Google Sheets and ChatGPT to understand, could we make a model where when we ask ChatGPT, we have this issue, this market opportunity or this issue, what would you suggest that we should do from a commercial point of view to generate more revenue? And you ask ChatGPT that right now or Copilot that right now, it will give you plain answers. It will not be very, very interesting. But then we created, we thought about what do humans do when we have to be creative? Well, we always look at an objective with different perspectives. So what if we created a suite of different perspectives in a Google Sheet where the ChatGPT or the Copilot always had to ask and refer back to those realities or perspectives? and cross-reference those to create new evidence and new ideas. And all of a sudden we saw that it created creative ideas, probably not the most bold ideas, but it created new ideas that we didn't think about, which meaning that it was a helping hand to be more creative and innovative around on the idea stage. So we basically have an Excel, Google Sheets setup where we have, I think, 10, 15 different perspectives. That could be the flight industry, that could be the whole hotel industry, it could be service, it could be something different. that the, the core prompt of the question needs to circle back and, and have a few circles around different perspectives, just as we do as humans when we have to be creative and, and ideate. And then it spits out something that could be interesting, and then we work on that as a, as a starting point, meaning that the starting point of ideation is much more informed and richer of, of the, of the creativity than, than before, which shortens and, and makes the ideation better quality, basically. So that's what we do right now. And then we are, of course, looking forward to the synthetic database, but we're not there yet.

Have you found, particularly from sort of a customer acquisition point of view, so thinking about like, you know, sort of say like an online ad test, for example, have you also found that you're able to increase the volume of sort of multivariate and sort of split tests that you're running? One of the Um, sort of things that we're really enjoying at the moment is the ability to not just have sort of 3 to 5 multivariate tests running, manually analyze the data, manually sort of update creative copy proposition, all that sort of stuff, but to be able to kind of run thousands, pick a winner, run 500, pick a winner, run 250, pick a winner. Are you also seeing kind of like a big payoff in terms of the volume that you're able to sort of test out as well.

Yeah, definitely. We're not in the thousands yet, but we had— we had— I think we had one case last year where we tested 100 different— 150 different variants at the same time in multiple countries. And, and yes, there's— you have to be clever about the data pickup and the analysis of it because there's a lot of data in there. But, but all of a sudden you see nuances here, nuances there, combine those into one new and then continue to do that. So the iteration speeds up and accelerates because you get so much insights and data. And all of a sudden you get back with a very informed solution and the conclusion on this is the creative, this is the semantics, this is the text box you want to utilize to create the best result in the end of that. And we've seen, we did a test, I can talk about that. We did a test where the assumption was that the way we as a telco talk about 5G as a technology is very much technology driven in the marketplace. So we basically made 3 different ads in LinkedIn towards the target segment for small business owners. We basically made 3 different LinkedIn ads where the one, the control one, was exactly as we do today, very much tech heavy, talking about 5G as a technology and all that. Then we took the number 2, was also 5G. We didn't mention 5G or technology. We mentioned the feature of what the new 5G network could be able to do on security, which is much, much better on security than the other networks, 4G and 3G. But we didn't mention 5G. And this last one was on sustainability where the 5G network is, it's a much, much lower electricity capacity than the other networks. And then we combined it to see how could we just generate interest. And we put a link to one of our articles on our webpages just to see what kind of interest does these 3 actually account for. And the control one was of course the lowest one because people don't really want to listen to 5G and technology talks and all that. But the security one, we kind of understood that that was what would be the winner due to the circumstances in the environment as well. But it was 500% more effective on creating interest than the the technology one, which means if we just push just some of the feature talks and feature semantics into our commercials and ad marketing, we all of a sudden generate 500% more interest in what we are saying, which is significant, basically. So if you can do that on a larger scale with, let's say, 1,000 different variants to understand which kind of semantic word will actually drive that traffic, then all of a sudden you've optimized your marketing engine quite a lot.

So you talked a little bit about kind of using, yeah, using AI at sort of the ideation level, maybe sort of the customer acquisition level as well. And I'm just thinking about, let's take, I don't know, let's say a, um, sort of landing page test or like a single feature MVP. So thinking about where we're sort of driving and pushing that traffic to, have you used any AI tools to build out those landing pages or variations of? And are you using it at a sort of prototyping level just yet, or is it more at the sort of customer acquisition level?

When we always test for— first of all, we have a sequence of testing, so we always test for interest. If you can't create any interest, we shouldn't go further in the funnel. So, but when we validate interest and we go into desirability, where there's a little bit more skin in the game, people have to sign up for beta group or something like that. And if that also validates, then we go into testing the willingness to pay where we actually have a checkout flow on a landing page often. And what we've done from the start is basically using no-code tools and all that to make those. But we also tried the different AI tools that way you prompt for a webpage basically, and it constructs all of a sudden a webpage. And when we do testing, because we don't hit as many people as we normally do in operations, nothing needs to be picture perfect. So having a construction of a web page that takes 2 minutes from an AI to do that is as good as it gets because you don't want to have 100,000 people visiting that site. You only probably want a few thousand of that people visiting that. So it's okay to have that kind of tools. So yes, we are utilizing some of these AI-prompted versions of how to run checkout flows. And we also have incorporated AI bots and chatbots into some of the flows that we try to make it as real as possible. So, so, and there's a multitude of tools and, and services out there that you can utilize. The, the drawback, of course, is being part of a big organization is that we, we cannot utilize most of them because there's some security levels of, of how much we can do that and what kind of data we, we can, we can influence into, into, into these tools, of course. But I think it has never been easier for anybody to just go out and, and do some of these tests because most of these tools are free. Then you can do the first 5 examples for free, basically. So you can basically start 5 companies and let them run against each other to see which one actually wins the bet.

Which is great for corporates, but also terrifying, right? Because it just means that the barrier, you know, from a competitive standpoint just continues to fall, right? Not just from other sort of, you know, players within sort of your market, but also, I guess, from a sort of startup point of view. You know, we were speaking to a venture studio a couple of weeks ago who are automating as much as possible. In fact, you know, to your point around testing distribution first and testing customer acquisition first, are kind of using a similar model but pretty automated in order to, yeah, test something like 10,000 different business concepts every, every year, which is as frightening as it is exciting, I guess, right, from a corporate standpoint.

Yeah, but that's the way it needs to be, and we will be heading there no matter what. So let's just get there as soon as possible so we get that competition going, because I think this is what we are trying to do internally in our corporate is to be able to run, let's say, 100,000, 10,000 different ideas at the same time in a construction of the entity of the organization. If we cannot do that, then as you say, there will be people outside the organization that will be doing that, and they will be threatening our businesses left, right, and center. So if we don't do that at the moment and try to convince ourselves that this is the path we need to lead, then we are up for grabs and our customers are up for grabs. And that's just the name of the game. I really welcome the innovation acceleration out there because it's more a push towards us as well to say we need to get our shape together and get in that.

What are some of the sort of internal challenges that you've had in, and, you know, I'm not familiar with like the governance model and, you know, your existing sort of technology stack. Some of the, um, kind of pushback, I guess, that we get at Future Foundry from, from, um, sort of partners is we don't have clearance to use this like technology, right? Or our CSIO hasn't signed it off, or our Chief Information Officer, kind of whoever it might be. What did the process look like? Because I guess you had to get get the broader organization on board from like a tooling perspective, right?

Yeah, definitely. And it's still an issue, to be honest. But when we started the organization and the unit 3 and a half years ago, we were basically formed by the CEO and the general management. So there was a buy-in from the high seats at the top table from the beginning, which is very important because otherwise you cannot drive this kind of change into organizations. We had 20,000 colleagues, so we cannot drive that change without having that top table backing. So that was first step of it. And the second step was that I started immediately to write a, basically a manifest, basically to say, and went to a legal guy and said, this text, you have to either write it, sign in blood, or bless it somewhat. Because this is what we're going to do. This is the process we're going to lead. This is how we're going to run experiments. And if there's anything within this text that you don't see is legal, please say it now. Don't say it every time you come and ask for it. Because this is a rubber stamp, basically say every time we go testing, we don't have to come for permission. But this this is kind of the security level that we need to adhere within. If we step outside, we'll come and ask you, but if we stay within the boundaries, we can just go and run. So I tried to clear that out very early in the process because I kind of knew that this would be an issue if we had to come every time we needed to run an experiment and run a case. And so we got the blessing from left and right in the organization. But I think the early start was we have loads of very clever engineers in our organization, and engineers like to tinker and build stuff. And when what I came in to say is that we don't build anything before we get any validation from the market. And that was a complete new thing from, from, from their side. And most of them wanted to, to stay in a basement with a lot of budget and do R&D work where I said, let's do commercial innovation instead of doing R&D firstly. So there was a, I had some, some interesting discussions to start with, let's put it that way. And, but as soon as people saw the data points and how fast we could create those data points and how cheaply we could create those data points, there was an understanding to say, all right, let's Let's do that first before we dive into the feasibility of the technology. But there's still headbutting around the organization. Of course, we are still a small team and we are a huge organization. So I would say the most difficult part of what we are doing at the moment is to reach out to a lot more people to bring this into the culture and mindset because we are running at the moment at a speed of around several hundred experiments per year. And that's around the capacity my team can run at the moment. We probably increase that with AI over the years, but 100,000 will not reach that unless we drive it into the culture. And that's the most difficult part where we also have a huge transformation program. And, you know, large organizations have all of these programs running at the same time. So we're kind of competing for interest here. So I'm trying to be a whiz of internal marketing at the moment, and it's an interesting story.

Yeah. And are there any other sort of challenges, Karsten, that you've encountered? I guess one of the other sort of obstacles, if you like. I mean, for our partners, it's relatively easy, right? Because ultimately, it's kind of like a Future Foundry sandbox. They're not running this sort of internally. So that's relatively straightforward. But one of the other challenges, I guess, is around expertise, right? Artificial intelligence, similar to many other systems, garbage in, garbage out, right? And you alluded to that when you were talking about ideation and the way that you had to train that model to develop sort of better answers. Do you, not necessarily within your own organization, but what advice would you give to sort of innovation leaders looking to sort of level up sort of skills and knowledge? Because I guess that, along with security concerns, can be a big blocker, right, for a lot of teams.

Exactly. Yeah. I would say, I think in this dying age of time, there are so many good experts out there, especially within the AI field, that invest, basically be daring about investing in good people. They are out there and you need to have them on your team. So one of the obstacles I met when I was recruiting all my experts in my team was that some of the best experts within experimentation is not necessarily coming from Denmark or Sweden. They probably live in Italy, US, or Melbourne in Australia, something like that. Yeah. And because we had a structure in our company at that point of time where we couldn't hire talents from abroad in that perspective, I basically missed out on 90% of the talent pool in the world, basically, because what we do is so specialized. And you have probably, you know this as well, there's not many experts within this space that you can just go out and get in and recruit because this is not something you teach in MBAs in universities. It's something you have to learn about over time. So the skill set is quite required and it's not everywhere. But with AI, there's much more talent pool out there. So go out and invest is probably the best thing that you could do.

Yeah, it's definitely not in the MBA module. I think that might be— yeah, I think we might get synthetic databases before we get AI into MBA module. But I might be, I might be proven wrong. I guess one of the challenges that a lot of clients will be having is that is quite a, you know, the competition for that talent pool is pretty fierce, right? From, you know, sort of global technology corporations through to, you know, startups who are experiencing, um, um, very successful sort of fundraising rounds because, you know, there's an AI sort of badge on the, on the front of it. So that's kind of something to, to, to watch out for. I guess a lot of people kind of watching this might be, um, I guess by virtue of the fact that they've joined us today, might be relatively, um, sort of early on in their journey, just starting to explore how they might leverage AI through their experimentation stack or their experimentation sort of process. What's your kind of biggest learning from over the last sort of 12 months? And what advice would you give those leaders who are kind of just getting started out here, Klaus?

I think one of the learnings that I've had in my process is that when we started, we were just jumping on all the tools that we could, basically, just to get familiarized with it and learn it and teach it and try it out. But quite fast after that, I had some serious discussions with our legal teams about what kind of data we put into these models and tools and all that. So, And so be, have the legal team on your side when you start that journey because you can very fast burn your fingers on that. If you start just to have a little bit of data points, which is not public data into this, you are doing it probably wrong if you are a large organization. That's at least for our sake. That's, but then right after that, we kind of learned that if you don't have the collaboration with the commercial organization in what we do and the innovation people around what we do, we're just gonna be a small little echo chamber sitting in the corner where we build stuff and try stuff on our own. And it has to be commercially interesting to actually, to the organization before we can actually generate real value. And instead of trying just to tinker with AI tools, try to actually make something that can actually generate value with the AI tools from the beginning because you then learn and create value at the same time. And because AI is moving so fast, you cannot sit for the next 3 months and tinker away with it because then the train has left the station. So, you need to both be able to implement it in some way of your experimentation stack and do some serious work with it and put it into places where you actually have live testing going on at the moment. Otherwise, you just miss the opportunity.

Yeah, finding a real use case is important, isn't it? Otherwise, everything feels pretty sort of hypothetical. Places that, that kind of might be, um, a, a good place to start is thinking about, you know, a lot of partners will do, um, they'll run a lot of sort of customer discovery interviews, you know, not to see whether they're going to buy the product. We've already talked about what a bad idea that is, but to understand, you know, what kind of pains people experience, what kind of problems they experience, you know, what, what is it that we should kind of go and solve for As a jumping-off point for ideation, I guess a really straightforward place to deploy AI technology would be to say, take these transcripts, take these kind of raw interviews, this footage, and help me develop some problem statements, right? That feels like a pretty safe place that most innovators could play.

Yeah, exactly. And find me the markers where there's overlapping elements and where the opportunity spaces are. Because most of the AI tools, especially the generative AI that we see today, if you actually feed it with context, context, it understands that context quite well and can actually do some, some pretty good analysis on top of that. So, that example is a very good example of doing that. Another example could be when you do run your experiments, make an AI to analyze your data points from, from, from that analysis to understand where do the caveats that you don't see hide themselves, because they are there. And sometimes you have your own biases because you looked at thousands of experiments, data points, and see, well, this and this and this will be interesting But you might overlook something that, that could be very, very significant for your business if, if you apply something like an AI to analyze, do another analysis on that, that as well. So there's definitely opportunity spaces in, in there. And then, but then there's always the, the biases in, in the algorithms themselves because they, they are built into that. So you have to be aware that maybe what they are, the AI is saying is not really what you want to hear, but it's what is built into that bias. And I think that will be a general issue for a long time with all the tools that is running on AI platforms at the moment, that there will be biases that we need to understand and we need to adhere for when we do the analysis and discussions with the AIs.

Thinking about some other challenges that listeners and watchers will be wrestling with is, what does it mean for me and my role in the future? corporate innovation, corporate venture building has had a tough few years. There's a slight post-pandemic boom when we were all trying to think about how do we deliver, create, capture value in this new world. It's definitely tough. A lot of people are being asked to do more with less. AI helps in that context, I guess. But how do you see the the human role of sort of corporate innovators evolving as AI technology becomes more prevalent in testing and validation? I guess most of the teams that we work with, 75%, 80% of their time is spent, you know, designing, um, sort of experiments, you know, kind of conducting those, analyzing the results, making recommendations around sort of pivot, park, persevere decisions What do you think they're likely to be doing in sort of 5 years' time? And how will that look, that role?

Interesting. I think in 5 years' time, I think we will have AI tools, AI agents that will basically be running themselves. So I think, and this is also what I do in my team and what I've told my team is that I think within the next 3 years, we need to kind of make ourselves unemployed basically, because I think the tools is there to actually generate So here's what I think is that within the next 2 to 3 years, we will still, as humans, be in charge of the AI. So we will prompt, we'll make the decisions upon that. But after that, when AI agents start to collaborate on top of that, they will be prompted with a strategy and a purpose for a company. And then they will start to formulate and ideate and develop and test and launch stuff into the market and experiment things in the market. Themselves due to the platforms they will be able to work upon. And then, then all of the things that you just mentioned will probably be machines doing that because they will be that advanced. And I foresee that within the next 4 to 5, 6 years, basically, that will, that will slowly start to happen. And then our role as entrepreneurs and innovators will need to shift because there still needs to be people that will be correlating data so that the AI still needs to be trained in the right way. So there still needs to be people that understand the process of how to generate it and design experiments and all that. But there will be, there will be a, I think that this is my belief. I think there will be a situation where within the next 5 years, we will, you and I will have, let's say, 10 AI agents working for us as digital workers doing small tasks. And that will evolve over time to be able to work together and deliver the full process of innovation. Whether that will be creative enough or not, Let's see. But if an AI agent can test 100,000 ideas towards a synthetic database within a second, I guarantee you that will be smarter than whatever you and me could find out of during a walk.

And I guess what we're talking about is a great democratization for experimentation, right? Do you think that we will see more corporations embracing business development, hypothesis-driven experimentation as a result of this technology? Or do you kind of think, look, if you're not on the bus, you know, the technology isn't going to make you more likely? Do you think we create more corporate experimenters, or do you not think it really has an impact on sort of take-up?

That's a good question because it can go either way. I hope it will because it's going to be more easy for corporates to actually add into or jump into that situation where the assumption testing method is going to be used more because it's going to be so easy. I think we will see also the big consultancy will build some of these things into their strategic process and strategic work. I foresee that the Jarvis of the AI is basically— I foresee something like that, that McKinsey or a Kearney will bring into companies where they have their entity, the company has their entity, and meet somewhere in the middle. And then the consultancy entity will have some sort of ideation, innovation, growth engine that will look at the data points from the organization and out in the market and say, from the strategic point of view, this, this, and this is interesting. By the way, we tested 100,000 cases in each of these areas. This is a use case you should go for and implement because that's the most valuable one. That's what I see can happen. Whether that will happen is still up for grabs, basically, because there's still a big gap between what we do on innovation and experimentation and the strategy part of that. Because if those 2 merge in the future, then it will be much, much more clear because the strategy piece is so close, much closer to the top tables, where the experimentation part is not as close to the top table, at least for now. So we need to put it up there in those terms. But if we can merge a tool, a strategy of experimentation into one, then you have a good chance that this will be the prevailing way of doing business in the future.

That's fascinating. And I think, well, you know, one of the things that I'm really, really excited about for all of the kind of, you know, corporate partners that we work with is that, you know, the big learning curve, I think, particularly with sort of venture building and kind of scaling new ventures is And with any kind of form of corporate innovation, really is like it's a power law asset class, right? In that you know it's the one winner in a hundred that's going to pay for the ninety-nine failures a hundred times over, right? And I think what I'm really excited by is the ability for AI technologies to to to kind of fill the top of the funnel. Still see way too many corporates placing you know a small number of very large bets. rather than a large number of small bets. And I think that this kind of changes that dichotomy, right?

I totally agree. And it's going to be interesting to see what the venture capitalists will actually do in this space, because if they get a tool in their hand where they can bet throughout the process of investing much, much clearer than they do today, will they still then go for the statistical way of investing in 1,000 cases and one pays for all of them? Or will that actually just kill off all the darlings that don't win the table on the validation bet? It's, it's going to be an interesting time.

Yeah, yeah, really interesting. Um, what in terms of, um, in terms of kind of technologies that you're kind of really excited about, we, we kind of talked about sort of, um, synthetic databases. Do you think that synthetic users are there just yet? Here are a lot of corporates talking about sort of using synthetic users to generate sort of early-stage insights. I remain relatively sort of skeptical on that. But what's your view on sort of synthetic customer research?

Yeah, I don't think it's ready yet, to be honest. I've been dipping my toes into some of the tools and trying to understand them a little bit. Maybe it's just 1,000 Indians sitting in a workshop in a call center in India doing the work. I'm not sure whether this is a technology that is there yet. It probably will in some years. It will probably be there. But then again, I think the core of where I think synthetic databases needs to go is that it needs to be a behavioral synthetic database, because if it's still just built on opinions, it's the same rubbish basically that you can utilize for that. You can still use it for direction and understanding and insights and inspiration, all that, but you cannot use it for decision-making on what will actually drive the business forward. So it needs to— synthetic database needs to be built on data that is behavioral science-based. based. And to be honest, companies like the social media companies are positioned very, very well to actually generate this because they have that kind of data. We have that data as well, but we don't have that huge amount of data points as Facebook have or LinkedIn has.

Yeah. Also Reddit, right? Like Reddit.

Exactly.

Where we spend a huge amount of time like, you know, testing new business ideas and Yeah, I think in their sort of IPO filings, they talked quite a lot about how, you know, this partnership with OpenAI could be a significant sort of value unlock for Reddit. And I think they're absolutely right.

Exactly. And maybe it's also the reason why Musk actually bought Twitter and renamed it to X, because that's the same. If you have the database of human behavior, you can utilize that to a really, really large amount of significant value when we go further into the AI acceleration. So there's a real gem in there and only the big tech companies have that gem, of course. That's the pity of it.

I mean, it's the only sensible reason that I can fathom for buying Twitter.

Yeah, especially for that price, at least.

And we've kind of talked about sort of the, the near-term clouds, but let's kind of, you know, like predictions are always kind of wild, right? And, and sort of, you know, difficult to, difficult to do. What do you think the corporate— what do you think the impact of, um, artificial intelligence on the corporate innovation landscape is likely to be over the next sort of 15 to— the next 15 to, to, to 20 years? If you, if you think, you know, sort of long term, maybe towards the end of our careers?

Yeah, I think it will. I'm a true believer that AI has a positive impact on what we're going to do on this earth and the society we're going to leave for the children that comes after that. So I know there's a dark side of all of this as well, and there's some risks in there, but I truly believe for the corporate world, there will be quick winners and quick death in the organizations around the company because Let's face it, there will be industries that haven't been primed for digitization yet, and they will now be met with AI infusing everything around them. And basically, a teenager in a small dorm room can actually do businesses that is far more intelligent than their businesses have been for the past 100 years. So there will be, I think there will be quick death, and there are probably gonna be consolidation left, right, and center in industries that we haven't seen. And I can't really fathom how fast it will go. I think the— yeah, it will, it will be significant. And I think from, from my own industry, the telco industry, we will always prevail by having the, the ownership of the spectrum of which we broadcast the telco signal over. But maybe that also will be obsolete at some point of time because we'll have satellites or nanotechnologies or whatever quantum technologies that will drive something new into, into the business side. And all of a sudden we are left with an industry that is, that is, hadn't, they didn't follow up on that. So I think it's now, now is the time for corporates to invest in understanding how they bridge that gap into that future and be that one that don't have the quick debt, but actually be the one that leading the change. And to be honest, the big old companies often don't think like that. So it's again to having that, having the CEO that runs and operates the organization very efficiently, but at the same time having probably a duo CEO that runs the innovation and growth area at the same level. So having that level of understanding that the future will be significantly different, it needs to be driven into boardrooms, it needs to be driven into top table rooms where I don't really see that happening at the speed that it could be and should be at the moment in the organization that I'm running around with. So there's a big risk coming up for old organizations that they the Kodak blockbuster moment is probably getting nearer for a lot of organizations because AI will be able to do things that we only can just dream of.

Klaus, that feels like a really, really good place to finish. Thanks so much for joining us today. Your insights, as ever, have been incredible. Thanks so much for sharing those practical examples with us. And yeah, if you'd like to reach out to Klaus, you can find Klaus on, um, kind of LinkedIn. I'm sure, you know, I don't want to speak for you, Klaus, but happy to answer.

Please just reach out.

Yeah, awesome. And, um, yeah, as for us, um, next episode of Future Foundry Live should be coming up in, um, September. Kind of topic to be confirmed, but you will, um, hear from myself and Taylor on coming along to that. Um, in the meantime, Klaus, everyone who's kind of watching live, I hope you have a fantastic, um, summer when it finally arrives in Copenhagen and, and London.