From the weeds to the stars - how and why to think about bigger problems - David Boyle

So it's titled from the Weeds to the Stars and it's about how by applying our skills to just. Does that sound okay to you? Sounds really funny to me here. Like it's strange. Echo. As long as it sounds okay to you. All right, cool. Step back a bit maybe. Thank you. It's about how if we apply our skills to just slightly different business problems in slightly different situations, we can have a much bigger impact. And I've seen this through my career. It's been an important part of how I've done what I've done. So I've tried for 17 years to help organizations to use analytics to help to drive evidence based decision making. I've learned loads of lessons on the way. I've screwed it up loads of times. So I want to share with you some of the lessons learned, some of the regrets I have for not having had a bigger impact at certain points. In the hope that you can learn some of those lessons, I'll save you some of the pain and in the hope, the overall presentation that we can inspire some of you to just think a little bit differently about the problems you're tackling or maybe how you're tackling them. And this is inspired by many years of coming to this event. I first came in 2012. I've been to the event in three countries now. I love this event. It's so inconveniently timed for me. I just got off a plane this morning and I've got to run after this. I'm moving house. Like the worst timing in the world. But I really wanted to be here. I've never seen such a group of brilliant minds in one place. I remember the first year, I was so blown away by the talent that was there and the quality of the work that I saw. But at the same time I was a little bit concerned by quite a lot of things. This is a theme every time. There's so many great minds being applied to problems that they just applied themselves to a slightly different version of that problem or they done it in a slightly different way. The impact could have been multiplied massively so from someone who cares a lot about business impact as well as the puzzle solving element, this is the inspiration for me. Not that just like solving puzzles for the sake of solving puzzles isn't an awesome way to spend our time. God, we need you if that's what you do. And I've certainly spent a lot of my career doing just that as well. I worked in corporate strategy helping like mergers and acquisitions, forecasting company growth and that kind of thing. For a while I didn't care about which company bought which other company. All I cared about was solving the puzzle. Don't tell my past employer that. But puzzle solving, great in and of itself. But there came a point for me, and this is like a real turning point in how I thought about how I spend my time. There came a point for me when I started to ask, like, how could the work I do have the biggest impact for me, that came because just one day I got bored of doing corporate strategy work and I decided that I wanted to change the world. It's a little bit naive and simple of me, but I woke up just feeling like I wanted to make a difference. And I think a lot of people might be feeling that way in the last few months. I found a job that was a direct parallel to the job I had in corporate strategy, but in a slightly different company just down the street doing market forecasts, market analytics, impact analytics. But instead of in the corporate world, in the non profit world. So I was using exactly the same skills. Instead of buying companies, I was helping large donors and grant makers to invest for social good, for charitable outcomes. Just simple application of skills to a completely parallel problem, but having a much bigger impact in the world also, I thought. And one lesson here I think is to constantly question, is this the best thing I could do with my time? Is this the biggest impact I could have? So I was pretty happy for, I don't know, 18 months maybe. Then I read a quote in the newspaper by George Soros and he said, I spend $450 million a year in my philanthropy, but the biggest good I could do, it's not that at all. The biggest good I could do was, would be to change the government. He was talking about the Bush Kerry election in the US at that time, I think, oh, right. So helping a government to stay in power or to get into power for that four or five year term far outweighs any philanthropic good I could do by helping a few million pounds to go to a slightly better charity. Lo and behold, we were coming up to an election in the uk, the Labour Party. Tony Blair was in power and it would give him five more years. It would be five more years of health care, education, foreign aid, community investment. Five more years of that would far outweigh the work I was doing in philanthropy. So a constant lesson here is like, question, are you having the biggest impact you could have? And I thought I was, but I wasn't. So that one quote calls me to quit my job Go work at the Labour Party. Try to help them to win the election. They won the election, by the way. So five extra years, a historic third term for the Labour Party in the uk. Absolutely. No thanks to me. Because other lessons learned is that even if you're doing quite brilliant work, it sometimes still doesn't have any impact whatsoever. And I want to share some of why that happened as well. There's lessons there for everybody in every organization. So my genius work at the Labour Party was that instead of just telling everybody the same thing about what we were going to do as a government, we could personalize messages. We could tell you a slightly different prioritized list of our promises based on your zip code and whether you had kids and your age than the one that we gave to you. We could help everybody to have their own personalized view of what labor would do for them in a third term. Of course, I was mainly interested in the analytics that live behind this and the optimization problems and all sorts of other things. But this is back in 2005, it was not a solved problem at all. And also, no one was very interested at the Labour Party in doing this. They totally didn't understand why this might be a useful thing to do. And although I managed to convince them to let me launch the website. On the Labour Party website, this is before Facebook Connect, so you had to actually type in your age and your gender and all this stuff. We couldn't just look it up, but never mind. Although I managed to convince Labour Party to let me do the work and to put the website up, no one really visited it. So I had to say, how else could I have impact? How else could I help this election? And it turned out that the biggest way the Labour Party communicates with voters is not the web in 2005 at all. It was through direct mail. We would send hundreds of thousands of letters out to people in their mailboxes. And that was the biggest communication method. Okay, great. How about we take the engine that lives behind this and apply it to direct mail optimization instead. How about we mail everybody a personalized letter with exactly the kind of promises and the achievements that would work for you based on your zip code, your demographics, everything else. They weren't doing this at a time they were sending the same letter to everybody. I kid you not, the same letter to absolutely everybody in the country with the same wording. Even for 2005, that was like a little bit backwards. So, all right, try to persuade them of this. They're absolutely not interested at all in doing this. They totally don't get that. But we managed to persuade them to instead let us do 10 variants of this letter. So instead of sending one, let's do 10. So we run all sorts of cool clustering algorithms on the population and great predictive models. We've ranked all the policies according to all the demographics, and we come up with not a personalized letter, but with like 10 variants. Like massively compromising on the power of the analytics. But you know what, trying to have impact, trying to make a difference. So you got 10 versions of the letter, we send them off to the approvals process. And it wasn't until after the letter had gone out that we came back and said, hey, so can we see like, what happened in the editing of those letters? Like, where did we get to? Sure enough, There were like 10 levels of approvals process here. And by the time the letters got to the end of the process, they were all averaged back to be basically the same letter again, completely ruining the point of it and completely failing to have any impact. The lessons here for me, again, no matter how brilliant the idea is, no matter how cool the analytics are that you're doing, no matter how much the analytical side of the project works, we see so much of that here. If you can't get people to understand why you're doing it, if you can't get them to act on the back of it and do things differently, we may as well not have bothered with all that analytical work. What we needed was a different culture in the organization. A culture that was excited about personalization and data and the possibility of analytics. I was way too junior. It was way too early in my career for me to change the culture. But what I learned since is that this is my advice to everybody. If you need to change the culture in the organization you're in, you actually don't have to be in and out of the boardroom. You don't have to be working at that level to change the culture. Like everybody who's doing great work, who's got a positive attitude, who's generous in their work and shares the work that they do, everybody contributes to changing the culture in your organization. So I look back and the time definitely thought that I'd completely failed. Now, knowing what I know about organizations and how they learn and evolve, I'd like to think that although my work had no impact on the election, maybe I made a little bit of difference to the way the Labour Party think. And maybe they're a little bit cleverer and more data driven now, thanks to that although they're not winning elections these days, so quite possibly not. So constantly think about, are you doing the right thing? Are you working in the right area? Is an adjacent area or an adjacent problem a bigger impact problem for you? Think about organizations that have the right culture to let you have the impact that you need. Let your work make a difference. So I quit the Labour Party. I wasn't sticking around five years for the next election. I moved to a place with an awesome culture. I moved to America where everything's bigger and better and data driven. I worked in American politics for a few years. And in America, lo and behold, of course, you've already worked out that you can personalize direct mail. This doesn't need to be invented. And I go to D.C. and there's an army of brilliant people I'm surrounded by in Democratic political analytics, an army of absolutely brilliant people doing amazing work on personalizing direct mail, on which doors get knocked, on what messages get given to which voter. It is amazing, I tell you, I've never been around such an exciting, vibrant group of smart analysts all working together. This is a proxy for it. But you're not all working together. It was amazing. Really, really amazing. So go find somewhere else that that works. But keep asking yourself, are you working on the biggest problems? Are you working on the most important problem? Is your skill being best utilized for the outcome you're aiming for? So just look at this last election. Like, how much money was spent on direct mail? Like, not that much. How much money was spent on TV ad buys? Shitloads. If you're doing analytics and analytical optimization, and if your objective is to try to make a difference in the election, an army of brilliant people working on direct mail or door knocking or phone call optimization is fantastic. But if you're not working on the biggest problems out there, where to spend your TV ad money and on what messages and in which communities. And that group of analysts, that smart brain trust wasn't working on that problem. That was media agencies. And it was largely driven by skills and expertise, not data. If you're not working on the biggest problem, are you having the impact you could be having? And again, like, how many times in this story have I made that mistake of thinking I'm working on the biggest problem and then not, and then go to the next one and not. And still I never did get to work on Media Buy. And that group of people I worked with never did either, I think, to this day. But what they did do is to get a lot better after I left politics A lot better at direct voter prediction and modeling on a person by person basis, which messages, what likely vote are you going to cast, et cetera, et cetera. But I think they still didn't ask themselves, even with media buying, still didn't ask themselves, are we working on the right problems? And I think there's a problem in politics about where smart analysts spend their time, which ultimately led to be a big factor in the last election outcome. And that's that whilst they were still optimizing at a personal level, whether you're going to vote, whether you're persuadable or not, whether you're going to support our candidate or not, they weren't aggregating that up to ask themselves about audiences and audience behaviors in aggregate. Which audiences do we have as a campaign, which audiences do we not have, which audiences does our opponent have and which audience does he not have, and how are those audiences trending? It's a really different question. It's a really different way of looking at data than saying at a personal level, shall I send you a piece of mail or knock on your door or target an ad to you, or not? Which ad shall I target to you? They're really different problems. And the Democratic Party in particular, obviously Trump didn't do very much analytics. The Democratic Party in particular got really, really good at that grassroots optimization problem. They didn't get very good that macro problem of like, where are different audiences heads in this election and what's going on? An example of this working like, brilliantly that I use a lot in the corporate world is Affinio. They analyze mainly Twitter feeds in this case, but followers of different brands or candidates or topics. They cluster them according to their affinities, according to the other accounts that they follow, to allow them to interpret even quiet and dormant people on Twitter and segment them into similar attitudinal and behavioral groups. So you can start to say, who is my audience, who am I talking to, who's following me, who is my opponent? Which groups does he have? Where are the crossovers, where are the overlaps? How is this trending, which just wasn't done in democratic politics. I would argue they got really good at solving a particular problem. They didn't ask themselves, are we working on the right problems? So never get tired of asking yourself that question. I want to use some examples from the music industry now, where after I left politics, I went to work in the music industry and EMI Music, a record label with the Beatles and Katy Perry and Coldplay and others, we built this amazing Data set. The music industry was largely undata driven at this point. This is 2007 or something. Largely undata driven. But we built this amazing data set. We interviewed more than a million people in detail around the world on every possible topic that we were interested in. We were going to use this data set to make all sorts of interesting business decisions. But that wasn't how the music industry worked, as it turns out. This quote from Richard Branson is fantastic. I make up my mind about whether a business proposal excites me within about 30 seconds of looking at it. I rely far more on gut instinct than research. That's how the music industry work. That's how many businesses work, actually. Right? So how, if you've got great data or if you're doing amazing work on great data, how do you have impact in a world like that? How can you make a difference rather than just solving great puzzles? How can those puzzles actually contribute to business decisions? Let me give you an example of why that's much more complex than it might seem and why I think Richard Branson's right in what he says. I'm an analyst, I'm a data nerd. I think he's right. This is why we built this amazing data set in the music industry. And one of the things we said we could do is predict the success of a song by playing it to loads of people and analyzing their reactions and interpreting that, but ultimately predict the success of a song. One of the A and R people had signed this new band. He was incredibly excited about them. He'd also signed Coldplay back in the day, like this, top of his game, signed this new band and he asked me, can you use your tools to tell us which song should be the first single from this band, which song should be the first song that anyone heard in the world from this band that would launch this new band's career? So we researched lots of different songs, lots of different countries, loads of interesting clustering work on this huge, complex data set to kind of tease out different audiences, perceptions and reactions and the likely performance. It was really cool work. You'd have been proud. So I show him the results and one song outperforms all the others. I show him the results and he says, okay, well, that's not the first single then. I am furious, right, Because I've done all this hard work. It's bloody good work as well. And he's going to do a Richard Branson on me and just use his gut instinct instead? No, he explains to me, the band have only just come Together, they're not ready for a big hit. If they have a big hit, they won't sustain it, they'll fall to pieces. As a band, they need practice, they need more time. They also need an audience. They need a niche song that builds an audience. Once they've built an audience, then they can go in with the hit and capitalize on that and grow their career. They can't just go in with it. There'll be a one hit wonder. So what I realized, despite fantastic analytical optimization work that I think I've done, the problem is something different. The problem is actually more like a multi goal optimization, like success, longevity, impact, strong audiences. It's a multi goal optimization problem though without clear goals, constraints and trade offs. So you can't actually write an algorithm to solve this problem. What's more, it's a complex system with all these moving pieces of the business and the radio stations and the bands and the personalities, the interactions, they interact in pretty sophisticated ways with feedback loops. And it's purely observational. You can't ab test this very easily as well. So you're relying on observing the world and unable to tease out causality. It's an incredibly complex problem. What seemed like a very simple problem. I naively went in excited that I had solved this problem for him. I picked the first single, I totally hadn't. And Richard Branson was right. It's much more about Gut instinct. But informed by the fact that he knew he had this great single in his back pocket that he could pull out one day, I'm quite sure that influenced all sorts of decisions that he made in the months and years afterwards. Knowing that that was there ready for him. Quite sure it affected his investment in the band, the confidence that the band had, how they gelled as a team. So my work actually did help, I think. Even though it didn't do what I thought it was and it didn't solve the problem, I thought I was trying to solve the lesson for me. There again, if you're trying to have impact with your work, you're trying to make a difference, not just solve interesting problems. Is to give people insights about songs or about other things that they can carry around in their heads when they make those gut instinct driven decisions. Give Richard Branson facts and tips and conclusions and insights that you've learned along the way from your work. Don't give him the work, give him the insights from that work. And when he's in a meeting looking at a proposal, he's got that in his head and he can use that as part of his gut instinct driven decision, that is the way you can have impact at these kind of levels and particularly in creative industries. But I would argue in all industries as well, which is really different from how we usually wrap projects up. What are the key insights that people can carry around? Somebody who would talk eloquently on this topic if he could be with us today is this chap who I had the pleasure of working with in the music industry. Or my team presented to him some of the work we'd done on clustering audiences and predictive models around his future success. Really, really cool from that massive data set. But it was boiled down of course to these simple insights in particular. There was this one cluster, didn't use that word, of course. There was one type of consumer. There was an immense untapped opportunity for Snoop dogg back in 2008 or whenever this was, which is the time they're called gadgeteers, like very digitally savvy, always on their phones. We painted a picture of like how immensely exciting this opportunity was without using, I think, any statistics, probably some graphs. I always insisted there were actually graphs in the presentation even if the end user wasn't going to use them. But immense data set, really complicated data science on top of it, Very, very simple conclusions. And he absolutely fell in love with these conclusions. He kept interrupting the presentation saying like, we got to get them gadgeteers. We've got to get them gadgeteers. I'm going to get them gadgeteers. If you look at his behavior in the coming years, he went crazy on social media. Sorry about that. And all sorts of digital partnerships with cool tech companies. Like, he really got it in his head how important that conclusion was, that insight was. And then all the, I'm sure gut instinct, slightly hazy decisions he made following that was in his head and he was using it to make decisions. We had impact with that work. The clustering data science, quite big data, made a difference in a Richard Branson type way by giving him insights that he needed to carry around. That's one immensely powerful trick for making sure your work has impact. And it means doing things slightly differently. It means not necessarily asking yourself what's the optimal clustering algorithm that predicts or describes sales. It's not about the optimal solution mathematically or scientifically. It's about what's this balance between something that's scientifically, I don't know, good enough versus something that actually is insightful. It means something, it's understandable. It can be carried around in the head of Decision makers. That's a trade off that we don't make very often. We optimize. And optimizing analytically often means it's less useful to the end user and a compromise would have much bigger business impact. This is a huge lesson and it's really hard to learn it again and again. And it's hard to get people to not give the optimal answer. But this way, Snoop carried that round in his head and made decisions on that every single day. And the social media universe has paid the price for that ever since. So for me, this is about designing not for optimal solutions, but designing for the people who are actually making the decisions. Hopefully you don't have to influence Snoop Dogg and Richard Branson, but think about your leadership or your decision makers and what is it that would help them to carry insights around in their head. There's an immensely helpful parallel here, I think, between chess. I love this example so much. It is how I think we need to think about data and organizations. Not that chess will computers will outperform humans. That was the lesson many years ago. Of course, it's not the lesson anymore at all. Not that computers will soon make decisions and humans will be redundant. If you look at the world of chess now, the understanding of the role of data and algorithms is much more sophisticated than that. Two key lessons that I think completely apply to business. One, you absolutely have to partner with data if you want to make good decisions. You absolutely have to. And in chess, teams of humans plus machines dominate even the strongest computers. So that partnership is absolutely essential. So that's Snoop Dogg using the algorithm. Second, though, partner well, get really good at doing this. So a not very good human like chess player with a computer and a better process, they're really good at knowing when to use it, what to use it for. Was superior to a strong computer alone and more remarkably superior to a strong human, better chess player and a computer where they hadn't quite worked it out. So the optimal algorithm was actually less good if it didn't fit into the process, if it wasn't usable by the person or they didn't know how to use it. It's not about optimizing the best chess algorithm, chess computer in this case, it's about the best relationship between the computer and the person. That's optimizing for Snoop Dogg to carry it around in his head, not optimizing for the mathematically optimal solution. I think that's exactly how we think about data in terms of making an impact in the world. And the phrase might be augmented experts. There was a phrase that was doing the rounds of human in the loop computing, which I think is nonsense. Alistair Kroll's adaptation of that is computer in the loop humanity, which I think is a much better way of thinking about it. It's about people. People make decisions. Whether you're Snoop Dogg or Richard Branson or in my earlier life, whether it's mergers and acquisitions in the corporate world, it's still people making decisions. How can data help them to make that decision is the real question, and helping them, not solving it for them. So I want to end with a couple of calls to action, maybe, or like opportunities. As I see it, if you're a brilliant mind doing fantastic work, how could you have a big impact? What untapped opportunities are there out there that I think are going to come through in the coming weeks and years? I think they're both around this theme that we are drowning in data these days. Whether you're Snoop or Richard Branson, whether you're in a record label or a TV production company, you are drowning in data these days. This is a perfect illustration. This is a periodic table of marketing signals. Isn't that exciting? So each one of these is a signal that you get if you work in a marketing organization these days. I probably can't read it from here. So cost per point, cost per reader, cost per something else, whatever. View, view time, average time on site, bounce rate, interaction. What have we got up here? Cost per order, Media efficiency, revenue. That seems like that's important hidden away in the middle on the right conversion, media efficiency. Every one of these is. It's a kind of interesting signal, actually something you probably should be paying attention to. If you run a marketing organization. How the heck is anybody going to have any kind of opportunity to look across this, understand what's really going on in their organization? Any one of these could be quite a significant driver of performance. This is just marketing. Whether you're in sales channels or distribution or logistics or health care or. The parallels are everywhere these days. Along with digital complexity and new opportunities comes this. How do we help marketing or decision makers in businesses to navigate this? This is a problem that can't be solved with anything other than brilliant data science analytical minds helping to understand the root cause of behavior across these and prioritize these metrics, maybe helping to alert to issues and challenges. Or it's another brilliant example from a different industry, or mixing them all together and telling you in some kind of combined way how performance is going, which would require immense analytical power. Here's another company alongside Affinio I mentioned earlier, Para analytics is the other company I'm really excited about. They work in the TV space mainly. And what they do is to recognize that in the TV industry for TV shows, there are all sorts of signals of a show's success and engagement and interest out there in the world and nobody can look at all those signals. Same problem we just saw. So, for example, for a TV show, consumers are doing these things. They're file sharing on P2P platforms. A TV show, that's a signal something was interested in the show. There's photo sharing sites, video streaming sites like YouTube and that lot, microblogging sites, Twitter and social media and Google searches. There are hundreds of signals, or probably hundreds of platforms, each of which has potentially hundreds of signals of demand in this case for that TV show. And what they did is to say, hey, TV industry, stop trying to monitor these yourself. Everyone's monitoring one or two or three or four, doing not a very good job, rewriting API calls all the time, databasing it badly, interpreting it horribly. Stop trying, stop everyone trying to do that. Why don't we do that for you? Why don't we aggregate all of these signals for every TV show in every country on every day? Why don't we do all the data science work for you to interpret the relationship each of those has back to demand for the TV show and why don't we aggregate it up to a simple single number? Demand, demand for a TV show in a country on a given day. It's amazing, unprecedented simplicity of like one number for each TV show in each country. From all of this built on this wealth of insight from our digital complex world that we live in. And while some TV companies have done some of this themselves, like the scale at which Para analytics is running, this is something that no individual company can compete with. So I wonder, this is TV only. This is such a big need in politics, it's such a big need in movies and music. It's such a big need in every industry. I think this is the future of high level KPIs in most businesses. This is the only brilliant model of it I've seen. So this is one big opportunity, I think, to make a difference. They're similar but different from that marketing periodic table thing, which is a hugely complex thing businesses are facing. So I think tackling some or both of those are some of what I think are the huge exciting opportunities going forward. But look, whether you're on the mathematical puzzle solving side of this, which I've definitely been and I want to keep one toe in that water or whether you're on the kind of how can I have the biggest impact side of this? Which I sit more on that side now, but I personally try to stay somewhere in the middle. That's me on an average day in the office of trying to have the impact I can and trying to solve some puzzles along the way. I would urge you to keep asking yourself like are you having the biggest impact? Kind of slightly adjacent question or slightly adjacent application of my skills have a bigger impact and am I solving for mathematically optimal solutions or am I solving for something that decision makers can use actually have a difference in the world? That's my call to action and hopefully a little bit of inspiration. I hope you've got some questions and comments. So the question is, are British political parties as good now as American political parties are? They're definitely not. Whether you're in business or in politics or in nonprofit world. I've worked in all of those on both sides of the Atlantic. Everything's bigger and better here. Just with scale comes sophistication. So there's a natural, you know, there's hundreds of people working analytical politics here. The market could sustain like 20 in the UK. The innovation spend here far outpaces it. I think there's a cultural challenge as well in the UK around investing in this type of activity. So I don't think that battle got won. So two good reasons. That said, I think what you see with me being excited about para analytics and about Finio is that it's increasingly the solutions to these problems are increasingly not like internal data science teams. If you're on Netflix or Uber, then you have perfectly sufficient data sets to sustain and engage brilliant minds. If you're most other businesses or organizations like the Labour Party has an election every five years, how are you going to sustain and attract and engage talent over that time? I think the solution is increasingly contractors or outside agencies or startups. I think in that sense you see a lot of them playing both sides of the Atlantic now and around the world as well. So they probably have some of the tools even without the culture and they certainly don't have the staff. But I think startups, smart individuals, contractors are where I lean for most of the clever work I do over the last few years that used to be in house teams and now it's switched completely. That's a really good question. So if you serve an individual business unit with your data analytics or insights, how do you help the rest of the wider organization to benefit from that. They can. But how do you do that? So my experience is that most of the time that doesn't happen. That data and particularly insights are incredibly siloed in an organization. So my experience is that it's a huge opportunity. I think it's also quite hard because you need people with time and a mandate and permission to actually go do that as well. And when you're, you know, nobody has free time these days. Everybody's stretched too thin and could do with 10 extra people on your team. You know, nobody's got time to just go knock on the door of adjacent divisions or other or other countries in a multinational company and share and learn. But when I've done it really successfully, it's been therefore a global team with leadership that is excited about that outcome and is willing to invest in it, but invest time and money in it. And I've seen it work really, really, really well in some companies, like in EMI Music and the record label. I think we successfully shared everything across every division and across every country. It was this. It's never going to be repeated. I'm never going to make this happen again. But it was this amazing collaborative experience where tools, techniques, methods, insights were shared. In Democratic politics, that was the case as well. It was brilliant. There's an organization dedicated to doing just that, actually across Democratic political analytical groups. But in other companies, when I've tried to do it, I won't name them, but it's been this immensely hard slog. And even when people are excited about sharing and learning and collaborating together across divisions or across companies, it's all too easy to go back to their day jobs in their different country or in a different division and carry on as they are. So despite inspiring collaborative events, it's just not sustained day to day. So I don't think there's a magic answer. I think it's an immensely exciting opportunity, I think with hard work. But it is hard work. It can be hugely rewarding, which I've seen. I've also completely failed to be able to do it, despite trying my hardest in other environments as well. So, yeah, I think it's a huge untapped opportunity. Particularly I think the best example for me was in politics. And it started off with a lunch group. Like 20 people would get lunch. Different Democratic groups, the environmental groups and women's rights groups and Democratic Party, they're analysts, single analysts. They all get together for lunch, do presentations, share and learn. It became conferences and it became an organization. It became a staffed organization with a newsletter and all this stuff. And that was the mechanism by which the Democratic Party, not any one institution, Democratic Party, got so much smarter, so quicker. Republicans before that were the leaders and centralized data and analytics and elections. We were nowhere with this bunch of scruffy kids doing analytics. All uncoordinated. But through loose coordination and innovation, we far outperformed the centralized bureaucracy of the Republican Party in just a few years. Actually, I think it's what you said which is sharing across adjacent use cases and divisions and in the Democratic Party's case, women's groups, environment groups and all sorts of things coming together. So huge opportunity, but quite hard work, very well phrased. How do you recognize if your product is not good enough to be sellable? Two minutes. And it's one very good question. And I should say in politics, like, candidate trumps everything. If you look at like knocking on doors, it's statistically shown through controlled experiments to have like a 2 to 3% difference in the election result. Candidate is like, I don't know, 30 or 40% difference in the election result. Right. Like candidate trumps everything. Literally. That was funny. Come on. Candidate trumps everything. Okay, it's not funny. I get it's not funny. Your real question was, yeah, are you messing with the product by doing optimization? And that was the big concern of the Labour Party. It's like, no, this election is about the economy. We don't want to write to you about foreign aid and to you about education. It's about the economy. So we built that in though. That's very simple. Right, so the headline of all of the 10 variants was always something to do with the economy. It was actually a slight variant on the main pledge. But the second level down, you could vary that a little bit. And by the time you get to the third or fourth paragraph, you could vary that a lot. The truth is, there were a hundred policies that we had laid out meticulously. On my desk I still have a copy of the policy handbook. It's this big specific policies laid out. There was quite a lot to choose from. It's just a matter of not describing it differently, but just giving you different highlights. You might care about foreign aid, but somebody who's on the poverty line, worried about health and education doesn't want to hear you're spending money on foreign aid. They want to hear about how their local school's going to get better. So I think we manage that trade off. I think it's totally manageable. But yeah, the candidate can't get around the candidate sometimes. Is that ok? One more question. I didn't get told I was. One more question. Yeah, great question. Our audience is getting smarter for this, or do we have to get smarter at communicating to our audiences? They're absolutely getting smarter. Is my experience in the you know, when I first started, I mean, I can remember flying across the country to Phoenix to sit next to somebody and to get them to let me look at some data they had on their computer. Data was a scarcity and people didn't share it. And now, of course, data is everywhere. Everybody has access to it. Everybody gets that it should be used. They just don't know how. So it's definitely about helping them. But the appetite, enthusiasm has definitely, definitely changed, in my experience, so significantly over that time. Thank you. Thank you so much.