David Boyle at Qualtrics Summit 2017

Would you please welcome to the stage David Boyle.

Thank you very much. Ahead of us, we've got 30 minutes of fun talking about entertainment, mostly TV shows, but I might talk a little bit about the music industry where I worked before, maybe a little bit about book publishing where I worked before, but that's not as much fun as TV and music, so we'll see where it goes. Someone else can talk to those later outside in the future of Insight. So my topic is going to be how has Insight, how is understanding consumers, understanding businesses, even understanding markets? How has that changed over the last few years? I'm going to talk about some of the themes and trends as I've seen it and then what does that mean for the future? What does that mean for the skills and capabilities that we need, the types of activities we need to be spending our time on, our future? What does it look like? So I'm David Boyle again at the BBC Worldwide, which is the commercial and global wing of the BBC, the British broadcaster. Our job is to take some of the great content created in the UK by the BBC and to find audiences for it all around the world. So big TV shows like Top Gear, which is the biggest bachelor entertainment show in the world, broadcast in more than 200 countries. Doctor who. Is anyone a Doctor who fan in the room? Thank you. Yeah, the applause is ok, you can really tell your audience by whether anyone's heard of Doctor who and that's possible by the way, or whether a good majority of the room have. That tells you a lot about who you've got in the room, particularly in America. Big respect. And then natural history. So we're going to talk a little bit about natural history content. And Planet Earth 2 for example, which is on TV now. It's amazing. Has anybody heard of or watching Planet Earth 2? Wow, we got a good crowd here. I guess it's self selecting because all the people who like the BBC came to this session. I'm going to assume you're representative of the Qualtrics audience. So that's me. And then the. Really there's three topics today and then some recommendations. First, like what is happening with all this innovation in the world? What does that look like and feel like in the media industries? Second, what are all the complexities that that are introducing into our lives, whether that's in the business side or on the inside analytics side? Third, like what does that all mean for how we understand the world, for how we understand consumers and the businesses? And then again, I think there's really one quite simple recommendation out of this, but there's two or three others that will come out, so it's going to be fun. Please save up all your questions. I want lots of questions at the end. I'm hopefully going to leave quite a lot of time to have a good discussion and a debate maybe. So thoughtful, provocative, challenging, Interesting. Get them ready. Okay. Oh, come on. That was lame. Okay. Questions? Yes. Yes. Thank you. After lunch. We need some energy. First on innovation then. The world is changing so quickly these days. We all know it, we see it everywhere and it manifests itself in, for example, the time it takes to build a business. So While it took 75 years for a telephone to reach 50 million people, Pokemon Go took just 10 days. Ridiculous acceleration in the time it takes to go to market. And it's having implications across everything we do, both in our day to day jobs and in the businesses as well. It feels crazy and ridiculous and fast paced, particularly in media. Maybe. If you look at TV for example, what does this change look like? We started off in a relatively simple world, which I'm sure if you asked TV executives from back in the day, they thought that was quite a hard business to run. Then it got a little bit more complicated. We had electronic program guides, we had computers and cell phones doing more and playing more of a role. We, whether it's discovery or sharing or consumption or tasting products. Then the world got more complicated again. We got smartphones and the online delivery of television in more interesting ways. And then it got more confusing and complicated again. You've got cloud delivery of television on demand, cloud recording of television on demand, so you can watch any show you want at any time on any device. Cloud based recommendations across platforms, across providers. Like a radically different world today for television than it was just a small number of years ago. The innovation is providing all this extra complexity. But of course we've got about the same number of staff we had back then, about the same number of hours in executive team meetings. And they're spread now very, very thin. So all the more important to do what we do and help executives to understand the world and make decisions because they're stretch, they're stressed and they're divided across all these different opportunities. The bad news then is that if you thought that was tough, it's going to get way, way worse. Change is going to accelerate. There's a couple of different slides on this. Because it's such an important concept. We need to be fearful of the world today and the pace of change. We need to be ready for what's coming at Us in the next few years, it's going to get worse or better, depending on your outlook. I'm glass half full most of the time. The world's changing because primarily you've got these digital natives. People who grew up using technology, take technology for granted. They're demanding new and interesting ways to consume content, engage with content, share content. But it's not just that. Older people are gradually adapting behaviors to do the things that younger people do. So Netflix is growing in popularity. YouTube's new television offer will grow in popularity and will become pretty popular. Old people take quite a long time to change, but they're changing, increasing the pool of people demanding new services in new ways. Of course, kids are growing up and consuming, so that's adding to the pool again. You've got massive rise in the number of connected devices and therefore the number of people consuming or engaging with content in interesting ways. Plus you've got access to data and code like you wouldn't believe. So new apps and new services and new ways of discovering engaging content are cropping up all the time. And then you've got these digital natives coming of age and this feedback loop where instead of using services and products that we, the older people have built for them, thinking they might like them, they're building their own products and services now. So innovation is accelerating massively. So get ready for that. And then another way to look at it, I think, is to step back in time, back about 5 billion years to the origin of life on Earth. If you look back at life on Earth, about 4 1/2 billion years ago, the Earth was formed. About 4 billion years ago, the first life forms arrived on Earth and then not very much happened for quite a long time. It's quite interesting, the parallel being maybe the television, the polar television, just sticking around for quite a long time. And then you've got this really interesting period called the Cambrian period. Lasted about 40 million years. Blink of an eye in the grand scheme of things. But in this one period, everything changed. I think that's where we are in technology today. It changed in that this relatively simple life form, single celled organism suddenly evolved into all of the basic forms of life that we have today. All of the essential elements and ingredients were put together after that. There's obviously been a ton of evolution, but the basic elements all came together really quickly in this really relatively small period of time. Blink of an eye. And why did this happen? It happened for a couple of interesting reasons for us and for the work that we do. One is that the environmental conditions were right at that time. And second, because all the basic building blocks of life had been put together and had come together and were easy to combine in new and interesting ways. And that's where we are in technology today. We're approaching what you might call a Cambrian explosion of technology, I think, of discovery, engagement, consumption and sharing apps. So, again, life in media feels very fast paced right now. There is so much change going on, it's hard to keep up. I think it's going to get a lot worse. I think we're ready for this huge explosion of new and interesting ways to do all the things that people do with and about media. I think there's immense parallels in every single industry. This is my view from the media industry. There's definitely immense parallels in the work that we do day to day and the tools and capabilities that we use to do our jobs. Get ready for it. We live in very exciting times. So one illustration of that in action for our Show Planet Earth 2, which is airing on BBC America right at the moment. We partnered with Snapchat to deliver the content in a new and interesting way. So in the old days it would have been a TV broadcast. Lovely. Then we ended up with a Netflix broadcast. Afterwards. There were DVDs, there was a nice selection of different ways you could consume the content. But there are so many people consuming content in new and innovative ways. Our partnership with Snapchat is a great example of that. So we took lots of the content from Planet Earth 2. This is a documentary about animals and the world around us. We took lots of the content From Planet Earth 2, content that had never been seen before as well, and edited it into Snapchat friendly vertical video stories, four to six minutes each, and we put them on the platform. Really interesting example of how you engage different types of consumers in your content. Types of consumers who may not be watching regular television as frequently as they used to, and with great success. So it turns out that younger people who are spending their life on Snapchat really interested in nature documentaries. Loads of people watched it. Really happy with the numbers, really happy with the demographics. Absolutely Reached a brand new audience. But it was really hard work to do it. It was really hard work to go back and cut the content, to create the new content, to license all the footage, all this stuff. It was immensely hard work. And that's the world we're in. New and innovative ways to reach consumers are absolutely essential. But it's really hard work to understand that opportunity, to get the business excited about that opportunity and to go through all the different steps you need to go through to realize it. So really, really nice example. And it's live right now, which is doubly exciting. And Snapchat, from what I read earlier, had a very successful IPO this morning. Seems like the price has gone up wonderfully. Everyone's very happy. I'd like to think it's down to planet Earth 2 displaying that Snapchat isn't just about fun and pop and exciting gossip. It's about the real world and engaging consumers in a meaningful way. So well done penguins. So that's how the world feels in terms of innovation and it starts to touch on some of the complexities of it. Let's talk a bit more about complexity. Of course, all this innovation is really more routes to market, more ways to engage consumers, it's more ways to help them to discover content. And it's more ways. It's not replacing old ways, which is what we thought was happening a few years ago. So for example, like linear television, the old fashioned TV that you can read about, TV is dying. It's absolutely not dying. New ways are coming alongside the old ways. So Decipher, who are a UK media analysis company wrote this. I really like it. TV channels are not just alive and healthy, but more interestingly, the role of linear scheduled, like the electronic program guide, is vibrant. There's tons of innovation there which is supporting and helping people to watch live TV and to care about live tv. So we're in this world where there's all this innovation, but it's not killing the old business models, it's changing them slightly, but it's adding to new and interesting ways that people can discover or engage with content. And consumption is growing as a result. If you look at the latest Nielsen numbers for total media consumption, media consumption is growing year on year, thanks in large part to people spending time on tablets and on smartphones. So it's not that the new methods are cannibalizing the old ones, it's that they're adding, largely adding incremental media consumption time in. It's a really exciting time to be in the media. It's really hard and lots of things are changing, but there are tons of new opportunities. So onto our world and what this means for us, there are some of the challenges that we're facing. And the good news is that even old fashioned media companies like the BBC or the record labels or book publishers, thanks to all this innovation and all these digital avenues are now absolutely swimming in data, glorious data. Cloud computing is everywhere. It's really Easy to look after this data and store it somewhere. And data science is freely available. Tons of fantastic open source algorithms and tools and techniques. So even pretty old fashioned businesses now have all of this at their disposal to help them to understand the world, to help them to make business decisions and to help them to understand audiences. So this is great, this is a really good opportunity for all of us in this room, but it's going to be really hard work. And this is what I think that looks like from trying to help media businesses to understand audiences and make decisions about what they should do, whether it's what shows they should commission or renew, what artists they should sign at a record label, which countries they should prioritize for a product. This is how I think that's changed. So our understanding of the world has changed in recent years. And the answer comes down to the same thing we've seen on the business model side. You can't have a massive increase in complexity on business models. You can't have tons of new ways for people to consume and engage with content. You can't have all these new apps and systems and technologies in consumers hands without also having tons of extra data streams, tons of extra business questions and tons of extra tools and techniques on our side. So the complexity in the consumer side is making our lives incredibly complex. Now we need multiple methodologies, multiple data sets to solve any problem. It's a real challenge. Didn't used to be the case though. It's not how it felt just a small number of years ago in the media industry. Used to be pretty easy to make a big difference in the media industry. In TV you had Nielsen ratings which described the linear broadcast of the show, the show on live tv. Pretty much all the show did in the old days. So that captured consumption really nicely. There was some stuff missing. Right. So you'd do some surveys as well. But between Nielsen and surveys, if you could do both of those things and you could maybe join that data together, even then you could pretty well describe a television show. The opportunities, the challenges, the performance. It was what now looks like a pretty simple world. Of course, the world is nowhere near that easy anymore. And whilst plenty of people are still experts in Nielsen or still experts in survey research, that's nowhere near enough now to have impact and to help people to make decisions. So a real challenge, and in particular one thing we've done is to take that survey research, where we still do the survey research, we still do a brand tracker for all of our TV shows, 26 countries around the world, it's really good. It tells us a lot. We still do it, but we've banned a debrief on the brand tracker. We've banned saying, hey, the brand tracker results are back. Here's what it says. Because it doesn't describe enough of the world for us. Now we still have to do it, but we can't have a meeting where we just talk about the brand tracker. We have to talk about all the other data sources that help us understand our brands as well. Now, if you've been working on the brand tracker for a few years, which by the way, I have, along with a lot of my team, it's quite painful to not be able to stand up at the end of the project and do your big presentation. By the same logic though, we can't also have Nielsen reports being circulated. That doesn't describe enough of the world. Now that's just one small bit of the world. We can't have a Nielsen report emailed around, doesn't tell enough of the story. We also can't have the social media team sending social media analytics around because that just describes part of the world. We can't send the CRM report around describing what's happening in CRM world and we can't just send the web analytics report around. They're all interesting in their area, but that area does not describe enough of the business for senior stakeholders and for big decision makers to have. They need to weigh these against each other. It's a horrible challenge. It's a really horrible challenge. But it's the right challenge to wrestle with. We're trying to do so. So we take a look at the brand tracker. For example. We used to have this meeting. Brand tracker audiences debrief how our audience is feeling about our brand. Let's have a big meeting. Lots of people would come, they'd be really interested in it and we'd have a great discussion. So this is what we've banned now. We ban it for a couple of reasons. What you want is that you can't just talk about the brand tracker, as I said. So we now call it brand performance, not the brand tracker. And brand performance is not just about the research, of course. It's about all the first party data that we've got showing transactions and purchases and engagement. That's really important alongside the brand tracker. Also, all the social data has to come into it. Maybe that tells the same story, maybe it tells a different story and loads of other data sources as well. So we still have the meeting, it's just called something different and it brings lots of data together. And of course we can't just talk about audiences like it's crazy to just talk about audiences because how much money we made is really important as well. So we have to include financials. And historically, in my experience, the research teams, the insight teams, have been pretty separate from the financial teams. So financial reporting's over here. That's all handled really nicely. We look after the audiences bit. That's great. That doesn't make any sense, particularly in this new world. You have to bring them both together, which is another immense challenge because none of us or few of us were trained in that side of things. So it's collaboration. And that's one of the big lessons from today, I think, is this immense need for collaboration across departments, across experts, to really tease out the right story, rather than just saying this piece of data says this thing really challenging. We're right in the middle of what this means for us right now. But it's the right thing to do and there is a lot that needs to come together. Collaboration, yes. But there's some basic ingredients for that collaboration in my experience. So for any reasonably sized problem, particularly in the media industries, you know, should I recommission this show? Like, how is my show doing, how our audience is behaving? I think there are really four big elements that I tried to make sure are included. I mean, the first is, especially these days, you have to take into account your behavioral data, your transaction data, how people are actually clicking or purchasing or watching or completing a TV show. You have to take that into account. You can't ignore that. Second, of course, we still need all the research. So that's really, really important part of it. We have to have that included as well. It tells you things that you can't get from the behavioral data. Of course, people who live in behavioral data world will be quite happy to tell you they can give you the answer without the need for research. You still hear that, of course, you need research alongside. I don't need to tell you that. But that's two things you need to bring together these days. You have to bring together data science expertise to do a good job of this. You have to automate some form of teasing out of conclusions or patterns or checking for patterns. It's so easy to do it. You know, you're crazy not to, but also it's so powerful. You will miss things or you will not be able to cover the breadth or depth if you don't integrate data science. But it's again a different set of skills that we traditionally haven't had in research teams. And it works really well when you've got them to hand and you can engage with them and apply them to problems, but if you have them at a distance or far away, it really makes your life difficult. And then on top of all of that, none of that's good enough. You have to be great at storytelling. You have to be great at looking at all this data and teasing out the story, because that's all that the executives need at the end of the day is the answer. The conclusions, the insights, the story. The data just supports that and proves it. So you have to have storytelling skills. And again, in my experience, it's sometimes been a challenge to have those in the team and sometimes been a challenge to have them work across all these different areas and understand these different areas as well. So big collaboration across lots of different types of experts is essential. Some people in the data science world will have you believe that this can all come together in a fully automated fashion. Don't believe that for a second. And not in my experience. The sweet spot in the middle I like to call ocular regression, which is to use your eyes to tease out the patterns and to tease out the story that's actually happening. So a lot of money and time has been wasted in some of the businesses I've seen trying to fully automate some of this and join data. No, all of those things are needed, but it's people at the end of the day that have to tease this story out. Again, it's semi natural for a researcher to think that, but it's not how quite a lot of executives think. And it's certainly not how people with a data science background tend to think. But it does take a person at the end of the day. Yeah. And if you miss any one of these, you're in real trouble. If you miss the behavior data, well, the behavior data lacks all of the why that you've got in the research. The research data lacks all of the what that you've got in the behavior data. If you just have data science, you can't get to scale across the right number of business problems. And if the storytelling isn't there, then you're not giving people the right answer. So really, really, really tough. There's more implications for our world though. If you believe some of that, if you believe some of these lessons, it means we can't lean on full service agencies the way we used to. Right. It means you can't ask a full service research agency to do a project for you and to come back and to tell you what the business conclusion is because they don't have all of that first party data that we said was essential or if they give it to them, you've got to invest all the serious time in bringing them up to speed on it. So where we, in the, in a simpler version of the world where we used to be able to outsource quite a lot to full service agencies, I just don't think we can do that anymore. I think they'll tell us part of the solution, but they can't play the role that they once played. That's pretty scary. Also, it means if your colleagues in your team have just one of these single sets of expertise, they've got some real challenges in front of them. Because unless they can incorporate data science into what they do, unless they're great at storytelling, unless they're great at first party data as well as research data and joining it together again like full service agencies, they're just giving you part of the story. They're not able to tell you what the business recommendation is, the business answer. Somebody else has to do that when they join it all together. That's a real challenge for people in the research world. And finally, which I think no one in this room will fall into the trap. But data science people like to think that they can solve problems all by themselves. They absolutely can't, of course, because they need us, they need research people and they need storytellers as well. So there's some real challenges for a lot of people with a lot of skills and a lot of ways that the world used to work. It's tough, but it's exciting as well. I think in particular, all this big free data about Facebook likes and YouTube views and Twitter tweets and all of the different digital engagement metrics that are out there floating around big and free. I think they're radically changing the entertainment industries. I think they're radically changing how we think about which countries our content is going to be successful in, where the opportunities are, how well we're performing and some of the challenges where we used to lean on billions of dollars of Nielsen data, I think we're increasingly leaning on this when combined with data science that tease out conclusions. But most executives in the media industries don't recognize this at the moment. Most of them still lean on old tools and techniques. I think the new world is a lot about this. Again, Twitter and YouTube and Instagram and all the ways people are giving you signals of interest in your content, whether it's A TV show or a music artist and a record label, or whether you work in politics and it's a candidate or an issue. All of these signals are there for you to understand, engagement and interest and belief in those topics or those products. And when you combine it with data science to tease out what that means, the conclusions from that, it's incredibly powerful. It's changing everything about how we think of it. For example, one application. Let's come back to Planet Earth 2 again. In the old world, what we would have done for planet Earth 2, or for planet Earth in general, or for natural history, to understand who our audiences are, we would have lent on surveys. We would have done wonderful survey maybe in Qualtrics and used SPSS to tease out the segmentations, the understanding of what's going on in the world. We are in fact pretty good at this. We've done a lot of it and it was a tool, tried and tested technique that we'd used a lot. So in the old world that's. That would have been our key way of understanding audiences. But not anymore. Our go to our first port of call for understanding audiences these days is a social media analytics tool called Affinio. And they look at Twitter data, just Twitter data in this instance, and they take each person who follows that brand on Twitter. They take all the other accounts that they follow, their interest graph and they cluster people together according to their interests. Really interesting. So you end up in a fully automated way with these different clusters of people that follow Planet Earth. And then you have to name the clusters yourself. That's the manual bit. By reading the kind of things they follow, reading their interests, it's pretty easy to do. You click on it. And it turns out there's a group of photographers, they're following Planet Earth 2, but their primary interest is the photography, it's the art. That's one audience they're separate from and relatively loosely connected to a bunch of people who really care about news and information. They want to learn about the world and how the world's working and what's going on in the world. It's not photography for them. That's not their primary interest. It's a different type of content we would generate for them. A different type of editorial maybe. And it's different from the people who are primarily interested in sustainability and environmental. They're different from the earth loving people in quite interesting ways. Science and biology and animal lovers. So in a fully automated way, with just manual labeling of the clusters, we've got really interesting conclusions on who our Audience is. And again, in the old days, it would have been spss and survey data. In the new world, it's this free data that's freely available with data science layered on top. So I think everything is changing. Not that we don't need to still do the survey research. We absolutely do. It's still fundamental to what we do. It's just that it's not our lead first port of call. Now, really interesting. Of course, both of them combined is really the answer. So in terms of recommendations then, I mean, the big one for me is about all of this new technology, all of this complexity requires us to really democratize what it is that we do. Insight analytics research. It's about helping people all around the business to be able to access all of this great work, which can be quick and simple now. And it's for us to get really, really, really good at how we do it. So democratizing it across us and then across our businesses as well. So let me tell you what I think that means and where that lesson comes from. I'm really, really inspired by chess. Not the game, but the way that computers have played a role in that game over time. There is a fantastic lesson there which matches my experience in the business world. And the more interesting thing is that we're in danger of drawing completely the wrong lesson from chess and the role of computers if we look back in time. So when a chess computer first beat a grandmaster, the lesson might have been that humans are becoming redundant and machines are taking over, and that decisions can be made better by machines than they can by humans, even in chess. It really might have been the conclusion you drew. It would have turned out to be completely the wrong conclusion. Add a few years, add some technological advancement, and the conclusion is twofold. And it applies to everything we do in our work, day to day. The first part of the conclusion is that you have to partner with a computer to be the best at chess. So people working with computers beat both computers and they beat great chess players as well. So by partnering, you're better than any of the individual parts. So we have to use data and analytics in what we do. It's obvious to us. It's not obvious to every business executive out there. So that's part of the lesson. You have to work with technology, with computers, with insight, with analytics to be really, really good at what you do. You can't go it alone. You also can't let the computer make decisions for you. You need to work together. So that's part of the answer. The next part is way more interesting for us, though, which is that you have to be really, really good at how you work with a computer. So in chess, a weak human, like a not very good chess player with a computer, but with a better process, so they're really, really good at knowing when to lean on the computer and when not to, what the computer's for and what their brains are for and how they work together. Someone who's really worked that out and is very, very good at it is superior not only to a strong computer, but more remarkably, to a strong human, a really better chess player and a computer, but where they haven't quite worked out how to do it. So the lesson is you don't need the best tools and techniques, you don't need the best, most sophisticated algorithms or data. But what you do absolutely need is to get really, really, really good at how you use that data, what you use it for, how you access it, how you run queries, how you run segmentation, how you run research. And that's, I think, where Qualtrics comes in. For me, throughout all of this research is absolutely essential. This role has diminished in my life. It's changed, but it's still absolutely essential to everything we do. And that lesson from chess is why I'm so excited about Qualtrics, which is that you've got to partner really, really well with the tools that you have and you need great processes. And so for me, when I needed a. We do lots of brand tracker and research, I use agencies mostly. But I had one project that was really urgent a few weeks ago, and I used Qualtrics for it. And I could program this quite complicated global study really quickly, really easily. It felt like a chess computer where you're really, really good at it. I'd never used it before, but it's really easy to get really, really good at using it when you need it and when you don't need it. So that, for me, is the key lesson here, is pick tools and techniques, pick processes and get really, really, really good at them. Just like Qualtrics. The lesson being throughout business and in our world as well, it's about taking people and helping them to be better and smarter. It's not about letting computers make decisions for us, it's about using tools and techniques to help us to get really, really, really good at what we're doing. So, again, what it feels like in media these days, unprecedented innovation, unprecedented levels of complexity, which is great. More routes to market like Snapchat, reaching new audiences with those cute little penguins. But that Complexity in the business side is absolutely matched by the same complexity in what we do in our tools and techniques, our systems and our processes. It's a challenge. But if you get really, really, really good at using these tools and techniques and you bring them together, you democratize them across the team and across the business and get really, really great at it, I think there's never been a more exciting time to do what we do. Very exciting time. So thank you very much. I hope we've got some time for questions. Great. I guess we're shouting maybe. Hi.

So you talked about kind of bundling together insights that you might gather from different avenues of data. So research, data, science, you know, the first party sites, especially when companies are moving quickly. How do you justify kind of like pleading those things usually don't happen all at the same time. To bundle those insights together so that they can picture versus sharing it as it comes in. I can imagine that the challenges with having transparency with your executives, giving them what they want, but also kind of the caveat.

Yeah. So the question, to repeat the question for everybody else, the question is like, how do you know what data sources to bring together when you're doing that? How do you know what to leave out because you can't possibly include everything. How are you transparent enough with your executives? And hopefully I've painted a picture through this of both seeing the world clearly. Like you need all these data sources, but also these problems are unanswered and we're wrestling with these in real time. So I'm definitely not a world expert in this. I think the, the thing for me is to start with the question you're trying to answer and then ask yourself honestly, what data do I need to make that decision? Not what do I have at my fingertips. I could run a survey. Excellent, right? No, you can run a survey. You've got that at your fingertips. But you actually need this. And you need this as well. Go find it. Right, but it's to start with the question, work back to what data you need, then go find it. Don't start with what data you have available to you and then try and use that to answer the question. That for me feels like the biggest mistake that researchers or analysts of every kind tend to make. And I think you need to be ruthless as well about, you know, this data is good enough to make that decision. I've got research, I've got some basic transaction data. You know what, that's enough. It's clear enough. I don't need all these other sources. So you've got to be bold and not waste your time with a long tail of data sources that, you know, seem like they're interesting but really aren't going to change the decision. So I think it's really tough. I don't think there's any simple answers. But start with the question, ask honestly what data you need might be the single best piece of advice. One from over here. So how do you staff differently? So how do you staff differently in this new world? I think there's a bunch of retraining of people on the team. So somebody who previously just worked on the brand tracker. I don't think we have anybody who just works on the brand tracker now. They now also take into account at least some other data and have some role in storytelling across multiple data sets. It's a big change in their roles. I think things like Qualtrics are really important, otherwise I wouldn't be here, which is that we've traditionally lent on market research agencies to script and to carry out our research for us. I think we're going to bring that in house now with tools like Qualtrics, so we can be more reactive and more nimble internally. So that's a change. We previously outsourced it, we're bringing it in. And then I think there's some stuff on the data science side that previously we've tried to do ourselves, but the world's got so much more complicated. We're outsourcing that now as well. So, like we used in the music industry, I worked at a record label. We used to gather like Wikipedia page views for all of our artists, Google searches for all of our artists. And we databased it and graphed it ourselves. It's not scalable now. You need Facebook and YouTube, you need Instagram, you need Snapchat. No media company can resource that internally. You have to lean on a partner to do that for you. So some stuff that's got so complicated you need to outsource it. So I think everyone's role is very different now than it was even six or 12 months ago, I think. And I think it's just about saying where is the best place for that work to happen, internally or externally?

So would you say your Net option, staffing, maybe trading program dollars for people?

Well, that challenge is to do all this on a flat budget. So, yeah, so being very, very nimble and creative about how you do this. So, yeah, no, let me just think about total budget. No, because every time we find a cost saving somewhere, we incur a new cost somewhere else. So we're paying for someone to track all the social media analytics for us, which is new. We save money on the brand tracker every year and we've retasked people. So I think it's. I think businesses are the same size they were 30 years ago, roughly, but they're way more complicated and that's our job to deal with that. It's tough. So let's take one from over this side again at the back.

I was wondering if you have discovered any platforms or tools or if you're considering building tools to help you integrate all the data from all the various sources.

That's a quick answer. So am I planning, have I found any tools or have I built any tools to integrate all these data sources together? Absolutely not. It seems feels totally unscalable as a problem. Feels like you just pour a ridiculous number of developer hours into it and you're constantly having to change everything. So, no, each element we're really, really good at. We've got a tool that's great on the social media, we've got a tool that's great on the research, We've got a tool that's great on understanding ratings, TV ratings, like Nielsen. So each tool is brilliantly efficient. Some are internal, some are external. Depends on. Then it's a human, it's ocular aggression. It's a human that looks at all of those and teases the story out. Incredibly time consuming, but I can't see a way to automate that. I'd love to hear of one. All right, then.

So what I'm interested in hearing your feedback on is, so you talk about using all this data to help make decisions. So you made the decision then how do you measure the feedback, the effect of that decision?

And do you? Yeah. So such an important question. How do you measure the feedback? How do you know the decisions were the right ones? How do you know if they had impact or not? And it's something that in my experiences, media companies don't do. That's pretty scary. They make decisions, they don't measure the impact scientifically and they don't look back and say what worked, what didn't. They don't have things like control groups where we roll out a business decision to one region but not another region. They don't do things like make this price change to these products, but not these products. So I can tell the difference. So we've started to do that. Learning lessons from working in politics with some people in the room here, where it was perfectly commonplace 10 years ago, that kind of behavior. After a big struggle, we started to learn lessons from that. So we made a change to the price of our products on itunes. Thanks to some consumer research, we had control groups and we randomly allocated some products to price brackets so we could scientifically understand whether the price change made sense or didn't make sense. It did, by the way. But, yeah, so we've started that hard work, though, just as it was. I watched it be very hard work ten years ago in politics to tell candidates they shouldn't talk to people in a certain precinct because we need a control group to understand the impact. It's hard in a business to say, okay, change the price on these products, but not these ones. So we can scientifically understand the impact, but we're starting and it's very hard work. Is the answer not very frequently done. Can you elaborate on.

So you just mentioned randomly changing prices based off of what? For demographics or like, what sort of.

If you may elaborate on that. Yeah. So on price changes, for example. And we've done this in the music industry and in book publishing and in TV now. So the same logic works. Consumer research tells you whether people think the price is too high or too low or about right. And it pushes you in, gives you most of the answer, frankly, in those industries. In my experience, turns out none of the industries have optimal pricing strategies, which is good. Then it's an opportunity to fix it. So you persuade the business and you make some changes. In music, I won't talk about television, but I'll talk about music. In music, digital albums were too expensive. Single tracks were too cheap. And so we put this price of tracks up, we put the price of albums down, and we watched the impact of it. And sure enough, tens of millions of pounds of profits flowed in. Everyone was a winner. So, yeah, research pointed the way. Real world test and learn showed us whether it was effective or not. That's, by the way, quite a rare example of all that coming together. But it does sometimes question at the back and then I don't know where we are on time. So we got three more minutes, maybe in the brown. Yeah, please, near the back.

What insight are you most proud of? That wouldn't have been possible in the past with a more silent approach.

I think that to come back to one of the many themes like this, free data that's available now can tell you so much. For a TV show, we work with a company called parrot analytics. Parrot, like the bird. They track all these social media signals and they tell us for a TV show in a country on a day how much demand there is from all these free social signals. So we now know, thanks to them, it's completely outsourced. Thanks to them, we now know in between seasons, like, how popular our show is, how quickly it declines when we're off air. We now know in some countries there's more demand than we see in viewing. That is, people care about the show. Doctor who in Korea, there's loads more demand than there is people watching the show. So obviously it's on a channel that's hard to reach. It's maybe an expensive pay TV channel. In Korea, there's a business model challenge getting in the way of that consumption. So, okay, we can go and fix that business model challenge now, because we see really clearly that there's demand there. So I think that free data telling us, all right, people really care. You might not see it in revenue, you might not see it in consumption, but people really, really care. And, you know, they care, and you can see who those people are. That's really changed quite a lot of things. And I think that will change everything about the TV business model. Actually, we're on the cusp of that now. It's really exciting. But again, completely outsourced. Really hard work in terms of a partnership and integration of data and teasing it out for the right stories, for the right processes, putting it alongside Nielsen data and helping the business to understand it's important. Alongside Nielsen, tons of work, but actually completely outsourced because it's no way we could afford to the engineering or the data science capabilities. All right, I'm getting this. Thank you very, very much. It's a real pleasure.

Ladies and gentlemen, David Boyle.