Super Crunchers | Ian Ayres | Talks at Google

Hello everyone, and thanks very much for coming. Welcome to today's Authors at Google event and it's a great pleasure to have Ian Ayres with us today. He received his J.D. and Ph.D. from MIT and is also the William K. Townsend professor at the Yale Law School and a professor at the Yale School of Management. He's the author of seven books and over 100 articles, most recently super why Thinking by Numbers is the New Way to Be Smart, which he's here to speak with us about today. Writing about the book in the New York Times, David Lenhard wrote, Ayers point is that human beings put far too much faith in their intuition and would often be better off listening to the numbers. The best stories in the book are about Ayres and other economists he knows, whether they are studying wine, the Supreme Court, or jobless benefits. Ayres himself is one of the statistical detectives. He has done fascinating research. Ian will be speaking and then taking questions from the audience. So if you have a question, please wait for the microphone so that your question can be immortalized on YouTube forever. And after the event, he'll be signing books. In case anyone's interested, the books are on their way and they'll be here shortly. And what we'll do is be distributing from the front on the way back. So thanks for being patient with that and with that, please join me in welcoming Ian Ayers to Google.

Thanks a lot. So I'm here to talk about super Crunchers, which is really about how a new type of data crunching is helping people, helping businesses and other non businesses make better predictions. I know beyond a shadow of a doubt that the one woman that I've searched for my entire life, I found, the person I found in Lance is incredible. Eharmony helped me find the love of my life. @eharmony.com we match you based on the deep dimensions of compatibility essential for a meaningful relationship. Without any question, I will love you for the rest of my life. And so I could not be happier than at any point to be able to ask you if you would do me the honor of becoming my wife. Okay. Can anybody predict how she's going to respond? I can tell you how often, exactly. Well, you know, the fact that they put this into a commercial gives us one indication of what she's going to say. We've all seen lots of eharmony commercials, and they're in some ways so standard that it's easy to jump over the fact that eharmony really is a very different type of matching service. The traditional matching Service asked you who you wanted to go out with and would try to match you with that type of person. And eharmony is something different. And it's not exactly what we normally think of as the wisdom of the crowds where you go out and you ask people what they consciously think. Boy, the old wisdom of the crowds would say, who do you think Ian would match well with? Eharmony does something different. They take the wisdom of the crowds, but it's kind of an unconscious wisdom. They go out and they run regressions on thousands of married people and they try to find out what personalities work well together. Those 29 dimensions of personality. They basically give their clients a personality test and they run regressions on which personalities work well with each other. And they may end up matching you with people that you never would have thought or that the crowd never would have thought you would have done well with. Pandora.com does something similar. They use machine learning to look at the structure of songs and its preferences. And I put in that I like Bruce Springsteen and they come back and they tell me that, that the Bodines produce similar music. This is a song called Idaho. The geek like me never would have gone without their help. So Super Crunchers is about this rise of database decision making. Data driven decisions are changing the field, field after field. Why is it happening now? Here's my nephew Marty. Many of you may have seen this T shirt. It's kind of a test for you to understand what this. Many of you already maybe seen this T shirt before. The back explains it. It's really. It's not are there 10 types of people in the world, but there are people that understand binary and those that don't. The reason that Super Crunching is happening now is that it's become radically easier to capture and store and access information. I love being at Google, you know this idea. It's because of you that this is happening. And I actually think that Moore's Law is important, but that Kryder's law doesn't get enough attention. That it's become so much cheaper to store information makes it possible to do all kinds of things with data that we weren't able to do before. So what's new about super crunching isn't particularly the techniques. There are some advances, important advances in machine learning, learning, but it's the size of the data sets that we're thinking now in terms of terabytes and petabytes and that it's the speed and scale of the impact that in the old day data driven decision making Took months and years to implement and impact decision. Now it's taking days and in sometimes hours. So I mentioned how eharmony is using regressions to figure out who will be your true love. This gentleman right here is named Orlie Ashenfelter. He's one of the great statistical economists in the world. And he has this passion for wines and he went out and he ran regressions. This actually is the output of one of his core regressions. Trying to figure out, trying to predict the quality of Bordeaux. And it turns out that with just three variables, he can do a pretty credible job of figuring out the quality of Bordeaux vintage before anyone has even tasted it. And the core thing that a regression does, many of you will know this, is it comes back with the weights and the sign on these coefficients tell you the direction of an impact. One thing that, by the way, is that you can tell here. Sir, can you help me play a game here? So if we have a higher average growing temperature, what happens to the quality of Bordeaux? It says that the quality goes up. Exactly right. All right. And indeed, a convenient truth is that the last decade has been an excellent decade for Bordeaux because we've had the highest growing temperatures in the Bordeaux regions that we've had for a long time. That's not to say everything else is noise. Well, there's the fact that now, by the way, this is not a staggeringly accurate prediction. But. But my main claim here in this book and in this talk is that statistical predictions do way better than humans. And who's the great human that you're running against? It's Robert Parker. And you'd say, well, of course, with Bordeaux, how could it be that a statistical prediction could do better than actually, why don't you just go out and taste the wine? But it turns out you can't taste the wine for three months. And then when you do the first tasting is putrid. And that Parker, on the base of this early tasting, makes worse guesses than this simple three variable regression. And by the way, an iron law of resistance is that the traditional decision makers do not enjoy it when they have to compete against the quants. And Parker wasn't amused by this. But actually, if you do a test of on Parker's predictions, he started to pay more attention to these variables. He's getting more accurate. And his greater accuracy is he's mimicking Orly's predictions much more than he did in the past. So this idea of using historical data to predict things, it's not just about wine or eharmony. Or Pandoras. People in this room are using it for all kinds of things. Personalized search features that do a better job of predicting what you want to see. Cap one is the leader of using it in upselling. Now I'm not a great fan as a consumer to upselling, but I like it a lot better when they upsell me with products that I actually like. I prefer to see ads that I actually am interested in. Even more importantly, this type of historical data crunching is really causing revolutions in evidence based medicine. Better predictions about what kind of diagnostics help people out. And to me, one of the most important and under emphasized parts of this regression analysis is that the very output that tells you the prediction, it simultaneously tells you the precision of the prediction. If there's not a lot of data supporting a prediction, the regression will still make a prediction, even if it's noisy, but the output will tell you that it's not a very tight prediction. If there's a lot of data supporting the prediction, it'll be tighter. You can see this in one of my favorite consumer websites is Faircast. So farecast crunches terabytes of data on historic airline prices and it makes a very simple and clear prediction about whether it thinks that the fare on an airline route is going to go up or down between now and when the plane takes off. So if you wanted to go from New York to Los Angeles a couple months from now and you don't know whether you should buy now or wait, it'll crunch terabytes of data and tell you to wait because it's likely to go down. And by the way, this isn't a good example. I'm a great fan of the traditional wisdom of the crowds. There are circumstances where the crowd and their conscious predictions do a better job certainly than individuals, but. But this is a place where the traditional wisdom of the crowds is not going to do a very good job. The crowd can't make good predictions about what's going to happen to airline prices. It's too hidden to the crowd. Better to crunch terabytes of data. And what's really especially nice is it not only makes a prediction, but remember, it tells you the precision. With this type of prediction it's right 80% of the time. And other times it'll make a prediction, but it might only say it's confident 60% of the time. And so it tells you whether to wait or not. And it'll even stand behind its prediction by offering you some insurance. So the big thing though that I want to emphasize is Moving on to a second tool, and that's what I'm going to spend most of my time here today is talking about. Randomization. This is such a powerful tool. It comes, of course, from medicine. If you want to know whether a drug works, you divide your patients into two groups and you assign them randomly to either a control that doesn't get the drug or a treatment that does get it. And there's such a beauty to these and simplicity, a transparency to randomized tests. At the end of the day, you just calculate two averages and you compare whether the averages are statistically different or not. It's not as much, trust me. Statistics and what's happening, what has happened for decades in medicine is coming to business, to government, to lots of fields. And I think we need to establish much more. An ethic that anytime you have an initiative to do something online or offline, one of your gut instincts should be, is there a way that I can do a randomized test on this to really see whether the initiative works? Here's a single element that was tested over@monster.com can I ask you to play with me on this? Which one do you think was more successful at getting employers to sign up for Monster.com, the lower or the top element? Which one do you like better? I would say. Excellent job. See, so you got better intuitions. I like the bottom one. I like the curved lines. Graphically appealing, but you win this contest. Employer spent 8.3% more. I don't actually know why, but in some sense, you don't deeply need to know why. It's so cheap and so quick. The speed and the scale of the impact are very quick when you do these randomized tests. Joann Fabrics a similar story that emphasizes when the cost of randomized testing on the web goes to zero, you can start testing crazier things. So one person had this suggestion. Why don't they test whether if customer bought two sewing machines, they would give you 10% off on both of them? And this was not in their top five or top ten ideas because first of all, sewing machines isn't a central product for joann's. And who needs two sewing machines? It sounds like a dumb idea, but it was a tremendous success. And by the way, I shouldn't overstate here. In combination with other things, this played an important role in increasing the revenue per visitor by 209%. But specifically, after the fact, you can always tell a story. Turns out now they believe that this crazy idea started turning their customers into salespeople. That one person would call up a friend and say, hey, I'm buying a sewing machine. Would you like to buy one too? And you can now it makes sense to me, but no one could have seen that going in. And you want to give yourself some freedom. You just put it up there for 12 hours and see if it takes off or maybe for 48 hours. And so it gives you the opportunity to do real time optimization. By doing these online marketing tests, Google AdWords allows you to do it automatically. You can shift Your put up two different AdWord campaigns. It automatically starts shifting you toward the one that has the higher click through rate or the higher conversion rate. You can also, some people don't understand, but you can also test whether you should have a single campaign or whether you should have a segmented campaign. And that itself can be subjected to randomized testing. So here's an example from musiciansfriend.com where they started out having a one size fits all landing page, which usually had a bunch of guitars in this one area. And they decided to test whether having the central element here depend on where they got the handoff from, what the Google search was that people were interested in. And they just ran an experiment against a one size fits all versus an affinity or a contingent element page. And they again had dramatic results. One thing you can see is they didn't have a very big lift with regard to guitars. But that's because the control itself was dominated by guitars. But on the more esoteric things like drums, which normally didn't get much play, you got a 52% increase in the click through rate. And it's not just in click through. Overall, going to a segmented approach increased revenues per visitor by 15%. Or to give one more example here, this comes back to eharmony. Its central product grows out of regression. But eharmony does tons of AB testing with regard to its website. This was its standard landing page. They want to maximize visitor registration. That's really important to their stickiness of their model. But they wanted to test something. They said, well, we're forcing them to register with us before we give them very much information. What if we give them just a little bit more information? We'll give them a few links where they can get some more information about what eharmony is. We'll give them just a slightly bit more copy here. Will that do a better job of actually increasing our registration rate? Theory can't really tell you how much information to give people before you make them register. You just have to go out and test it. Well, interesting results here that in the United States less information won out pretty clearly that if you gave people more information, they abandoned in mid course. Better to get them to register and then you have a relationship with them and you can convert them later on. But here's the real payoff. This was a result from the United States, but in Canada, just the opposite result. And the reason again after the fact makes some sense. They don't run TV ads in Canada and people don't know what the product is. And so giving them a little bit more information is something that actually increased the amount of registrations that you got. And so you can use this type of testing to find out not just whether something works, but you can identify sub segments where the information works. By the way, I don't have a slide on this, but it's kind of interesting. EHarmony had long internal hand wringing about whether they should include Google AdSense ads in the right column of their site. Oh come on, we have a clean site. This will be off putting. And they finally did a randomized test on it and it increased the length, the stickiness of their site. Clients were sticking longer with eharmony if they added on the Google Ads and that was enough to stop the hand wringing because they found out that it was adding value, it was adding making their clients happier. So this big push that I want to say, and basically when I'm talking to corporations, if you're not doing some randomized testing, start, and if you're doing it, start doing more of it. I have yet to find a business that is doing too much randomized testing. And this is not in tension with doing traditional number crunching on historical data. But when the historical data runs out, you can start making your own data by doing randomized testing. You can test behavioral and demographic segments of consumers. Again, that's the example of Canada versus the United States. And finally, once you do randomized testing, you can actually do run regressions on the output of that to see if there are hidden underlying factors of what was going on in your treatment group to define different treatment effects within the test. The other thing that is very important is this movement toward randomized testing. And let me just stop to say what's going on with Web Optimizer is incredibly important. This democratization of making it really easy for people to run all kinds of randomized tests and get usable results on their own is quite important. But one of the things that you're going to be able to do with randomized testing is go back and forth Use it to see if it impacts your offline sales results. So, for example, you can do a randomized test of a web promotion and you pick 50 cities and you randomly say, divide them into two groups. We'll run the web promotion to computers in 25 of the cities. We won't run it on the other 25. And you see if the physical stores start having a spike in sales so that you can use the web. And here you can finally, then instead of requiring that the Google Ads have an impact on your, on your online sales, you can use this to find out using randomized testing to see if it has an impact offline. Other ways you can do this is to see if the content tests that are getting a lift online. You can start using that as a basis for doing some of your direct marketing mailings. And while I've been emphasizing, and in some ways this, well, I've been emphasizing how online randomized tests are so cheap to do, I really want to emphasize that randomized testing is not just about the Internet. It's incredibly cheap to do on the Internet. But you should be doing randomized testing in your HR department on questions of inventory. Here's a specific example, and let me disclose. I'm a founder of a very small startup company called stick.com which is trying to help do randomized testing on commitment contracts. So this is a very simple economic idea. We'll let people put some money down, they get the money back if they achieve their goal. The money is forfeited to a charity if they don't. Simple economic incentives. Does that help people? Well, Dean Carlin did a randomized trial in the Philippines to see whether it helped people quit smoking. And again, he took a bunch of people who wanted to quit smoking, flipped coins. Half of them he gave an option to enter into commitment contracts, half he didn't. And the beauty of this is you don't have to know much to trust the power of this. Of the people that got the commitment contracts, 40% of them succeeded in quitting smoking versus 5% in the control group. It's not a silver bullet, but it is a substantial increase. And this could happen. You could use commitment contracts on weight loss, on smoking, on virtually any dimension in which you can collect data on whether people succeed or not. I've not just founded this company, but I take this number crunching seriously. I'm like the hair club for men. I'm also one of its clients. So I had a BMI, a body mass index, over 25, on December 26th of last year. And I entered into a commitment contract where I put $500 a week at risk. So basically, Weight watchers cost you $500 a year. And randomized study shows that you lose four or five pounds at the end of a year for $500. I have put at risk $500 a week, and so far, though, it's cost me nothing. If I ever go above this line, I lose money. And you can see on a daily basis what's happened. And let me tell you, I've been a yo yoer in the past. The fact that I lost weight is not so substantial, but the fact that I've stayed below here. So it's worked in the Philippines on smoking. It's worked with me. The real question now is this is not enough of a data set to figure out whether it'll work generally, but you can start doing randomized testing with companies like Google on whether it would help people be healthier. And randomized testing is, interestingly, a place where government is ahead of business. It may be the fact that government is often bipartisan and they can get bipartisan agreement on this neutral method to test. And you see if it's. And the political parties have different views of what's going to work. So they both think that the other. They think the other side's ideas are going to fail, and they think their ideas are going to win. So they are willing to throw them up to testing because they think that they'll get political success out of showing when the test results come in. One of my favorite government examples, just hugely important, is something called Progresa, which was run in Mexico. They wanted to see if giving contingent food to poor mothers, if their families went for medical checkups, would that help them be healthier? And they would give contingent cash, again, just to poor mothers if their children attended class 85% of the time. So a little bit like stick. It's another economic incentive to do better. And how do you know whether it's going to work again? This is not rocket science, but they got themselves to do it. They took 506 villages, poor villages. They randomly assigned them into two groups. Half got the progressive payments, half didn't. And then you just sit back and you see what happens. Well, the progressive boys attended school 10% more. The girls 20% more. Serious illness and anemia were down 12%. And for me, the real kicker is that the kids in the Progresa villages, after not even three years, were a centimeter taller. And this is really huge, you know, a centimeter in three years is healthy kids grow faster. And that itself shows a huge public health impact of doing this. This little randomized study has had huge impacts on the world. In 2001, Mexico expanded it to all of its poor families and now it's covering more than 2 million families. It has a $2.6 billion Mexican budget. And progress alike programs because of this one randomized study have now spread to 30 other countries. Indeed, Bloomberg is trying to bring it to New York City. So this idea of doing randomized testing to figure out what works, it should start at home. The very title for super crunchers came from a randomized test that I did on Google Ads. So I went into this project just loving the title, the End of Intuition. I thought that sounded really cool, but I wasn't sure. And so I ran a randomized test on search words such as data mining and number crunching. And you probably would have gotten this right. But again, I was wrong. And super Crunchers had a 63% higher click through rate and that was good enough for me. So this idea of using statistics, of using randomized studies to make predictions, very powerful. But really one of the biggest questions that often confronts businesses and decision makers, is it better than an expert with 20 years experience? And the answer is, yes, it's better. Okay, here's an extreme example. It's my own field of law. They ran a test on the Supreme Court predicting Supreme Court decisions. And in one Corner they put 83 legal experts. Many of these had clerked on the Supreme Court and had gone on to be law professors and expert litigators. And in the other corner, they ran against this statistical model that just had six variables and incredibly crude. None of the variables were the specific issue involved. So you'd think, of course, that the. You'd think, of course, that these experts that know what the case is about have got to do a better job predicting what the Supreme Court, whether it will affirm or reverse. And what are these six variables? Well, since we're sitting here in California, they're variables like the case was from California. I don't know how many people have been following the Supreme Court. But the Supreme Court hates California. Okay? Really, really hates. And more specifically, the ninth Circuit. So cases from the ninth Circuit were much more likely to be reversed. And so what happened? Of course, the model won. The model predicted 75% of the court's affirmation or reversals. The legal experts only 59%. The model was particularly good at predicting the votes of the swing justices, Justice O' Connor and Justice Kennedy. This is during the 2003. And to give you an idea, this is actually the flowchart for the prediction algorithm for Justice o'. Connor. And look up at the top, it starts and it says is the lower court decision liberal? If the answer is yes, it predicts o' Connor will reverse and that's the end of the flowchart. Okay, so this is what bait the experts. And it's important to emphasize because why humans do so badly is they don't assign the right causal weights to the really important variables. Of course they know that o' Connor is more likely to reverse if it's a liberal opinion. Of course the experts know that the Supreme Court hates California, but they can't bring themselves to put enough weight on that variable. They fall in love with these esoteric influences that aren't important enough. And secondly, experts and non experts, humans, we tend to think that we are doing the right thing. We're damnably overconfident. It's hard to get us to put precisions on our predictions. But if you ask people, experts to figure out how much confidence, how tight is this prediction, they are remarkably damnably overconfident. And in fact, the more complicated this is an important error that we make. I think that humans think that the more complicated the problem, the more we need an expert. But I think it's just the reverse. The more complicated the problem, the less likely that an expert is likely to beat a statistical algorithm. That if you have so experts are actually pretty. Humans are pretty good at one factor problems. If you shake up a Coke can, it's easy for a human to predict what's going to happen. But if you have a process that has more than 10 underlying influences, that's where humans go wrong. And the statistical, the machine learning algorithm is likely to do well. This Supreme Court example is not cherry picked. That there have been over 200 examples of expert versus machine contests in all kinds of fields, business, purchasing, medicine, trying to predict whether a parole decision, whether a parolees is going to commit a crime or not. And in example after example, there are only about 6% of the time that the experts do a statistically better prediction than the humans. And by the way, this Is with a 5% significance rate. So it just might be randomly that they're doing better. It's much More likely about 40% of the time the machines are doing a statistically significant better job of making predictions. So what's left for us? Well, humans still are vitally important. We're the ones that generate the hypotheses. This book is not about the end of intuition, but you just have to be willing to put your intuition to the test. And by the way, let me break off and just say that you can still hypothesize. Let me throw a big hypothesis about what Google should be doing. I'm so excited to be here thinking about this. I think that you guys should become should start offering compensated viewing for all kinds of different ad platforms so that on Google TV if somebody actually doesn't fast forward through an ad, you guys know it and that you could compensate people. You know what's going to stop TiVo from killing TV? Well, one of the things you guys could do it by compensating people for doing this. If somebody the do not call list has been killing telemarketing, but you guys could route calls to Google cell phones or to landlines and you could compensate people for listening to it. You can even change your model on AdWords and you could start compensating people for clicking through and for keeping the landing page open. And this is not at war with trying to make advertisements relevant, but it's a way of letting advertisers credibly stand behind. I believe this is going to be a relevant ad for you and I'll put my money where my mouth is because I'll compensate you for it. There's a little bit of a distortion that's still out there that you guys could make progress on. It seems to me because on Google TV you better have something splashy in the first three seconds or people are going to fast forward through it. But you don't have to put something splashy, just put a band around telling people they'll get paid if they listen to it or 95 characters can't always tell people what a new product is. But if you say look, I trust that there's something of value here and I'm willing to compensate you for it. So why am I throwing this out? Well, I just want to say humans can still hypothesize, but it's also related because one, I think you guys have great data to be the people to do this and secondly, don't do it without testing it. By the way, another small implication of this, this would radically change Google Checkout. Instead of being a micropayment system, it would be a micro compensation system. And let's see what you could do a randomized test about whether offering this increased acquisitions on Google Checkout. So hypothesize but then be willing to put your hypothesis to the test. Finally, let me say just a little bit about some of the concerns about this. I'm a great fan if you can't hear it in my enthusiasm for super crunching. But there are some ways that we should be concerned about the way it impacts us as workers and as consumers. On the worker side, there's definitely part of this trend is a shift of discretion. Frontline employees who used to have discretion are losing out to super crunching. And a clear example is loan officers. And the revolution here is pretty much complete. In the old days, loan officers, they'd be able to look you in the eye and say, ah, this guy is credit worthy. I think he'll pay off his loan. That was a stupid model. And now instead you want to run some regressions, you want to do some statistics. And now loan officers have no discretion. They're glorified secretaries. They enter in data and the computer tells you whether it's done. And even though we're going through a subprime lending crisis now, I promise you one thing, we're not going to go back to the old days of discretionary loan officers. We'll get a revised statistical algorithm that tightens stuff up. But we're at least I will predict, I'm willing to run this test, that we're not going to go back to discretion. Physicians are starting to lose their discretion. Patients are coming to them using Google as a diagnostic tool are starting to think of them instead of substitutes for a web portal such as WebMD. This is a brave new world where lots of people are losing out. Indeed, once you have statistically validated a script, the person who reads the script doesn't have to be sitting in the United States. So it's even, I think, tied a little bit to outsourcing. Now, if you listen here in the back, here's another version of scripted the impact of super crunching on script. This is a controversial teaching technique called direct instruction. And if you it's completely scripted, the teacher doesn't have to do any prep. He or she goes in, opens up the script, says, good morning class. And if you listen, you'll hear get ready. Over and over again. Yes, steal. Get ready, get ready. And lots of people have actually seen this clip, but they haven't heard the audio like that. What it comes from is actually from 9 11. This is what was happening when the second plane hit the World Trade Center. And we've seen this. Some of us might have seen this in Michael Moore's film 9 11, but Bush was doing this photo op because he wanted to support statistically validated but a very controversial training program. And there's a lot of data behind direct instruction. But it raises a real question. You know, I care both about employees and about the kids. I'm not sure that I could flourish having to read a script all day long. But here's the tension. Scripts often do a better job at teaching. Project follow through studied 79,000 children for 20 years in 180 different communities. And direct instructions just beat the pants off of 16 other teaching methods. It was first in reading, first in math, first in spelling, first in language. But wait, it gets better. It was especially good for kids reading below grade. It helped economically disadvantaged kids. And this isn't a very pleasant emphasis, but if you have poor teachers, it does pretty well because they just read the script. And even here, as far as the teachers go, it's kind of interesting. Some teachers who thought they would hate this after a semester, they see their kids being able to do things that they've never been able to teach them. And there is a certain joy about actually succeeding in teaching. But the bigger point to keep in mind is this tension. Are we moving toward a world where line employees have less discretion? Let me turn then to consumers. One of the simplest ways, and this is too much of an oversimplification with regard to quality. Super crunching I think is pretty much a win win. It helps consumers. This is part of the long tail. It helps consumers find the goods and services that they like. Hard to tell a bad story. But when a seller super crunches on price, you better hold onto your wallet. They can start. Increasingly we might see randomized tests to see how much individual consumer segments are willing to pay for price even on some non price terms. I'm concerned about the possibility of randomized test on whether you should submerge your warranty terms or what it takes to return an item or what it takes to cash in a rebate. Not clear that all of this testing that leads to higher ROI is going to lead toward more consumer satisfaction. Another version of this has to do with customer relationship management. Again I should speak clearly. I'm on net a great fan of customer relationship management that generally this is a little bit like eharmony's. In the business of predicting happiness, CRM is trying to figure out which consumers are happier and which aren't and generally trying to make its customers happier. That's a good thing. But there is a little bit of a downside here that some firms have incentives to engage in happiness discrimination. That if you once you Figure out which of your customers are happier and which are less happy, you might have some incentives to discriminate against the happiest consumers. So let's say that you and I were just on a plane, on a flight that got canceled, and there's one seat on the next plane. You know who gets it. Well, before CRM, let's say you had more frequent flyer miles than me. You'd probably get the seat on the next flight. But under CRM, they may predict that I'm less happy than you are. Maybe they know that I had two flights cancelled in the last six months and they may give me that next seat. Indeed, because he has frequent flyer miles, he's less likely to change things. So your frequent flyer miles make you less likely that they'll give you the seat because being a happy and loyal customer isn't all that it was cracked up to be. And so there is this aspect of CRM as a kind of corporate Zoloft that evens out the highs and the low of happiness. If they can't make both people happy, you can chisel a little bit on your happiest consumers. So this is. I don't know how many of you saw this Nicole Kidman film the entire. I heard something today. A firm called Epigogics, it used neural networks to try to go out and predict how much the box office on this movie would be. And they predicted 69 million. And lo and behold, it was a pretty good guess. The actual box Office was just $4 million more. This was part of a nine movie test where epagogics predicted nine movies just based on the screenplay without. This is a little bit like Orly Ashenfelter, who predicts the quality of wines before he tastes them. This is predicting the quality of the movie before they even start shooting it. And it wasn't perfect. Epagogics made the predictions, accurate predictions on about six of the nine films. But that was. What's important is that the studio was only accurate on three of the nine films. And doubling your accuracy rate so that you don't make unsuccessful films is incredibly important to a bottom line. But what's more controversial, Epicogics will not only price out the current script, it will make recommendations. And on this very film, which they didn't take the recommendations, Epicogic said that if you had put in a young sidekick, it would have sold 12 million more in tickets, or if you had changed the emphasis more to New York city and the UN that you might have made $5 million more and a lot of people respond to this with horror. Isn't this the death of art? But here's a place. Well, at least we should have a conversation about this. But producers have been intervening screwing around with scripts for decades. At least now they're doing it based on good information. So where are we on this? The future of intuition is going to be one where people toggle back and forth between their intuitions and data. Again, it's not the end of intuition, but you have to be willing to put your most precious hunches to the test. And by the way, just as a fun place to play, I've created now of 30 or 40 prediction tools and collected some others from the web. And you can go here and use and make some of your own predictions. For example, here's a little widget. If you're pregnant, put in some information about yourself and it will rely on the regression analysis and tell you what your due date is or how tall your kid is likely to grow. So just in conclusion here, super crunching is already remaking how decisions are made. Discretion is shifting from traditional experts to people that are relying more and more on data. Particularly I want to emphasize there's just tons of low hanging fruit with regard to randomization and firms like Offermatica or this web optimizer that's really democratizing the ability for us to run web experiments is so important. Don't get left behind. Thanks a lot. Happy to take questions. Yep. So who's going to be the next president? Excellent. Well actually there is a election prediction on here that so we got our two best guesses. One would come from the wisdom of the crowds. The prediction markets of course say that Democrats are 60% chance of doing it. But there's also a regression based approach that says right now given what our growth rate has been, that the non incumbent party is more likely to win. So on. Both of these are pointing toward Democrats as the likely as our best evidence of who's going to win. And both of these I think are better than just do better jobs in the polling results. Yeah, sure.

The examples you gave were the typical ones that we learned in school. Create a hypothesis, then test it. Is there anything to the now that we have the capacity to do a lot of number crunching, is it worth us to think about doing a fan? Just a spray of cross tabulations and guessing at well this seems to correlate with this. Therefore there must be some causal relationship. Are we to a point where mathematics, the power of our machines is now able to do this for us or do we need to go back?

So it's an excellent question. I don't think we quite know yet. All right, so here's. But this is what if you are going to do that, you're going to need to save some of your data and do some out of sample testing. And. To my mind, there are literally infinities of possible things that could be causing any process. And so you're going to have to use some limitation to cut down. So even with petabytes of data, conceptually they're more than a petabyte of potential hypotheses that could be driving that. So you'll never have enough data to do everything. But that may be. But it may be that we can do a. Some kind of interaction use. The fact that I'm wearing a yellow shirt probably doesn't have much of an impact on who will be president. So if we can exclude a lot of completely irrelevant stuff, of course, how do I know that's irrelevant? But I'm using something about my intuition or experience to do that. If we can somehow limit it, there may be ways of trolling for causality. One aspect of this that I think is definitely worth pursuing with machine learning and things like neural networks is there's a question of what hunches we're going to test. And then there's the functional form of how they integrate and the neural networks. One of its powers is it is massively nonlinear. It lets the data tell you the functional form of the interaction. You still have to tell it the 200 variables, but then it supplies whether there's some logarithmic relationship between them or linear or quadratic. But the question of finding whether it's 200 or 400, I think it's still even at this date, an open question of whether we can allow the machines to go out and troll for correlations. What I am confident saying, if you're going to do that approach, you better be especially vigilant in doing out of sample testing on that because it's so much more likely to be capturing chance correlations that don't hold up out of sample. Yep. I remember a few years ago Amazon.

Got into a little bit of attention because they were offering different prices for the same items to different customers.

And there was some consumer outcry and even people talking about regulators being involved.

Do you see this number crunching, super crunching.

Getting a backlash and possibly meddlesome.

Government getting involved with it?

Yes. And it's really possible in the Amazon example, people thought they were doing Segment contingent pricing when they were really doing a randomized pricing experiment. But even. And they responded by saying that they wouldn't. They didn't say they'd stop doing randomized price experiments, that they said everybody who participated would get the lowest price regardless of what price was posted. But it is troubling to me that we might get segment contingent prices and beyond price segment contingent terms so that they might offer me a longer warranty or a shorter warranty or they may play with the substantive terms. PPOD got in trouble and did a consent decree with the Massachusetts attorney general, I believe because they were offering different prices online than in their stores and it wasn't adequately disclosed. So one of the questions to me is, and my first instinct is that if you're going. That if you're going to have regulation, it should be disclosure that if you. That we might have a default, that your prices are not demographically contingent or behaviorally contingent. And if you. And that is an implicit representation that any E commerce firm makes to its customers and that you're free to do demographic or behaviorally contingent pricing or terms, but you just have to disclose that you may be doing it. And that would give some possibility. But the other thing that we're just not. While I talk about some of these shenanigans, I should emphasize too, remember Faircast, there's going to be a market niche for people to come in for firms to come in and do counter crunching. And we see this already in comparison pricing. And it's not clear who's going to win out even on the pricing term, whether it's going to be the super crunching to extract money or the super crunching on the customer side to spur competition. And I'm just kind of agnostic about which way that's going to come out. And I could see some role for a little government informational nudge, but I don't quite see. But I think we should go cautiously because right now, you know, the world is getting better and we should wait until, for example, we might wait until we see some bad actions. And right now there's a possibility of bad actions, but at least none. That's no smoking guns that have come up. Yeah, basic question. The slides say Google Confidential. Yeah, the slides say Google Confidential. But this is not going to be confidential. I think it's going up on YouTube. Yes, you right there. I'm. Oh, you're waiting for the mic. So the mic keeps running around.

So randomization, randomized trials are done because there's no. You can actually Infer causation. There's nice properties, but in a lot of cases you can't do true randomized trials. There's lots of examples of this obviously in the world, in the medical world especially, and things like that. I mean, even in a web world, you can imagine something like you have a website and you have a new version of the website which asks users to log in or create an account or something like that, right? And you can't split the people into the users into two groups and force the ones in the group that you decide to be the experiment group to actually create the account because you don't know whether they might do that if they don't opt in.

Right.

So do you have good examples of super crunching in these kinds of conditions too?

Yes, yes. So first of all, I'm with you that you can't do randomized tests on everything. But even on the example that you're talking about, the theory on randomization has gone pretty far and it's still developing. You can do what's called an intent to treat test. And let me first start about the Philippine study where we with on commitment bonds to quit smoking. So you can't force people to give you their money either. But really what the treatment is that you do is the treatment is the offer. Half the people we give the offer to enter into a commitment bond, and half the people we don't. And at the end of the day you end up with three groups. You have the people, the control group that wasn't offered. And then you have among the people who are offered, you have the people that take the offer and the people that don't. And so one of the ways that you can do it, you do two different things. One is just say, does the group that got the offer, is their overall average of quitting smoking higher than the group that didn't get the offer? And then separately you can think of what about those that accepted the offer and those that didn't. And indeed, in the example that I gave you, I just gave you two of the three, it was 5% for the control, it was 5% for those that were given the offer and didn't take it up, it was 40% for those that got the offer and took it up. So overall the treatment of getting the offer had the impact, not a 40% impact, but a blended impact between 5. That same intent to treat methodology can be used in your example too. The treatment is giving the person the offer and then you're going to have those people that take it up and those that don't. And so you can't always test the exact thing. You can't then rely on that test to say mandating something or forcing everybody to take something is a different question. But at least you can get a, a randomized test on whether the offer has an impact or not. Yes, sir. You. Oh yeah. Time for one more question and then we'll distribute those afterwards.

I just wanted to bring up, you know, the more complex systems where, you know, there are all sorts of interactions that can occur. You know, Amazon can find that 10% increase in prices is they're able to get away with today, but you know, in a month people will learn that they're expensive and to go to some other website which is a consequence they don't want. Even though the data may show it looking good today or people learning to game a system, you discover that you get a different price every time you go to the website. So you just go to the website four times and pick the cheapest price. I mean, all these systems are really a lot more complex, I think, than meets the eye and have to be dealt with.

So I agree. One of the ways to think about this is if you're going to do randomized testing or even if you're going to do regressions or some kind of testing on historical data, you need to have output measures, outcome measures that you believe in, that you care about. And if you, in your examples, if you're just relying on short term outcome measures, when really your profitability is going to term on long term outcome measures, you can make a mistake. And there are contexts. One of the limits to when you can do randomized testing or when you can do regressions is an inability to get the right outcome measure or not to be able or not to be able to easily get it. And so some of these you can do the quick randomized test that I've been extolling in 12 hours, but you really need to do it over six months. And that is at a minimum going to increase the cost, the cost of doing this a lot. For important things though, you might. Well, I think the eHarmony example that I talked about of whether they add in the AdSense column or not is a stronger one because they let that run long enough to see whether it impacted the number of weeks that people stayed as clients of eHarmony. And so they actually were trying not to get a quick hit, such as themonster.com, but you're trying to get instead a longer term. But more generally, I want to concede that there are going to be some complex systems where it's not just a question of short versus long, but it's a question of, of five or six different variables. And I so I actually see that randomization tends to be the easiest to do on the web and it's pretty easy to do on off web marketing such as direct mail and newspaper advertisements. It gets harder and it's a little bit harder in HR departments where there can be multiple issues and it has to run longer maybe. I tend to think that things like inventory, you can still do randomized tests there, but there are really a lot. Inventories start to get messy and there are interactions with other things that you're holding in inventory and it's hard to figure out all the impacts of repackaging and things like that. So yes, randomization is not a panacea for every business problem. But I think that Google, that corporate America would still do a lot better if it at least asks itself the question before it takes any initiative online or offline, at least ask the question, could I do a randomized test first? And to tell you how much I believe in this, I'm writing a piece right now. I think Congress should do this too. That they could actually do randomized tests of laws before they roll them out, before they put Sarbanes Oxley mandates on corporations or they can even do this to repeal laws. Should we repeal Sarbanes Oxley? Well, let's how about repealing it on a randomized sample of 10% and see what happens and let it run for five years or three years. So not everywhere. But. And I don't suggest that we do randomized testing on the Fed's interest rate, but there are going to be a lot more places than we're currently doing it where you could actually find the cause of some important variables. Thank you so much.