#53 Data Science, Gambling and Bookmaking (with Marco Blume)

This week I'll be speaking with Marco Bloom, Trading Director at Pinnacle Sports. Marco and I will talk about the role of data science in large scale bookmaking, how Marco is training an army of data scientists, and much more. At Pinnacle, Marco uses tight risk management built on cutting edge models to provide bets not only on sports, but on questions such as who will be the next Pope, who will be the world hot dog eating champion, who will land on Mars first, and who will be on the Iron throne at the end of Game of Thrones. We'll discuss the relations between risk management and uncertainty, how great forecasters are necessarily good at updating their predictions in the light of new data and evidence, how you can model this using Bayesian inference and the future of biometric sensing in sports betting. And as always, much, much more. For the record, we recorded this conversation in December 2018. Welcome to Data Framed, the weekly data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I am your host, Hugo Bown Anderson. You can follow datacamp on Twitter @datacamp and me hugobown. You can find all our episodes and show notes@datacamp.com community podcast. This is Data Framed. Hi there, Marco, and welcome to Data Framed.

Hi. Thanks for having me.

Real pleasure to have you on the show. And I'm actually really excited to have you here today to talk about sports betting, how data science plays a huge role in what you do as trading director at Pinnacle with respect to sports betting and also the fact that sports betting in your line of work doesn't only allude to sports, but that at Pinnacle you do lots of different types, types of bets. And I'm really excited about getting into the weeds there. But before we get to all of that, I want to find out a bit about you. And so I'm wondering first what your colleagues would say that you do risk management.

I think that's probably the best assessment. I'm responsible for managing all the risk that is associated with wages at Pinnacle. Overall, sports life, pre life, any aspect of the betting. I manage the risk at Pinnacle.

Fantastic. And do you think your colleagues, as you do so much quantitative stuff, they have an awareness of kind of the ins and outs of your daily life? Or do they think it's all, let's say, texters and whiteboards or pen and paper or writing code and building models?

I mean, there's a lot of black box for them. I mean, at the end of the day, I mean, most of them have different areas of expertise and inner workings of the trade floor are just too complex and too specific these days. But I think that's true for most areas. If you deep down look inside, how much do you actually know about other areas anymore? So I would say the day to day is probably unknown for them. But my actual day to day job is.

Yeah, and I think you're right that due to increasing special across so many disciplines that it is, things do become more black box as we head down that path. So maybe we can step back a bit and you can just tell me a bit about what Pinnacle actually does.

This is 2018 is our 20 year anniversary. We are one of the largest bookmakers in the world and we are known for being a very efficient bookmaker. In terms of pricing, we are considered. Some people compare us to the NASDAQ of prices, meaning that the traditional bookmakers that people know and heard of are usually more the recreational field of work. And Pinnika is actually a true bookmaker. That means we have very low margins, very high limits. Our website is not so flashy, but we have an API that people can interact with. We are like a real true bookmaker trying to quantitative analysis of sports events and or other events and allow people to build models against us and place wages with us.

And so then as trading director, what does your day look like? What are the ins and outs of your actual job?

I mean, it largely depends on season. So sports is obviously very seasonal. You have, you have big events. Like this summer we had the World cup, which changes my job dramatically. But overall, day to day would be sitting down with my managers, maybe going over the week, going over the month, discussing some plans about some products that we want to roll out, discussing some models that we need to test, discussing some of the new strategies we want to try. And overall it's like a constant. Strive to improve our product and obviously do analysis about things that we tried that didn't go so well. That's the bread and butter of my day to day.

So how did you get into data science?

Initially by sheer for so I was always like math guy, but I was never in data science. And once we started building our quant team out, our quants started before we used Excel for everything. And then the quants started using R and they were coding in R and I quickly picked up that the level of efficiency gain they had over me was order of magnitudes. They could analyze data so easily that was inaccessible to me just because of the natural restrictions about Excel. And so I started, started at the Coursera course, did my lectures there and started coding R. And then pretty soon it became a bread and butter tool for me. I couldn't actually believe that I didn't have these skill set before and did my job.

And which Coursera course was it that you took?

So it's the very first one. It was the very first course. I think it's actually the data science track. That's what it's called back in.

So that's Roger Peng and Jeff Lee.

Yeah, Roger Peng and Jeff Lee, exactly. That's the original first course I took.

I'm in exactly the same position actually. I spoke with Roger about this on this podcast that I was actually in one of the first cohorts and maybe you were too, around 2012, 2013, something like that.

Yeah, maybe around that time. For sure.

Yeah.

It could have been tough because I didn't come from coding. So for me this was brand new. I thought it was a really tough course for me. I actually struggled quite a lot. And the thing is I knew I had enough expertise in my team that the answers were available to me if I ever had a question. And I knew exactly what I wanted to achieve. So I had a very clear goal in my mind. What do I want to achieve? I wanted to interact with our data directly. I wanted to access our data database directly and do analysis over it without the need to ask somebody for a data pool. And then this data pool had some missing columns, some missing attributes. I need to ask them again and I need to give to the analyst team. I just wanted to reduce the red tape and be able to be self sufficient.

So some people might have a question revolving around the Venn diagram of data science and sports betting. And I'm wondering historically up until now, what the role has been of analytics and data science for bookmakers in bookmaking.

You have a few vector of which makes it really interesting. You have the classical sports analytics, how does a sport work? And sabermetrics for the people who know baseball was leading in many aspects. But you know, sabermetrics, ideas and concepts are now almost existent in every other sport, especially soccer, you know, football for the Europeans. There's a high level of analysis done right now. But this is all the field that surrounds the sport data analysis. And but since we are trading house and we actually have a ticket flow coming in and out. And so we also have the traditional financial analysis about risk management assessment and basic game theory strategies and all of that stuff in addition. So we have a very nice overlap between those two worlds and have to manage both separately and then mesh them Together eventually. Which is often the hard part, I'm sure.

So when I came into this, our first conversation earlier this year, I was under the misapprehension that sports betting was really only about sports. And you opened my eyes. And at Pinnacle you do all types of bets. So I thought maybe you could run us through a few of the more interesting to your mind types of bets you can make inside and outside the sports space.

At Pinnacle, you can bet on literally every single sport that you could possibly imagine. And this includes dart and chess and anything you see obviously esports your video sports very popular with us, but you also have politics. Politics is a big batting field. You have a few of the more exotic and fun stuff. Since you're recording this from New York, I believe we do Nathan's hot dog indie contest, since he is. I know exactly when Kobayashi was not on top of his game anymore. I remember that. And we do. The Pope election was a fun one, you know. No, not, you know, that was very interesting to try to price up the Pope, the Pope election. And so it's almost any event in the world. You know, you could even go as far and do stuff about Game of Thrones. So we have a Game of Thrones prop up. Who will sit at the Iron Throne at the end of the season. Oscar betting, Golden Globe betting, you name it. Literally any event you could possibly think of, you could place a wager on.

That's incredible. Of course I don't want you to give up any of your IP here and of course you won't. But I'm wondering like let's take you know, hot dog eating contest or Game of Thrones or who will be the next Pope. I'm wondering how you even, I mean you have the technical skills, but in terms of domain expertise, I don't suppose.

This hero obviously zero expertise. So I mean, I mean, let's talk about the Pope. So how does it work? We read columns about populist writers. What do people believe to be the truth? And then we price according to this. We don't have any insert information. We don't know anything about it. But we read up a little bit. We try to price as good as we can. And then you let over market efficiency boost all the crowd. You know, effects shape the price. Game of Thrones, obviously we have a defense ourselves. So we speculate ourselves and. But we don't know. We don't have an insight. We don't know George R.R. martin personally or anybody. It's guessing. But to be, I mean, to be fair And Frank, these are entertainment props to bet on. They are not in comparison, like on a, let's say on a World cup game in soccer, you can bet up to 500,000 or a million dollars with us without even question. And on these kind of props, the limits are low, maybe $1,000, maybe $500. So there is a difference between the level of scrutiny that goes into pricing one or the other.

Absolutely. So in terms of pricing them, I suppose, can you talk us through the process from go to woe in the sense that I presume you, you know, you have some model which ends up with a probability distribution or probably mass function or density function with respect to outcome and then you price according to those distributions. But maybe you can spell that out in a particular example.

It really depends. It really, really depends. We obviously have, have exactly what you just said. In some aspect it might just be market prices. So the market has a price already. What I might mean by that is like if, if you would like to open an exchange that trades Apple stocks, you wouldn't need to do all the analysis yourself. I mean, Apple stocks is traded at many, many exchanges. So you have an idea what the price should be. And that's the same in sports betting. Many, many bookmakers exist and they were connected. But especially when you talk about a life game, you know, there we have tons of models running. So you, you, you feed the model tons of inputs and, and then it crunches the numbers and spits out something. And there's several layers of models and all kinds of AI and machine learning elements. And it gets very sophisticated depending on the sport and depending on how much betting there's done in the sport. The more betting is done, the more sophisticated we have to be because the more sophisticated people on the other end die as well.

So I think once again, this speaks to your job of essentially managing risk. And can you just say a few more words about what risk management or managing risk in general amounts to for you or looks like or how you think about it?

Yeah, I mean, probably risk is always like, even if, how do you maximize equity over probability space? Meaning even if you have a coin flip both at 50, 50, you don't gain equity there. But I hope maybe somebody pays you a little bit more, but you also now would lose a little bit of money. How do you hedge yourself against this risk? Can you take it? Are you willing to take it? What happens if you lose? Does it have an impact on the financial bottom line? Are you exposing yourselves? And all these kinds of questions, how do we think Long term about manag book. You know, we, we, we a big company, you know, like you know, everything has, has to stay afloat and there's a lot of regulations and in it. So you have to be very carefully managing your risk probably and, and then try to balance the book in some aspects. It's not always easy to, to better balance the book. You know, the, the way that betting works is often based on news. You know, somebody is injured and if, if you're not on top of that and then you, you don't pick up trading frequency very very quickly. You just get overrun by wagers and then you're exposed in a very unfavorable scenario.

Right. That makes perfect sense. And how does to uncertainty in general?

Uncertainty is there's a few level of uncertainty. I mean obviously you have an inherent uncertainty because it's a spot event which is a non perfect environment. So you never know exactly what other parameters that matter. But you also have uncertainty variance meaning some events are naturally just more volatile and less known as other events. Quite clearly it has to do with the historic data available. If two people had a, you know, did the same competition 100 days in a row, you have very, very, very strong data that you know, eventually backs up one percentages or the other. And then you have events like for example a soccer world cup where Germany plays against Uruguay which has not happened ever basically in the sense that these exact teams have never played against each other at the end of, at a, at the end of the NBA season. All the teams have played against each other many, many times over. So you get a very good idea about the relative strengths of San Antonio spurs to the Golden States Warrior Cleveland Cava years. Even though maybe the Cavs and the spurs have only paired off two or three times. But because of cross relationships you have a very good idea how the south strength is. But Germany playing Uruguay, you actually have no idea. I mean what does it matter what German Uruguay was 20 years ago? Like none of the players on the pitch anymore. The game has changed. So you have a lot of different kinds of uncertainty in, in this. In. In the gambling world.

That's interesting. So it sounds like there's a distinction between kind of uncertainty that you can quantify so that would be risk and uncertainty that you just don't know a lot about the situation so you can't say so much.

Exactly. You have the known known and the known unknown and the unknown unknown. It's very tricky. I mean especially if these big events are very, very tricky and you notice it. I Mean, baseball is probably a prime example. I mean, with 180 games per season, at the end of the season, you have a very good idea what the strength is. But all bookmakers, in the beginning of the season, you will notice that the lines are much more volatile. Bookmakers, including us, are much more careful in taking on risk because we don't believe that the underlying odds are as certain as we wish they would be. But at the end of the season, if you want to bet against our lines, we actually are much less willing to adjust our probabilities based on betting behaviors and are willing to accept much more risk just because the certainty has grown so significantly.

Right. So you're saying then that when there's more uncertainty, when you have less data, less knowledge about the space, your lines will be more responsive to betting behavior?

Absolutely. A wager that could move a line in the beginning of the season, 3% might move the line 0% or 0.1% at the end of the season. Absolutely. You know, just because the certainty is there, eventually, you know, this is the price. Everybody has spoken, the entire world has placed a wager. We know the price. We're willing to take a gamble here.

Yeah. And I don't know whether you think about it in this framework. Here's stuff like this. I think about it in an almost a Bayesian sense that you have some sort of, of prior knowledge about, about the space. And then once you start getting more and more data, you can update whatever you're interested in and get a more precise estimate as you keep updating. Essentially.

Yeah. Bayesian thinking is predominant in our world. Like almost everything is. We do. From a Bayesian point of reference.

Yeah. Fantastic. And you know, I think, as I said, this idea of updating, like that's what came to my mind when you were discussing.

You have to understand, like, if you price anything up, you know, any, any event, like you see two people on the street and you imagine, let them do 100 meter dash, the initial price might be 50, 50 or something. And then you see one guy actually is on crutches. Suddenly the line moves 30%. Right. Or 40%. Now it's 90 10. But then you see, he throws the crutches away and then you're like, and you're on and on, like. But eventually you have a pretty good idea about, oh, okay, now, now I have, I know this guy's overweight, this guy looks fit, and you're pretty sure about your price. And if somebody tells you it's the, it's the complete opposite. You Might not believe it anymore. So. So not believing means in our lang, in our language, we're willing to take on a lot of risk before we actually get moved over to the new price.

We'll jump right back into our interview with Marco Bloom after a short segment. Now it's time for a segment called Data Science Best Practices. I'm here with Ben Skranker, an independent data science consultant. Hi, Ben.

Hi, Hugo. It's great to be back on the show. So what do you do when you have a bug in your code? Maybe you've tried unit testing, but you still can't figure it out.

Right. So a first approximation, I think, is to include some print statements in your code to see what's actually happening in there when you execute it.

Actually, that's a very common approach. We are all guilty of sprinkling a lot of print statements throughout our code to find a problem. In general, good programmers are lazy, but that is not the case here. First, you will add more and more print statements as you track your bug down. These print statements make your code harder to read and you have to find and remove all of them later, which is a waste of time.

I can't argue with any of that.

And it could be even worse than a distributed system. When I was in grad school, we had a visiting scholar who was debugging a Fortran program which ran in parallel to find a bug. He instrumented his code to log information to a file. He didn't really think this through because he was running at scale. He filled the head nodes disk and crashed the entire cluster. I was delighted when his visit was over and I could run jobs again. Sometimes print statements can be really bad.

That sounds like the stuff of nightmares, Ben. Tell me what the alternative is.

It turns out our engineering friends invented a magic tool long ago called a debugger. A debugger allows you to run your code a line at a time to see how it executes. You can examine how variables change as your code runs. Debuggers have pretty much the same interface for every language, whether it is Python, R, C, Java, Matlab, or something else. Once you learn one debugger, you can operate any other in Python. Use PDB in R. The debugger in RStudio is a great place to start. If you are using an ide, it should have a debugger.

And for those like me who love IPython, there's also IPDB, right?

So often when someone asks me to help find a bug, I always ask if they have tried using the debugger. Invariably the answer is no. I don't understand why so many people resist learning to use a debugger or actually using it if they already know how. It is not that complicated and you only need to master about four commands to use it. Being a professional means learning how to debug. It'll make you more productive and more self sufficient.

So how can our listeners get started with debugging?

Start by figuring out how to start the debugger for the language platform you are using. For example, in Python, run with the PDB flag which will put you in the debugger when an error occurs. Or import PDB and call SetTrace at the line where you want to start debugging and then then use the debugger to step through the code and examine your variables and call stack. These commands are identical. For any debugger, use step to advance one line of code including end of function call next to skip over function calls, finish to complete execution of the current function and continue to resume execution.

Great. So to recap, you use step to advance next to skip function calls, finish to complete execution of current function and continue to resume.

You got it hba.

So Ben, how do you look at variables in Python?

It is easy because you still have access to an interpreter. To examine any variable or execute any Python code, just type code into the interpreter like you normally would.

Anything else?

That's pretty much it. Other than if you want to see a local variable or function argument, you need to position the debugger to the correct frame in the function call stack, but you will have to read the Find manual for that. Learning to use the debugger will help you quickly find and fix bugs, allowing more time to focus on fun stuff. Stuff like doing science.

Thanks Ben for that delightful introduction into debugging your data science code. After that interlude, it's time to jump back into our chat with Marco. So we've spoken around this, but as we've stated as Director, as Trading Director, you think about everything from R and D to odds making to everything related to markets. And I'm just wondering if you could speak to how all these different aspects of your job are related. Perhaps speaking through the lens of a particular real or hypothetical.

One of our goals is to improve our models. Higher accuracy in our models, but also open new betting opportunities up to our clients and more interesting betting options allow people to then hone their own models and thus give us liquidity and then the machine gets rolling. At the end of the day, what we are we are a very Low margin, high volume bookmaker, a little bit like Walmart, right? We don't want to make a lot of money selling an orange juice. We just want to make a little bit where we want to sell a lot of orange juices. So the idea is like to develop a new product and let's take a hypothetical product. We want to, I don't know how many throw ins per halftime are there going to be in a soccer game. And so you start modeling this, you start putting it out, some beta picks up that your model has complete different wrong assumptions and will bet a lot of money on you. Then you go refine your model and so on until you get a solid market. And once you have this, you can roll it out over many leagues. You have to do more refining, more refining, but eventually you get to a stable product which now be something that a lot of clients enjoy betting. And thus you created a new market that clients like, you know, a new product that they're interested in betting. This product then will stay mainstream for the next 10, 15 years.

Interesting. When you're speaking about this once again relationship between domain expertise and data science skills, it actually made me think of, have you read a book called Superforecasters or Super Forecasting?

Yes, yes, of course.

For the listener. This is a project by Philip Tetlock and colleagues. And the basic idea is he found certain members of society who are better at forecasting than other people. One thing he does is kind of analyzes, looks at the characteristics of these people and sees what makes them better forecasters than others. And I'm just wondering, Marco, do you try to hire people who are super forecasters or instill this superforecasting culture within your organization or how do you think about that?

I'm actually paying them already. I used to call these people an army of consultants. Because if you're a super focus in my world, what it means is you're actually better than our, our models and you can actually predict the outcome better than we can. And thus by betting, making a profit. And so what I'm actually doing is I'm consulting you. Please. Here's my prediction for this event. What is your prediction? And by placing a wager, you're telling me your opinion, which I then can incorporate in my model and can change my prices, but I have to pay you the price. And so all these people who are great at forecasting anything are basically working with me on a consulting basis.

That's awesome. And I actually, I just had a thought that it kind of brings this, this Full circle in a sense that moved from data analysis and data science to bookmaking to the idea of super forecasting. And something that becomes very apparent in this book is that, you know, you don't necessarily need super special skills to be, to be able to be a super forecaster, but there are several key aspects, such as being less prone to confirmation bias than other people in the world. Right. Which of course is the hallmark of a great data analyst as well.

Yeah. So one of the classic training pitches that I used to give in for the longest time is when I get new training recruits in. And I mean these are all people, you know, they all are successful and bright and eager and obviously at the beginning of the career, but, and they want to make a mark, you know, so they're, they're young and willing to gamble. And I always tell them, you know, this is the strategy, this is how it works. And obviously you have to bring in your own fear field. But if you ever start gambling with our money, you know, then I show them a credit card, do it with your own money, and if you're rich enough, buy yourself an island. But you're not gambling with our money. Money. We are like, try to really separate for them. Like you, you think you might know something because you're sitting on this side of the table, but if you could, you will sit on the other side of the table. Maybe, you know, if, if you would be as good as you think you are and you would be a super forecaster and you can see everything so clearly, then, then just bet with us. You know, like on our end, it's, it's hard work, it's hard analytic work. Every day we have to grind, we have to refine our model, we have to get new data. It's a craft, you know, you have to hone it over many, many years. You don't just go in, in the classic Vegas movies and, and should be eight and a half year and the total is going to be 165 points. That's not how it works at all. Every day we go back in, every day we have people smarter than us, people better than us, outsmarting us in our own game. So we need to improve all the time. So I've seen over many, many years, customers who actually have a lot of talent then become lazy and the analysis becomes sloppy and at that moment they're not winning anymore because somebody else is more hungry than them and knows the numbers better than them and bets the other way.

Yeah, absolutely. And the other quality of super forec that I just remembered it speaks to this idea of updating and doing a Bayesian update essentially is that superforecasters are very good at updating their predictions and beliefs with respect to new data coming in as well.

So the way that trading works is one of the key aspects of trading is actually that the past is the past. You cannot change past wages. The only wager that you can change is the next wager. So what you're doing is you come up with a scenario like almost a probability tree and you say, okay, I put the line here. So I expect 80% this to happen, 20% this to happen, and whatever percent, maybe 0.1% that to happen. And this is basically your tree of probabilities. And from your experience. But if something unexpected happens, then you basically have to update your assumption very, very quickly because something out of the ordinary has happened, which means all your assumptions might not be correct anymore in the politics. Hope this word is okay. We always use it to call the, the ladies of service problem. You know, every politician, if you would ever be found with a lady of service, all the prior work would be almost meaningless. The odds would be abysmal. I mean obviously Donald Trump might, might refute this now, but back in the day it was always this problem that politics are so dangerous because if there's one character flaw being revealed of a politician or the analysis before becomes completely immediate, this, you know, then all then the odds would shift from 60 to 5% in a matter of seconds. And you, you have to account for the, for this possibility that somebody gets the information ahead of you. And we've seen it many times over the years that somebody has good information about something that that's not public yet.

So we've discussed a couple of tools and techniques from Bayesian inference to mentioning that historically you started using R, move from Excel to the R programming language. I'm wondering for people who want to enter, enter this type of space. Bookmaking, sports, gambling, or these types of prediction challenges in general. What type of tools and techniques in data science would you suggest they learn and speak to this from just general suggestions or like the type of people you want to hire as well? I don't have a strong preference there.

Yeah, so I mean classic. What we look for is the classic R. Python. Python stack. You know, like machine learning is highly thought after. It doesn't really matter which framework it is some machine learning framework where we can teach the other. It is a good thing. If you actually have done some sports modeling already, it doesn't matter which Sport, you know, but that you're familiar with how sports modeling works conceptually. Some of our people do like kaggle competitions, they're like that kind of stuff. There's a lot of different ways you can come from the same metrics, which is hardcore baseball analytics, but you can also come from different way. We have people who in the past were big in poker AI and did a lot of work on game theoretical approaches there. Because I mean our field is so diverse. You can actually come from a game theoretical point of view and build game theoretical trading models. You can come from a spot analytical model basis and build spot analytic models. And there's many different ways how you can bring in your creativity and your knowledge to get there. But the classic computer science background is very, very highly thought after. The alternative is a strong math background. You come from the other end, you're very proficient in high level math and now you learn some coding skills and are able to help and sit down with another guy to develop a proper AI model. And you do the quant stuff and he does the coding stuff.

So you've mentioned R and Python. Is there a culture of one of these more strongly in your organization than.

The so Pinnacle is very heavy reliant book on R. We much bigger in R than we are in Python. We do Python. It's not. We're not a cult of R in the sense that we feel like we need to use R. We feel that R gives us the best bang for our buck in many aspects. We have also been very active in the R community for a long time and speaking at conferences and we send people to almost every R conference. And it's a community that also embraces the idea of what smart sports betting. We have released data sets into the community. We have worked with members of the community to improve packages that we maintain, which are free and available for everybody that help with sports betting. So it's a great community. I mean most people who like analytics like sport analytics. Sport analytics are great in the essence. The difference between analyzing sport and many other things is that sports have a finite end. Like you can analyze the game as much as you want and then after 90 minutes or one hour, whatever, you have the result on the 10 table and now have the next game to analyze. There's always like something happening in sports, you know, which makes it very interesting. And with betting, you actually have a way of keeping track of your score. Betting is just a way of keeping track of your score. How good is your model? You know, the better it is, the more money you make absolutely.

And as you say, there is a final result in sports. Right. Someone wins and someone loses most of the time.

Exactly. You know, like, yeah, this is the big difference between financial trading and sports betting trading. The big difference that financial trading is almost infinite. Right. There's not an end to the price of oil. You know, commodity exists continuously while all the sporty events are discrete events. Yeah.

I'm just wondering with all the new technology and you know, deep learning and video analysis and that type of stuff. I know that like a lot of basketball for example, is captured on film and things people think about doing deep learning analysis on players movements and that type of stuff. Is this something you've thought about at all?

We think about it, we haven't done OCR analysis. One of the key features of us is that whatever attributes we want to use in our models needs to be available live and fast. You know, it doesn't help us to have a very rich data set of data, data points that we cannot get while the game is in play at a reasonable speed. And reasonable for us means maximum, maybe a second or two seconds slow. If something is 10 seconds slow in our world, it's, it's, it's basically like yesterday, it doesn't matter to us. So our world is very fast paced. So we need to find data points that can be analyzed and can be transmitted to us on, on a fast pace. This has increased. I mean if I remember, I mean if you can imagine like 10 years ago, 20 years ago, in most games the only data points that you would get are the very superficial high level one, something like if you talk about basketball, you might get rebounds, steals, points, blocks, the classic four. But now you, you get something like dangerous attacks. A human puts, puts an element of, of a concept on it, you know, and then gives, gives you a judgment that helps you understand the game. And so it has gotten a lot better. And now with eventually biometric wear, we will eventually get data super fast, super accurate that we can use. I mean it sounds fantastic to use heart rates and basically try to see if heart rates matter and all that kind of stuff.

Yeah, body temperature, amount of sweat, perspiration on forehead, you know, all of these things. I wonder how, I wonder how many dimensions you need in order to describe these things. I think that's a, that's a cool question. But in terms of processing all this data in real time, I suppose we have this misconception in the cultural consciousness that in data science that to run fast code and machine learning models in production, they Better be in Python and that's something we've been discussing on the podcast recently actually. But I'm just wondering if you can tell me about your experience with productionizing R code and efficiency which I know you're very interested in in these types of things.

We actually spent the better part of, I don't know, two years now thinking exactly about the question about how to productionize R effectively. So if you've done a lot, the traditional pipeline for us would be to transfer the R code into C C for production liasing just because of the speed. But we found now new APIs we're using plumber actually and in some aspects for the people who know R, which is an API interface to productionalize R in small scale test scenarios and it has been working well for us. So we actually running R code in production and production environment in trading algorithm environments.

Fantastic. And I actually had someone on the podcast recently who talked about using Plumber and ARC eras together that it worked very well for them.

Yeah, yeah. I mean it's still a little bit at the infancies and some of our team members are actually on the very cutting edge and working with the guys of these packages together to help to improve stuff. But it is promising and it allows us iterate even faster over our models because we don't have to take this extra step about productionizing. Traditionally Python was used in our world when the AI learning models didn't have a good interface into that's when you basically distinction was if you want to do data analysis you do it in R. But if you want to code AI, you have to do it in Python. But nowadays with R being interfaced and all the other big machine learning frameworks, you don't necessarily need it anymore.

And of course you're at RStudio conf in San Diego this year and JJ Allaire's keynote about R Keras and how we're seeing more and more of R being an interface language to the these pretty, pretty serious packaging infrastructures was really cool.

That was an amazing talk. I mean I also couldn't agree more with him that the way that they're pushing it is towards an open framework for everything. Let's make R interfaceable with everything and let's not try to close it up and build A and try to do it our own. Because then you get into this classic language wars problem about people being in a cult. People should use whatever language they want to use and R should be able to help them use all the tools that are available in all Other languages.

Exactly. So you've mentioned several times in various guises how Pinnacle and yourself work with the R community at large, from giving talks at RStudio Conference and other conferences to working with developers on packages. And I'm wondering if you can just speak to how important the sense of community in an open source landscape is for you in your job.

Oh, that's everything to us. I mean, that's part of why we liked R so much is at the very beginning when we had problems with the classic OCDP packages. This is Jim Haster, and you have direct access to the guy who made it and you can ask him a question, question. And if he knows that you know what I'm talking about, he's actually working with you on your environment to help troubleshoot and then improve the code. To have this concept that the developer of a package actually cares so much about making a bug fix that addresses a tiny problem that might only exist in your configuration, which is a bug in his code ultimately, and to make it better is amazing. So we've done so much work on that. That's why we decided eventually also to release some packages into the world to tell people, hey, if you want to get into sports betting, these are some ways, you know how. These are some tools that we used internally before that might make your life a little bit easier.

Cool. And we'll definitely link to some of those packages in them in the show notes as well, so interested listeners can check them out.

Yeah.

We'Ll jump right back into our conversation with Marco after a short segment. Now we've got another segment on statistical distributions and their stories with Justin Boyce, a lecturer at California and a Data Camp instructor. Hi there, Justin.

Hi, Hugo. It's good to be back for another season of Data Framed.

I agree, as it may have been a while since Dataframe listeners have heard one of these segments, or we may even have some new listeners this season. Can you give us a quick review of what you're doing with these segments?

Sure. There are many named probability distributions out there, and it can be a challenge to make sense of all of them. I find that it's easiest to think of the distributions in terms of the stories, but behind them, in each segment, we've introduced a distribution and a story. Along the way, we've introduced ideas such as Bernoulli trials and Poisson processes.

Great. So can you give an example of a distribution you covered last season?

How about the binomial, the number R of defective light bulbs in a production batch of n of them each with probability theta of being defective is binomially distributed. More generally, we can think of a light bulb being defective or or not as a Bernoulli trial, which has an outcome that can be coded as true or false or equivalently success or failure. The number R of successes in n Bernoulli trials, each with probability theta of success, is binomially distributed.

Great. So here we have stories plus changing around the nouns. The name of the game.

Yes, with the usual caveats. It's not the whole game, but you can get really far by thinking about stories like this.

So what other distributions have we covered?

Well, we did several discrete distributions in addition to the binomial distribution we were just talking about. These include the Bernoulli, Poisson and geometric distributions. We also did two continuous distributions, the exponential and the gamma. These are all used to model real world phenomena and you can review them and the other distributions and their stories and how to use them in Python and Stan at the link in the show.

Okay, I think we're up to speed. What do you got for us today, Justin?

Well, today we're going to talk about the story of the normal distribution, which is also known as the Gaussian distribution. I'm going to do this one a little bit differently though. I'm actually going to tell you the story of how the normal distribution was discovered.

I love history lessons, but why do some history for this one? You haven't done that for any of the others.

Well, in this case, I find that the history of how the distribution was discovered helps us understand the story behind the distribution.

Sounds good, let's hear it. But before we do, can you remind us what the normal distribution looks like?

The probability density function of the normal distribution is the classic bell curve. We are all used to seeing a symmetric peak with tails that fall off like E to the minus x squared. The field of probability began with studies of games of chance, notably by the Bernoullis. Jacob Bernoulli had already identified the binomial distribution as an important tool in understanding outcomes of discrete events, which games of chance often involve. Binomial coefficients, which are necessary to compute probabilities using binomial distributions were difficult to calculate, especially in the first part of the 18th century when all of this was going on. Bernoulli and others were working to find ways to approximately calculate binomial coefficients. And that's where de Moiv comes in. De Moivre found that he could approximate the binomial coefficients, at least for large n, with integrals we now know are related to the cumulative distribution function of the normal distribution.

So you're telling me De Moivre discovered the normal distribution?

Kind of. He did not understand the concept of a probability density function, and he was really only looking to approximate binomial coefficients. But looking back at it, his result is an important one. It says that the binomial distribution, which is discrete, can be approximately by a normal distribution, which is continuous.

Great. So do we have a story for the normal distribution then?

Yes, and I like the way you used an indefinite article there. A story for the normal distribution is this. If a variable is binomially distributed with parameters n and theta, with large n and theta not too close to 0 or 1, it is approximately normally distributed. The mean of that normal distribute is n times theta and the variance is n times theta times the quantity 1 minus theta.

So this is a story for the normal distribution. Are there more?

There are indeed.

Great. Let's have you talk about more normal stories the next time you're on the podcast.

I look forward to it.

Time to get straight back into our chat with Marco. You mentioned earlier that what you referred to as an army of consultants. And I love, I love your military metaphors and analogies in general, and one of the ones I really love is that you've stated that part of your mission is to train an army of new data scientists.

R was predominant or is predominant in the R and D team. But, you know, Pinnacle is a data driven organization. And so we, we had this huge gap between the people who know R and people who don't know R. And my belief was, you know, especially with the Tidy Ver coming along, that there was a path where people who are unqualified in the terms of they never done computer science before, they never coded, they don't have a math background, and they're not techno people either. They are just maybe in human resources, maybe they work in business analysis, they've done every. Or maybe we've met people who worked in customer service for years and that we can come up with a curriculum based on the tidyverse, based also on the mass of the tidyverse, lessons that I quite enjoyed myself and basically tried to build a collection room with the help of Data Camp and specifically tailored to the people. And the success has been overwhelming. We have now, we trained over 150 people now, maybe even more by now we have R being used in every aspect of our company. It is the smile on my face when I go around and I see somebody who I know doesn't come from this background showing me an R markdown that he just created. And a report that he sends to his colleague, which they're going to discuss is just amazing to me. I mean, there's a sense of empowerment at every level of the organization is just fantastic. We have a data warehouse where people can access the data. We've made an interface which made it very easy to get the data from the data warehouse directly into your session. We're using all kinds of tools that RStudio provides. We're using RStudio Server, we're using all the tools that they have. And it is amazing to see we now have people who biggest success story, I believe is a 45 year old woman who worked for us for over 15 years as a customer service rep, who now is a full grown data scientist with us, who has actually doing some phenomenal work. And really, really you wouldn't know that she would be, that she was in customer service for years. You would have no idea. Yeah.

That's incredible. My next question you could I suppose answer in the framework of her story or other success stories. I'm wondering about the, how you think about the relationship between using platforms such as, such as datacamp, which as you said, has been very successful for you and in person training, and how these two can complement each other.

Us datacamp was invaluable. We couldn't have done this without datacamp and hopefully we need a datacamp. What we did is we did complementary training definitely. So we basically took datacamp courses, graded them in terms of difficulty and then put them together in a logical order, which now I believe Data Camp has themselves. They, they call it tracks, I think. But back in the day they didn't exist. So we did it ourselves. We did these tracks at the end of each track. We got together in a group, we did group sessions. We brought up very funny problems. You know, we brought up interesting problems that we found often, maybe, maybe like a few problems actually. Real Pinnacle problems, real Pinnacle data. And we, we did some analysis over it and we showed the people about how, how efficient this can be. And what I was, I was trying to sell people was that R is nothing else than Power Excel. I tried to take the fear away from being in an interface where you have to type in something. I just tried to always bring it down to them in terms of Excel. This is just like Excel, but instead of being able to work on 60,000 data points or 60,000 rows, you can now work on a few million rows without sweating a beat.

That's awesome. I do wonder how things would have been different if it had been called Power Excel, that type of branding for an open source language such as R.

To be quite fair, prior to the tidyverse, I don't think there would be a fair clarification. But if you break down the tidyverse and you actually break it down to dplyr and ggplot and just those two, I think cover 95% of all data work needs. Everything else is specialists in many aspects. But you know, DPLYR and GGplot, that's all you need to do to do the vast majority of work. So we only teach basically dplyr, ggplot markdown because the company is markdown based. So we're sending out markdowns, we're sending to each other. And so we now in this nightmare scenario where someone sends you data and you don't know where they pulled it from, you don't know what filters they put on, you have no idea. Now we have a markdown, you can just look through the code, you see exactly what they did and you can point out, oh well, you cannot do this, you've forgotten this and this scenario and can help them right away to do a better job.

Absolutely. So this particular case, your colleague who moved from customer support to learning a lot of our using data camp and in person training at Pinnacle, then moving to a data science role, how much math did she need to pick up or statistics or machine learning or this type of stuff? Because I understand using dplyr and ggplot2 the classical data analyst in data science role, but then there is another step above that, right?

For sure. So I believe we now have given her training on forecasting models, almost every techniques just to get her into a different mindset. So obviously she doesn't have the, the traditional training, so. Yeah, but data analysis is such an important step for many companies for many of our areas where they just lack basic data analysis. Your help. So she was productive. I think from day one. We actually put her into her old environment in the customer service field and she was writing the framework and reporting framework for the customer team because she was obviously a subject matter expert, that she's been on the front line for years and years and years. And now how we, you know, a classic example is we've redone our staffing. We realized that we had our some of our local speaking customer service agents working at the wrong hours. Maybe not when the local speaking languages, let's say like the Swedish speaking customers were having questions, they were working at the wrong hours. So we now were able to match these two data sets with each other and Actually optimize our scheduling to service our clients better. So direct win for our clients and direct impact from her.

That's incredible. So you've spoken to a lot of different modeling techniques you use in forecasting, including a lot of machine learning. I'm wondering, like, we all still write a lot of code to build machine learning models and that type of stuff, but speaking to your colleague, you know, the work she's been doing to impact the business, there are a lot of people worldwide who can impact their businesses developing machine learning models, but may not be able to code. This is kind of a roundabout way of asking you about how you think about machine learning as a service and as a platform going forward. So people in businesses able to build machine learning models without necessarily writing code.

It's starting for us like this for sure, because some of us build frameworks for others where they actually don't really know what's happening underneath the hood anymore. But underneath the hood, there is machine learning. We're going to see the shift over the next years for sure. And it's also a good thing. Just because you're driving a car does not mean you need to know how the combustion engine works. That is not a requirement. And you can drive a car perfectly fine and you can do your job with it without knowing the inner workings of the combustion engine. And this is what we're going to see with machine learning as well. Yeah. Other people are going to do it for you.

Absolutely. But of course, what we need in place are checks and balances and processes so that when your car busts up, it doesn't explode and kill you. Right.

I've seen data analysis gone wrong in our company many times over, quite obviously. A very famous one is obviously if the subject matter expert forgets to tell the model, you know, that there are certain parameters which cannot exist. Like a classic thing is like a volleyball game is the best of three, and so they cannot be a fan, fourth set or a six set. So, sorry, you know, because it's impossible. That never exists. But mathematically you can easily forecast this. Right. You put some density, some Poisson distributions on it, whatever it is, and. And you get a distribution for it even. Yeah, okay. Yeah, but that's a classic thing where you have to train people how to. How to tell the model a little bit better. What are the frames, what. What's the framework that you're operating in.

And I love that you use the term data analysis gone wrong. I think we should have a segment on the podcast at some point in the future. Called data analysis gone wrong. Nightmare stories. So, speaking of data analysis and data science, I'm wondering what one of your favorite data sciencey techniques or methodologies is.

My favorite one myself.

Yeah? Yeah, just something you love to do.

You know, in my day to day, you know, because I'm actually too far detached now from this. I mainly stick to the tidy voice myself. Put some data graph a little bit around and then dig deep, you know, that's. That's all I do when I speak to the actual quants, you know, like think. The one that I always like the best are some genetic algorithms. I just find them so cute, how they develop and how they stumble around, you know, I just love watching watching models grow like this and eventually of something that for a long time has such poor results, eventually just like exploding and producing results which are far greater than you would ever expected.

That's really cool that we've got. You mentioned genetic algorithms, Bayesian inference and Bayesian updating machine learning models. I presume when thinking about time series forecasting, you think about ARIMA models as well. Is there anything else that is kind of the bread and butter of building these types of models in your line of work?

Bayesian affiance is huge for us. That probably is maybe our better balance in many aspects. Just because we have a classical lack of data. Very classical. Like every game is so short, right? So you have a classical. You never ever have sufficient data. So basing in films is very important for us. That might be the biggest one that.

We use just for fun. What are some current bets that you have that you're really excited about or you find cute or interesting?

Oh, I actually don't know. Like, I always like the Game of Thrones one. That's something I always liked. But we always try to pull some fun bets, you know, just. But it's. It's fun bets just come around that some of us talk about an event and then we post some odds. You're mainly sometimes to see who's right between us as well. You know, we're just guessing ourselves. It's not, it's not unheard of that people are betting on Game of Thrones events, you know, in the company, just because it's fun.

And you may not be able to answer this, but who's your pick to be on the Iron Throne at the end of Game of Thrones at the moment?

I'm going to go with Cersei, maybe. Yeah.

Yeah, right. Cool. Has that changed over the past several seasons or. You've been a Cersei stronghold for some time.

I think Cersei is. Cersei is so evil. Let's see. We're going to know the truth next year.

You know, we definitely will.

So let's see. What do we have? Let me actually check right now. Game of Thrones. I can actually tell you who's the favorite.

Please do.

So the favorite is. So we have Jon Snow, Daenerys. Oh, wow, Bran. Oh, wow, Bran.

Interesting.

Those are the three favorites.

Interesting. I wonder why Tyrion isn't a favorite. But I think we're dyke digressing now. We can have a Game of Thrones episode when it comes up. So my final question, Marco, is do you have a final call to action for all our listeners out there, something you'd like to see them do or implement moving forward in their data science careers?

To me, it's just like teach people. If you are in data science, help your colleagues to become data scientists. Like, empower everybody. It will make your life easier. It's not going to make their life, it's going to change their life. Just try to teach. Teach as much as you can.

I couldn't agree more. Marco, it's been an absolute pleasure having you on the show.

Yeah, it was a pleasure. Anytime you. If you ever want me again, I'll be a bit of it.

Fantastic.

All right.

Thanks for joining Our wild ride with Marco about the role of data science in sports betting and bookmaking and how he's building an army of data scientists and democratizing data science and analytics. So as a low margin, high volume bookmaker, a little bit like Walmart, Marco says Pinnacle doesn't want to make a lot of money selling orange juice. They just want to buy make a little bit. But they sell a lot of orange juice to do this. They're adept at using all the data they have, which may not be a lot, such as when Germany played Uruguay in the World cup, but as many superforecasters do, they're ruthless at updating their models and predictions in light of new evidence. And Bayesian modeling is a fantastic framework for both of these things. In terms of coding, Pinnacle is an R shop, and Marco, having come from the XL world himself, proselytizes the gains in few a efficiency, performance and scalability that R allows. On top of this, he busted the myth once again that R models aren't really scalable in production. He also told us about how he has used Datacamp to train over 150 colleagues to become more data fluent and spoke of one colleague who moved from customer support to data science and has already had direct impact on optimizing scheduling to service Pinnacle's clients better. I can't stress enough how essential that this is. Data fluency is becoming a skill spread more and more across organizations, and not only in the hands of the few. Next week I'll be speaking with Reshma Shaikh, a freelance data scientist and statistician who works in Python, R, and sas. Reshma is also an organizer of the meetup groups Women in Machine Learning and Data Science, otherwise known as Wimbleds and PI Ladies. She's organized wimlds for four years and is a board member. We'll discuss her work at wimlds and what you, our listeners, can do to support and promote women and gender minorities in data science. We'll also delve into why women are flourishing in the R community but lagging in Python and discuss More generally how NumFocus thinks about diversity and inclusion, including their code of conduct. All this and more. I'm your host Hugo Bown Anderson. You can follow datacamp on Twitter @datacamp and me Hugo Bowne. You can find all our episodes and show notes@datacamp.com community podcast.