Recommender systems and high-frequency trading

Noticing the similarities in the workflow between a quantitative trader and an ML engineer on working on a recommender system is what kind of gave me the idea for the book. Oh, I thought, I thought, hey, you know, this is. You can describe this all as, you know, one instruction book for an engineer and they both could use the same instruction book.

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com Our feature flags are powered by LaunchDarkly. Check them out@launchdarkly.com and we're hosted on Lino Cloud servers. Get a hundred dollars in hosting credit linode.com changelog this episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI programming languages, DevOps, data science, cloud containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, Then check out O'Reilly's online learning platform. Learn more and keep your team skills sharp@o'reilly.com changelog again o'reilly.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@chain.com community and follow us on Twitter. You're Racticalai FM.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a principal emerging Technology strategist at Lockheed Martin. How you doing Chris?

I am doing very well. How's it going today Daniel?

It's going great. I feel like it's been a productive week. Made some progress on some projects and have some new collaborations going and that's always fun. So what about you?

Same kind of thing. I've been just yesterday was like a 15 hour day for work.

Oh geez.

There's all sorts of stuff in the air. Lots of emerging technologies, emerging technologies, you know, and we were talking in the pre show about something that I have such a deep appreciation for AI right now, because, as I think I've mentioned before, I'm doing flying lessons, and I'm in this ancient 1973 airplane, and every time I'm in it, I'm just like. I'm. I'm thinking about us. I'm thinking about deep learning and all the things that we're able to do, and just. Just wishing that, because I need some help, man. I need some help. Okay. I'm just saying.

Did we not have an episode about autonomous flying races and that sort of thing?

We did.

Wasn't that a thing? Oh, yeah, you did, like, you were, like, TV host for, like, autonomous flying thing or something. Right.

We had a fellow named Keith Lynn who was running the show, and he came on. It was a alpha pilot a while back, and we were doing.

That's right. Y.

So I just want to say I am doing the opposite of alpha pilot and really wishing that some of that amazing technology was available to me. So I'm just saying I just wanted to come to the show because we're practical AI and saying there are moments in my life I need a little bit more practical AI to help me out.

Yeah, yeah.

There we go.

Hopefully that technology filters down eventually.

Absolutely. I'm counting on it.

I'm excited today. Occasionally on the podcast, we get a chance to talk to someone who has authored a new book or updated a book or something like that. And, of course, I always love that because authors have put so much thought into the topics that they're passionate about. So it's really good to get their feedback and views on things. Today we have with us David Sweet, who is the author of a new book called Tuning Up From a B Testing to Bayesian Optimization. Welcome, David.

Thank you. It's great to be here.

Yeah. Maybe before we get into what tuning up means from your perspective, could you just give us a little bit of a background about yourself and how you got into doing what you're doing now?

Sure. I started out getting a PhD in physics. It was theoretical and computational, so I did a lot of.

Another physics guy.

I know.

I love that. I don't know what percentage we have on the show, but it is a high percentage.

And Daniel is a physics guy, too.

Yeah.

I'm feeling very left out at this point.

Physicists get spread far and wide. It's a. It's a popular subject, and I guess the world's only got room for so many professional physicists, so. Yeah. So doing a lot of computer programming kind of lent itself to an industrial job and either in kind of in finance or in technology. And I guess I was an east coaster from New York, so that kind of drew me towards finance a little bit. But ultimately I wanted to go into industry because I really wanted to build things that kind of work. Like, kind of like most engineers kind of have this story. They grew up as a little kid working with Legos or I had something called an Erector set or some other kind of building toy like this. The fun of research for me was that I got to do something new, but I felt like there was like, there's something missing in it that was like, the end is you get a paper and it's wonderful because you get to kind of communicate with the world, but you don't get to kind of wind it up and watch it go the way, like, Latoys. I built another. So I still had some of that desire, and that's kind of what brought me into industry and finance in particular.

So I'm curious. When I was in physics, it was like this urban myth that, like, you know, there was a way to go from physics to finance. And, you know, there were these, like, mythical physics people that made that transition. But it was very unclear to me at the time, like, how does that happen? And, like, of course, I, as a physics person, didn't really know anything about finance. And. Well, I still don't really. But, yeah, I'm curious, how was that transition for you? And was that something that, like, was on your mind for a while?

I can say in there are two factors there that mattered. One was that there had been people from my group who had gone into finance before. And so that kind of set the stage for headhunters to come back to the group or recruiters to come back to the group and look for. Look for people. So I got a call from a recruiter, and he talked to me about what kind of jobs there were in finance or physicists, and the one in particular he had in mind. But he had others in backup as well. And that's what really sparked my interest. I had an interest in finance from the perspective of personal finance investing. I had bought Netscape the day after an ipo. That was my first trade. And of course it did really well, but it was very exciting. It's, wow, you can buy something and it can go up really quickly. Most investments, most purchases or trades don't go like that. But it had the impact that it had. So that was the first step, was that there's people in the group. But then the second step was just going. I went on an interview just to sort of see what I thought about it. And the company was a small company with lots of computing resources. And the job was to build a strategy that traded a portfolio autonomously, more or less, almost completely autonomous, autonomously. And that was very exciting. That really was right up my alley. I get to do coding, which is something I enjoyed doing for a long time. You get to do math, building the models, but then you get this kind of final piece of satisfaction of just kind of watching it go do its thing, and it's pretty cool.

So did you reach that sort of end to end state?

Well, I was a junior person at the time, so the people who had founded the company had built a strategy. It was a single strategy. So my job was to work on this and improve it and enhance it. And actually I got to learn how that kind of thing worked and at the same time make some contributions.

So what happened after that? Did you end up staying in finance for some time or how did things kind of progress from there?

Yeah, so I was in finance for a long time. My next step after that was to go and try to build my own trading strategy. And it was at a time when day trading and intraday trading was something people were doing, lots of people were doing by hand, very successfully. It was right after the nasdaq.com boom. So I partnered up with a buddy from my research group in grad school and we just started building. We just went on the web and went to the websites for the exchanges and just figured out how to do this and built a small but working trading strategy. And it was super exciting. In part because we got to build our own thing, but in part, or build our own strategy, but in part also because we were building our own company at the same time. For the two of us, it was interesting that it was a totally different kind of experience from anyone I'd had up to that point. Up to that point. I'd always, and I think a lot of people have this experience going through school is you rely on your practice to give you confidence that you can do a particular thing that you've learned. When you go to build a business, you have to kind of build this meta confidence that you can figure out how to do new things, which is nerve wracking but super exhilarating. It was a lot of fun.

Cool, that sounds cool. So I can't help but ask, being the podcast that we are and having you tackled the finance side of things, what is the state of AI Because I don't think we've had anyone on the show in that specific arena. AI, in the world of finance and trading, how do you see it? What do you think of it at this point? What's the state?

Sure.

What I see from talking to, I have colleagues, you know, from the past that have kind of spread out all over to different companies from like early in my career people progress and they go different directions but we keep in touch. And what I see people doing quite a bit in short term trading strategies, well I'll back that up long term for different reasons is I see three things. One is I see linear models with lots of regressors and automatic feature selection and kind of like pre deep learning, you may call it linear machine learning. Right. So you're building linear models but not interactively and carefully building, checking each regressor but putting in lots of signals and automatically turning them into a compact model. I always think of Hastie and Tipshirani. They have this, this book out. In this whole methodology you build a large dictionary of signals and use sparsity and whatnot. So I see that, I see a deep learning used for interpreting maybe alternative data sets. Sometimes you know, short term trading where you get lots and lots of samples and you need to take something like the order book which maybe all your listeners are familiar with this. But like it's a complex data structure and so to turn it into something, to something you can predict with, you have to do a lot of feature engineering and you can either do it manually or you can use some kind of automated feature representation learning. The other thing I see a lot of is optimization of a simulation. Now this is like old school, this is like 19, maybe 70s on, maybe even earlier depending on how you, what you want to call simulation. So optimization of simulation is like a, is an old school engineering technique. You build a model, you build a simulator for it and you want to run a controller. And controllers are hard to solve. There's no quick and easy solution like a linear regression for a controller. So you run lots and lots of simulations and you see which parameters work best. And there are all kinds of ways to solve this. And so that's a lot of what trading is. I mean that, that's sort of like the, the end state of building a trading strategy right before you go and put it online. And then yeah, then there's a whole other state. But so I think those are the three big AI related things in terms of simulation optimization. Today you might see evolutionary algorithms where when I first started out, you might just have one parameter at a time and sweep through 10 values and make sure it looks pretty good to the eye on the graph and then repeat that process with each new parameter. Now you know, it's more automated black proxy.

Do you feel like on that front that in the sort of trading space there's been a sort of a time in which people are considering and trying all sorts of new things as related to these new types of models that are, that are coming out? Or do you feel like it's been pretty progressive and incremental in terms of the same sort of, of strategy but with maybe a different model like you're talking about? Or are people trying, you know, like I'm thinking of things like reinforcement learning or other things to, you know, try sort of whole new approaches to the problem.

This is a good question I have. As you were asking, I came up with four different answers. I'm going to go, I know, awesome. Talk about them, maybe I'll try to do them. Most interesting to least. But the one thing, there's this interesting cultural dynamic in finance which is in trading, I shouldn't say finance trading, specifically quantitative. I even narrow it down more quantitative trading where people, especially when they're new to the field, they want to come in and they want to try the latest and greatest algorithms and ideas and everything they've learned recently in school or from papers or whatnot and make some money and build the magic machine that makes a ton of money. And on the other side you've got people who've been doing it for a while, usually mentoring the younger people who roll their eyes at every new thing and say, ah, I know that's not going to work. Neural networks don't work. SVMs don't work. And you know, and sometimes they're right, sometimes they're wrong. I think if you say something's not going to work, you'll usually be right, but you just won't be productive. So it's one of the unfortunate aspects of the distribution of new quality of new ideas in engineering. So what I find is that I've seen, you know, I've seen people try or I've been one of the, tried all kinds of things. Basically, you know, everything that you, if you wanted to just randomly throw out ideas and probably some, I've seen somebody try it and some of the things stick and you know, some people figure out how to get things to work. The big problems with financial data are the signal to noise ratio is very low. The signals aren't just small, but they're competed away. The act of going and trading on signals, which your competitors are seeing as well, is squashing the signals. And so it creates this non stationarity where over time your strategies become less and less tradable, sometimes very quickly. And so you constantly have to adapt and look for new ways to predict or to trade. One thing that you mentioned, reinforcement learning that brought to mind, I don't think reinforcement learning is ready to just, you know, turn it on and get, get a usable answer out of in finance. I haven't seen that. And I say that only, you know, I say because it's, it's hard. I feel like it's still cutting edge for solving this kind of problem. I see a lot of promise in offline reinforcement learning. What's been going on the past year or so, it's just amazing. And it's, and it's very much in line with like a, it's like a machine learning replacement for the old or an AI replacement, I'll say, for the old school. Simulation optimization. Like how do you make that more automated or you know, autonomously very hyper automated, but get that next level of automation. So yes, there's a lot of problems I don't see. I haven't, you know, somebody was just kind of taking that out of the box, making it work. Contextual bandit, on the other hand, which is a limited subset of reinforcement learning problems. Not only do I think that that's useful, directly useful, but I think people in finance have been doing, doing it ad hoc for a long time anyway. You know, if not the most, you know, super efficient way it could be done. You know, the way people understand it these days, I think since the beginning of my career when doing things that kind of look to me like a contextual bandit. Yeah. I mean what makes that easier than a full reinforcement learning problem is that you're only predicting the immediate reward so you don't have to worry about your decision now affecting the state of the world for your decision later and then have this compounding of state changes based on previous decisions. That's more IID sample, so to speak to build your model with.

This episode is brought to you by our friends at Rudderstack. And we're calling all data engineers to check out rudderstack cloud and start building smart customer data pipelines. Rudderstack is warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs. Enabling all functions functionality of a CDP with more security and retaining full ownership of your data. It's open Source and API first. Rudderstack can be easily integrated into your existing development processes. And because they're open source, you can see all their code so you don't have to worry about vendor lock in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500000 events and pricing scales transparently. From there, learn more and get started@rudderstack.com again. Rudderstack.com that's R U D D E R S T A C k dot com.

So David, I know once upon a time you were working at Instagram, right? And you were working on recommender systems. And so kind of curious, like, what is your perception of the differences? You know, we've been talking about finance and applying AIML techniques in finance versus the social media world. The recommenders that you were doing before, what are they like in and what differences have you experienced? You know, even though that you're applying some of the same algorithms potentially in both areas?

Sure. That's actually noticing the similarities in the workflow between a quantitative trader and an ML engineer on working on a recommender system is what kind of gave me the idea for the book. Oh, I thought, I thought, hey, you know, this is, you can describe this all as, you know, one instruction book for an engineer and they both could use the same instruction book. So what the workflow looks like typically is first comes ideation and implementation. Right. So you get some idea of like a hypothesis. I think if I did this, it'll make their system better and you code it up and the next thing you do is you check it offline. Now in finance and trading, that would be like running a back test or simulation of trading and recommender systems. You might do a data analysis, a data science kind of analysis of the data, or you might run survey or perhaps have something kind of like a simulation. I mean you can read in papers that there are simulations for these kinds of systems and then the final step is you put it online. Mike Tyson has this saying something like everyone has a plan until they get punched in the face.

Someone just quoted that to me in Slack yesterday, by the way. It's really funny that you're mentioning that because I came up with a plan and that was the quote they gave me in Slack. So anyway, not to throw you off.

Everyone has a model until they put it in production and then things change. And so it's this final step of trading means you go and you trade and you say, wow, that doesn't look like I made so much money in backtest, what's going on or recommender systems. And yeah, let's say recommender systems. You look like you've made a better prediction of whether someone's going to retweet or click like or do whatever and you put it online and it didn't quite pan out the way you thought. You might not. Your predictions might not even be as good as they looked in the model offline. That can happen too. So this final step is experimentation or the. Either it's a B testing or it's some other type of an experimentation method. I think calm is super important and it's the common final piece. And for me I feel like it's the hardest piece in a lot of ways because you have the least amount of data to work with. But it's the most satisfying piece because it's the most accurate. It's the real system. It's not a model, it's not a simulation, it's the real thing. You have this trade off between the offline stuff can be precise, it's very quick to iterate, but it's got bias. Model bias, simulator bias. The live stuff is noisy but it's accurate because you're really doing it. And so that's what I wanted to write about. And so what I'm writing in a way that, you know, can be kind of read by quantitative traders or ML engineers.

Yeah, that's awesome. It's definitely a common scenario that people face and I think a lot of people just don't know what to do when they get to that place because they put all their time and reading and focus on this sort of offline testing that is also important. Right. We don't want to like get rid of that. But often people don't necessarily know what to do. Maybe before we get into like the details of some of that, could you clear up maybe just some jargon? So, you know, I've heard all sorts of like hypothesis testing, experimentation, like you use the term system tuning. Are those sort of all things kind of people use interchangeably to talk about this kind of online testing scenario.

Yeah, I see them all as kind of stitched together as one topic. So the idea of tuning I think harkens back to like an AM or FM radio that had knobs on it from like 1965 where you take the knob and you tune a little bit to the left and you hear static and you go to the right and the signal gets better but you go too far. It Gets noisy again. So you tune your radio into, you know, 88.7, you know, Kroc or whatever, and you say, get to listen to your music. So that, so if you, if you analogize that and you say your engineered system, be it the recommender system or trading strategy, all the parameters in it are like knobs on a radio, but instead of one knob, you have maybe 10 knobs or 100 knobs and you have to tune them all to get, to get the thing to work. It work its best. Then you're, you know, you're tuning the system. But, but, you know, you might also call that experimental optimization. Right. So you optimize these parameters with experiments, but it really comes down to more of like a complex, focused and efficient maybe way of the experimental optimization is of doing some experiments. Like an A B test would be like the most basic form of that. And you really use the same, a lot of the same techniques and mathematics and analysis, whatever to do.

You're making me feel old because, you know, in this context, a B testing is so old. And I remember back when I felt it was, you know, new and hot and so I'm just going to sit here quietly at this point.

Well, it hasn't changed that A B testing is the most robust, reliable and believable way to do things. So it hasn't gone away at all.

So maybe before we go on, could we just talk about what an AB test is for those out there? That.

That's a good point.

Maybe are wondering about this thing that we're all so hyped about.

Sure. So a B test is like the engineer's term for medical researcher might call randomized control trial, which might be a phrase more familiar to people in these pandemic days. So the idea is you have some way of doing things. Let's say you have some way of recommending new posts to somebody on Twitter, ranking the post, and you have some system that does this. You come up with an idea to make this system better. And so A is what you'll call the original system and B, what is what you'll call the system with your change applied to it. So now you have two systems, slightly different, and you go and you run them live and you see how the audience or the users respond to these two systems. So you have some, usually you'll have some kind of business metric like how long does the person spend on Twitter, how long do people spend on Twitter using version A and how long do they spend on Twitter using version B? And you want to compare and you want to see that they spend more time or well, maybe you want to see if they spend less time in this day and age. But typically in a business you'd want to more attention from users.

Yeah, yeah. And I guess in terms of how this intersects with the AI ML community, I mean, AAB testing could be something like you, like you were talking about, that can be done outside of, you know, just changing a user interface or testing a drug or you know, something like that. But in terms of like the ML AI space, I think something people don't often maybe think about is like, I have this model, I deploy it and now I want to work on like version two of the model. But how do I actually know that that model is going to perform better? You know, other than on the test set that I already have, how do I know that that model is going to perform better? So do you use AB test for, for those sorts of scenarios?

Yes, absolutely. So what you need like the A B test is sort of like the least you can do. It's the gatekeeper so that you don't ruin your system. Right. So if your V version doesn't perform better, you want to throw it away and don't let it be the new standard version of your system. And so it sounds like you're kind of alluding to is this problem people run into, we've talked about even before, where you try something offline and by the metrics you use offline, the thing you tried, let's say, let's say to be concrete, a machine learning model, right? And you've done prediction with say cross entropy. You've got a data set and your cross entropy has decreased. It's looking better than the old model. But you put this, this now this model online, but it's online, it's part of a larger system. So you're. So this is where I like to differentiate between pre prediction and control or supervised learning. And we'll call reinforcement learning for lack of a better term. But these are, these are kind of different, different ways of looking at this problem. So that in prediction that's what you would do offline with via supervised learning, a classifier regressor. And you maybe you predict a probability, someone's going to retweet a tweet. And when you put that online, you might use that probability along with other probabilities of the probability of liking a tweet or the probability of quote tweeting a tweet or commenting on a tweet. So you have all these kinds of probabilities and combine them to create an interaction ranking the probability of engagement, the probability of engaging with the tweet. And that's going to be the number you use to rank the tweets. You want to put the most engageable tweets first.

So just to connect the two, like if you're stuck with me back in 2005 doing a B testing and we're moving forward in time and we're starting to apply that, how do you make that transition? How do those integrate in. How do you. What's that path forward in a practical sense for starting to move forward on the calendar closer to the present day? And we're trying to. To implement that. I guess what I'm asking is, is that a direct replacement for AB testing when you're starting to do recommender? What's the relationship between the two?

I see. Okay. So where the AB testing would come in is you'd run the recommender system with the old model and compare it to the new, the A to the B. And so the AB test would be an experiment you'd run. You might run it for a couple of weeks or for a month and then see whether users are engaging more with your old ranking model or your new ranking model.

Gotcha.

And do you feel like this sort of mindset of experimentation is fairly. Is embedded enough in the technology space that, you know, it's not hard to sort of convince the product owners to let you run experiments in production and, you know, how do people perceive this sort of idea of experimenting online in that sort of way?

The risk of it. Yeah, yeah.

So it's my understanding that at the larger companies, this is the standard practice is everything gets tested into production, nothing just gets. Or typically things don't just get put into production just on a whim or based on domain knowledge or just based on the offline results. But they really experiments. And now the bigger companies, there's lots of money at stake, so revenue can be high. So a small percentage change up or down can be a dollar significant. Right. Or engagement or time spent can be high. And so a small change up or down can be significant. So the, these things are. They're all, they're all tested the way I understand it, I would think, I think it's not an uncommon. Less common these days, but not totally uncommon for people to have a feeling that they. An intuition that they understand their domain and they understand whether something is going to be right or wrong. Whether said that it's obviously going to. Something is obviously going to be Better and it's so common to be proven wrong. It's, it's humbling. You know, a person, a person who goes through that I wouldn't say is strange or arrogant. I would say they're typical. And I've done sort of an informal survey over the past 10 years of quant traders. I always ask them what fraction of the ideas that you come up with after you've tested them actually work and you know, end up in your system is a good idea. And almost everybody who's, you know, had some experience says 1 in 10 and just with kind of like shaking their head. It's just the nature of the beast that you. With a complex system, just because things can seem very reasonable but simply be wrong. There are just too many dimensions, too many factors to keep in the human mind. And you really need to just go and test.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog better.

So I'm kind of curious as we kind of started to get into some of the techniques. I was looking through your, your table of contents on your book and stuff and you have all of these techniques, most of which I'm not familiar with, and I was wondering if you could take a moment and kind of, kind of tell us briefly what each is and what that relationship is between those so that I'll understand. I'm super interested, especially when I see things like multi armed bandits and stuff. It's yeah, sure.

So the book is laid out in the order really it's laid out in the order in which I learned these things. But that also happens to be the order in which they appeared chronologically in the literature. So AB testing has been around, you know, I have 200 years or something, 100 years when it really started to pick up. The next one is response surface methodologies, maybe from the 50s or 60s. And so where response service methodology does is it focuses in the experimentation technique on systems with continuous parameters. Right. So like a knob that you could tune continuously and if you had a couple of these, if you tune them simultaneously, you can be more efficient than if you tune them one at a time. You go one at a time, you might say, you might tune one and say, well, I've got the Right value for this one, it's seven. And then you tune the other one, you say the right value for this is three. But then if you go back and tune the first one, you might find that six is a little better. And so you can bounce back and forth and there's no guarantee that you'll ever find the single optimum for both. So if you do them together, you can get the right answer. So the response surface literally means a picture of the function, the surface of the function of like your business metric, like revenue, let's say, versus those two parameters. So if you imagine a 3D plot with one of those nice colored or, or wireframe surfaces, you know, and you're looking for the, for the top of that, the highest peak in that surface. That's what response surface methodology is about.

Gotcha. Thank you.

Yeah, so that kind of limits you in some sense. Like AB testing you could apply to any two choices. You know, should I use red or green? And there's no real surface between red and green. That would make. Well, you do frequencies. There might, there might be, but there's, you know, we'll think about this categorical, you know, red, green, blue, there's no surface you plot there. But multi armed bandits, now they're interesting. They go back to instead focus on systems where again like a B testing, you compare any two options. They don't really take advantage of the continuity of a parameter, but you can pick, you know, any finite set of values and compare them with multi arm bandits. The real change from A B testing is that you're focusing on maximizing whatever your business metric is, let's say revenue, you want to make money while you're running the experiment. So you. With AB testing, I feel like, I think of it as more you really want the information, you want a reliable answer whether A is better than B. Right? Because people are going to die if you get it wrong, like with drug testing, so that the stakes are really high. You really want to get the right answer. With multi armed bandits you want to make money. And you can afford to be. You can afford to be. Nobody's going to die if you get it wrong. Good thing. And you can afford to change back and forth very quickly. The typical way it works is you take up your set of. Let's take a trading example. Let's say in trading you're trading 100 stocks, you might say 50 of them to start out. Dedicate them to strategy version A and 50 to strategy version B. And as time comes in, as the returns come in and you see A doing a little better than B, you increase the number. Say you're 60 of them, go to A and 40 to B and you just keep scaling then 70 to A and 30 to B. Rather than waiting till the end of the experiment, keeping at 50, 50 the whole time, you kind of scale it gradually toward the one that seems to be doing better. And as the data comes in and the error bars go down, you scale back and forth however you need until the probability of being in, let's say, B is low, under 5% or 1%, and you get to decide that percent as your threshold. So now you're talking about practical or business significance rather than statistical significance as your threshold. Your criteria.

Is it ever the case when you're doing that sort of test and adjusting things on the floor, let's say you have model A and model B and it's sort of a mixed bag where, well, it turns out over that time, actually the mixture of models was better than either model A by itself or model B by itself. Does that sort of scenario occur?

Yes, and it could also be that A model A and B running together can be worse than either A or B running alone. There can be interactions basically depending on what it is your testing. And it doesn't just apply to trading. So, yeah, that can be complicated. So that process can be complicated by these interactions. That being said, it would also be complicated if you're just running an A B test. The interactions would interfere. So you kind of have to. You have this assumption of IID independent identically distributed samples in the analyses through these kinds of algorithms, typically. But in reality, your samples are in either I nor I nor D. So they're either I nor I.

And that, I guess, brings me to my next question, which is, you know, typically we think of a B test one thing or the other thing. Do some of these techniques work when you have more than two options? Like let's say I have A, B, C, D and E. Or is it better to just always sort of assume you're going to test one thing at a time?

As a general rule, it's better to test. It's more efficient statistically to test everything at once. Right. For the same reasons I was talking about with the alluded to, with the two knobs. It's possible that moving to the left and moving up in your space, moving to the left in your space is good, moving up your space is good, but moving left and up at the same time is even better. If it's that kind of space, you're testing then you should do the same thing with a B testing and test them and test them all at once. The virtue in doing only two things at a time is that your test gets done faster. As you increase the number of things you're testing, the amount of data you need increases so your test gets done faster. You have, you'll typically have less noise simply because you'll have, your patients won't be tried as much. So you'll have, you'll get, maybe you'll get the data you need. It'll be easier to make a decision. It's also easier to decide between two things than it is to decide between more than two. So you just communicate it to your team or other teams. So there are benefits to doing the simpler thing for sure, but it's a trade off.

Gotcha. So having asked you about your response service methodology and the contextual bandit, how does Bayesian optimization, you have that in the title, so I feel like we're moving up toward that. How does that fit into the equation? What does it bring us?

Bayesian optimization takes bits from, from response service methodology and casual bandits and multi arm bandits, but it makes this super automated. By comparison, response methodology is very manual. You make the surface, you look, you make a decision where to go. And Bayesian optimization does the modeling for you, makes the decision about where to test next, what parameter values to test next for you. It's nice. And there's this interesting thing. You just probably heard the phrase exploitation versus exploration and multi. Armed bandits will do this. That's what they're doing when they're waiting towards the FA is doing better and they're adding more stocks to A and fewer stocks to B. They're taking advantage of the information that's come in so far. They're exploiting the information to make more money, but they're still testing B because they want to explore, to bring the error bars down. And so Bayesian optimization will do that kind of exploitation modeling and decision making for you as well. And to top things off, it'll also work with categorical and discrete parameters. So it really brings kind of the rest of the book together into a single. The ideas from the rest of the book together into a single automated technique.

Awesome.

So I'm kind of curious, I guess, just bringing things way down to a practical level. What are the sorts of practical tools that you can use to do this sort of testing? Is it things that are available in like familiar tool sets maybe to data science or AI people, like the Python ecosystem? Is there common stuff out there that people use and how easy is it to set up these sorts of scenarios online in a reasonable and robust way?

Okay, so for AB testing there are lots of tools, both open source and commercial. And when I say open source, you'll probably find tools that you, you can make use of in R or in like the numpy Python ecosystem, but they're also there also commercial products that can help you do this online. They where you can sort of submit results to a web service or the other ways of accessing. So AB testing is well known. It's the most commonly used of all these techniques and there's lots of support for it out there. On the other end of the spectrum, Bayesian optimization, there's a tool called open source tool called Axe that's written and supported by Facebook and that's very good. That would be the go to, I would think. I suspect Google's got something similar, although I'm not as familiar with it. I'm familiar with their paper where they use their optimizer to make better chocolate chip cookies. Vizier, it's called. And I don't know for sure, but there's an open source version of that.

So I guess as we've kind of gone through some of the things you've covered, what are you anticipating going forward? So as you know, clearly from early AB testing and stuff, as we started the kind of, you know, the progression and you've just taken us into Bayesian optimization and like, what are you thinking of for the future? What are you getting excited about working on or utilizing going forward? Do you have any sense of where the next steps might be, what might go?

Yes, what I'm most excited about is offline reinforcement learning. So doing reinforcement learning from small amounts of data and really the key to that, it's not so much the learning of the controller, you know, or the strategy or whatnot, but it's the learning of the simulator itself. That's where the real power is. Right. If I had a simulator, a learned simulator that I could believe I could optimize it in, you know, five different ways, one of which would be, you know, through a reinforcement learning technique like, you know, PPO or something more, you know, more recent than that, but learning the simulator from a small amount of data and understanding, you know, what results you can trust in production, which ones you can't, and where to go and maybe explore next to get data built to improve the simulated. These kinds of questions, being able to do that well could be super valuable. I know when you read about These kinds of things, the examples are often from robotics. They can provide a great visual. You can kind of get data pretty quickly, but they're industrial processes, I think everywhere. So many things work via optimization of a simulator that I think this would be super valuable throughout society. Yeah, I'm very excited about this.

Awesome.

Before we end here, I definitely want to let people know to check out David's book. But also we have a discount code through Manning, 40% off, which is pretty spectacular. So you can use the code POD PracticalAI19. We'll put that in our show notes as well, but that'll get you for 40% off. And I know, I'm really excited to dig more into this book, but David, it's been super fascinating and I really appreciate you taking time and explaining some of this stuff to us. It's. It's really good and it's important and like I say, I think it's something that people need to think about more and often are, you know, perplexed by in many scenarios. And I'm sure your book could shed a lot of light on those situations. So thank you for your work on it and thank you for the discussion.

Thanks a lot for having me. I really enjoyed this.

Thank you for listening to Practical AI. If this is your first time, make sure you subscribe so you don't miss a thing. Head to PracticalAI FM to subscribe or find us in Apple Podcasts, Spotify or wherever you listen to podcasts. And if you get value from the show, please do share it with a friend or or a colleague. We appreciate you spreading the word. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by Jared Santo and our music is provided by Breakmaster Cylinder. We are brought to you by some awesome sponsors. Shout out to Fastly Linode and launchdarkly. That is our show. We hope you enjoyed it and we'll talk to you again next week.

Sam.