#59 Data Science R&D at TD Ameritrade

Hi, I'm Maggie, VP of Curriculum at.

Datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for.

Developers, and everything in between.

Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works.

Power your data and AI transformation journey.

On datacamp.com.

This week I'll be speaking with Shawn Law about data science research and development at TD Ameritrade. Sean's work on the exploration team uses cutting edge theories and tools to build proofs of concept at TD Ameritrade. They think about a wide array of questions from conversational agents that can help customers quickly get to information that they need and going beyond chatbots, as well as using modern time series analysis and more advanced techniques like recurrent neural networks to predict the next time a customer might call and what they might be calling for, as well as helping investors leverage alternative data sets and make more informed decisions. Now, what does this proof of concept work on the edge of data science look like at TD Ameritrade and how does it differ from building prototypes and products? And also how does exploration differ from production? Stick around to find out. Welcome to Data Framed, the weekly Data Camp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host Hugo Bown Anderson. You can follow Datacamp on Twitter @Datacamp and Megobown. You can find all our episodes and show notes@datacamp.com Community Podcast this is Dataframe. I also wanted to let you know that in the show notes you can find a form to make guest suggestions for the podcast. We welcome all suggestions and encourage you to have as panoramic a view as possible with respect to the diversity and inclusivity of our guests. Also, if you enjoy the podcast, it would be great if you could leave us a rating and a review on itunes, but only if you enjoy it or read out some of our favorite reviews on the podcast. So write one for your chance to be quoted. Now it's time for the show. In other news, I've managed to train an artificial intelligence to automate some of my work, such as hosting this podcast. This is the very first episode which is hosted by an AI Hugo trained on all previous episodes and you're listening to him right now. Hi there, Sean, and welcome to Data Framed.

Thanks for having me, Hugo.

It's a real pleasure to have you on the show, and I'm really excited to have you here today to talk about kind of the research and development sides of data science, particularly as it pertains to your work at TD Ameritrade Financial Services and brokerage. We're going to get into all of that, but before that, I'd like to find out a bit about what you do. But in an org such as TD Ameritrade, I'm sure there are a lot of different impressions of what data science and R and D do and are capable of. So I'd like to start off by having you tell us a bit about what your colleagues think that you do.

Yeah, so I think that a lot of times people know that or are aware within the organization that I have some sort of scientific background, research background, and so they think that I spend the majority of my time sort of dreaming up crazy new ideas or exploring, like, the deepest, darkest areas of research. In reality, while I might do some of that, I kind of boiling it down to sort of one sentence is that I tend to ask a lot of hard questions or interesting questions, and that's usually where I start.

So you ask a lot of hard questions. Do you get to answer some of them along the way as well?

Hopefully so, yeah. Otherwise it wouldn't be that fun of a job. But I think what is important is to ask a lot of hard questions, have a good amount of skepticism, or at least a healthy dose of that, to understand where people are coming from, what problems that we're actually trying to really address. Because sometimes when you peel back certain layers, you start to realize that maybe, perhaps either the underlying question is not well posed, or the underlying problem is either extremely hard or maybe perhaps even impossible to solve. Now, it doesn't mean that we need to give up on it immediately, but I think sometimes having a good amount of empathy as well as listening skills goes a long way, especially when it comes to data science.

For sure, I like this trope of kind of the crazy research scientists still exist in a lot of ways, but one thing we're here to do today is to demystify that. So you've actually told us a bit about what you actually do, but maybe you can go into more detail about what your job entails.

I work in a. Essentially a research and development group. The name of my team is Exploration Lab here at TD Ameritrade. We used to be called advanced technology. But I think more and more we've been focused on thinking about how different areas of research that sometimes are unrelated to finance could be applied in the finance sector and then thinking through maybe what new experiences customers might have. I think one thing to clarify is a lot of times when people hear that I'm in finance, they will immediately ask me to provide advice on financial markets or investment advice. I'm not a financial investor.

Well, it's funny that you mentioned that because ETFs went down in December and I was wondering if you could tell me a bit about. I'm just kidding. So yeah, go on.

Now, I'm probably the last person that you want to be asking for financial advice, probably because I'm not as qualified as many others are. But at the same time, I think that also provides a unique perspective where I'm sort of unencumbered by some of that history and I might bring a fresh perspective to things. And so being able to look out for again those new experiences for a customer is very important. And while that might come in the form of providing some advanced analytic methods to our active traders, or maybe to provide a more personalized experience for our retail customers, or even just to simply improve the process efficiency for our institutional side of the business, which is our advisors, there's really a lot of great opportunity, especially in terms of how we can leverage technology. So the focus for us is leveraging technology. But nowadays leveraging technology sometimes isn't efficient. And that's where sort of the data side of things come into play.

And so what's the trade off there? Or let's say the balance between like doing research, talking with relevant stakeholders about the business problems, sitting down and just thinking the act of discovery and all of these things.

That's a great question. I think initially, you know, as I transitioned away from academia into industry, I thought to myself that, you know, if I, I'm a hard worker, right? If I put my head down and sort of churn out, work 90% of the time and spend 10% of my time building relationships and, you know, attending meetings and then I'll do great. But very, very quickly, having been in industry for maybe three months to six months time, I quickly realized that that's not sufficient really. It's a juggling act where you have to spend the time effectively, not only to your point of doing some of the research, doing some of the experimentation, managing the relationships, and spending a lot of the time actually understanding what is being asked from our business stakeholders and being Able to even read between the lines, do your due diligence. Before even writing a single line of code or even thinking about how to solve the problem, you have to really, truly identify what problem is it that we're even trying to solve? And is there a problem at all? Maybe some of it is. Sometimes when you ask somebody what are your pain points? People will tell you what's top of mind. In reality, for our team, what we really wanted to do is think about problems or new opportunities that are that sort of span across the entire industry. If TD Ameritrade is the first one to solve that problem or begin shaving off some parts of it, then that's sort of a huge win for us. And again, we're doing exploration. My team, we're not doing production.

Right.

We're really on the research side of things and we might build out some initial proofs of concepts to prove out the potential of either a technology or in the data science case, a certain methodology that might be applicable. But we're far away from production typically.

Yeah. Interesting. And something you spoke to in there, I think is, you know, when people have a problem or think they have a problem, it's actually a question whether data science and. Or analytics is the correct way to answer it as well.

Right, Absolutely. And that's where I touched upon at the beginning of having a health dose of skepticism. Right. And not necessarily in a negative way, but to make sure that people are bringing the right problems to the table. And again, for our team, the team that I work on is actually composed of a very diverse set of people. So I am one of maybe two data scientists on the team and we're a team of about maybe 12 or so. That sort of fluctuates over time. But the rest of my team have skills such as ux, user experience, design. We have somebody with cybersecurity in the past, front end, back end, mobile, AR, VR, AI, you kind of name it architecture. We've got somebody who has some level of expertise or considerable amount of experience in all of those fields. And that makes it a very collaborative environment for our team to explore in.

That's really cool. How big is the team?

So about 12 or so people.

That's a really nice size with those types of skill sets. It does. So I want to ask you about a challenge such teams can present. Part of my background is working in a cell biology lab in biophysics and thinking about cell growth, cell division, these types of things. We had physicists, chemists, biologists, mathematicians, such as myself, all working Together collaborating on cell biological questions, which means that you get a lot of different points of view, in essence, a lot of different types of creativity and knowledge. Everyone brings a lot to the table. But a problem with that is that sometimes you're not having the same conversation.

Absolutely. And maybe that's the benefit of having myself come from an academic setting where we had a. You know, I worked in a large lab of 25/post and grad students and experimental researchers at the University of Michigan here. And many of them, to your point, were either coming from physics, engineering, computer science, mathematics, biochemistry. That's actually my background, doing sort of computational chemistry type work and being able to understand the different viewpoints and understand what people bring to the table in itself is a very important thing. And having spent a good amount of time there and realizing that there's no way even as a scientist or let alone a data scientists, that you can know it all, I think once you can set aside your ego and realize that the synergy that can be had with your collaborators can do amazing things. And I think that that's what, you know, having experienced that in the past when as we're building out our team here at TD Ameritrade, I think that's something that we very consciously keep at the top of mind and that we're not hiring the same people over and over again. In fact, my biggest thing is I'm not interested in hiring another Sean. Right. What's important is that we have that diversity of experiences and thinking that really helps push the boundaries of what our team can look at.

For sure. I'm not sure I could handle another Hugo Data camp, in all honesty. So are you actually hiring at the moment?

Right now we're not, but we always have openings up and coming in the future. But we also do have a data science team within the company here that also sits here in the Ann Arbor office with us. And we're actively looking. I'm also a part of sort of enterprise wide initiative which is an AI council. So I serve as an advisory member of an internal AI Council where we're looking at how might we apply data science and aspects of artificial intelligence, namely deep learning methods to help solve some real problems within the company and for those types of initiatives, I think that we're obviously going to be expanding into that realm even more over the next months. And so. So I'm sure that adding new team members will be a priority.

Okay, great. So we'll put a link to your careers page in the show notes and any other resources that may help people who are interested in opening a conversation like this.

Great.

You spoke a bit to your background in scientific research in a lab. I'd love to know just a bit more about your background and how you got into data science originally.

Yeah, sort of. Maybe going a little bit further back when I was growing up, I was actually coming from an Asian family. I thought that it would do me well to venture down the path of becoming a physician. And that never really panned out. I went to school for actually was a biology major with an emphasis on biochemistry. But I was actually doing that, being very data driven, if you will, in that I chose that major because that was the major that had the highest probability of being accepted into a medical school in Canada. But that didn't pan out. But in school growing up, I was always very, very good at math. So while doing biology, I actually did a minor in applied mathematics. And I spent a good amount of time doing research, summer research in a computational field or in a sort of geometry field, where I was exploring protein flexibility and how that has effects on how proteins bind to certain or different types of ligands. And once my undergraduate career was coming to a close, I was thinking about what was the next step, and it was recommended to me that, hey, maybe you should consider graduate school. And that was, up until that point, not something that I had never considered and. But looking into it, it was really a fantastic opportunity. So I know that you had several, I guess, past data framed podcast guests, such as Sebastian Raschka, Randy Olson, who all went to school at Michigan State, and that's right, that's actually where I went. And maybe a side note is that Sebastian actually sort of came after my time there. I overlapped at the time that Randy was at Michigan State too, but Sebastian actually worked in a lab that I also rotated in as well. So we have a lot in common.

Cool. And of course, yourself and Sebastian and Randy are all like strong members of the PI data community as well, which is really cool.

Yeah, yeah, maybe that's something that's important to us as well. So I'm one of the co organizers along with Ben Zeitlin and Patricia Schuster here in the Ann Arbor community. And we run the PI Data Ann Arbor Monthly meetup. It's hosted here in the TD Ameritrade office. And we really spend a lot of time thinking about what data science means and what value we can bring to the local data science scene, as well as, to some extent, the startup scene. In fact, last month, our speaker is a senior legal counsel here at TD Ameritrade, who was invited to give a talk. And she talked about a topic that was titled Privacy isn't Dead, which I thought was fascinating and something that as a data scientist is important for all of us to think about. It's not so much of necessarily putting your head down and crunching the numbers. There are people behind the data and it's important for us to always consider the privacy aspect. But all of our talks are recorded and posted on YouTube, so I invite everybody to go check them out.

Great. And we'll link to that in the show notes as well. So grad school, what happened after grad school?

During grad school I did a lot of computational work strictly in a dry lab. So I did computer simulations of like protein DNA interactions, some of the largest simulations of its time.

And that was molecular dynamics stuff.

Right, right, right. I think that there's some overlap with some of your Michelle Lynn Gill as well, who did some MD simulations.

And for our listeners, correct me if I'm wrong, but molecular dynamics is simulating stuff on a really short timescale, but all the interactions. And you need a lot of computing power to do this, right?

Absolutely. So I feel like a curmudgeon these days. Right. Because when, when, when I was simulations, which was realistically not that long ago, we were using CPUs in parallel computing on some cluster of computers. And it probably took me about six months to a year to produce several hundreds of nanoseconds or sub microseconds of simulations. And we're talking about simulation time steps of picoseconds here. And then now with the growth and usage of GPUs, people are basically reproducing simulations that I ran. Right. But within weeks, if not days. So, you know, people are maybe spoiled is an exaggeration, but definitely not.

And I do have this vision of you being like, back in my day, we never had gpus. We did went got by with blood, you know.

Right. I love it.

So yeah, grad school, molecular dynamics. Then what happened?

And so I, that's when I moved from Michigan State to University of Michigan to become a postdoc. So I was also in a very similar computational lab. In this case I was doing simulations of protein, protein interactions, protein RNA and what's sort of called coarse grain simulations where you can think about different scales of dynamics. So usually at the atom mystic level, you're looking at atom to atom interactions. But when you are able to scale that out using a more coarser grain type of model, then you're Looking at larger and larger dynamics and being able to study that. So I was looking at sort of binding of different proteins that might affect transcription. And so during that journey, during essentially my entire PhD and postdoc career, I think what people refer to as data science today, I was just referring to doing science out of necessity. Right. So things like applying PCA analysis or K means clustering to look at what protein structures look very similar to each other. What are some of those dynamics? People call it machine learning these days around my colleagues. It was just a necessity again.

Yeah, sure. And so then you transitioned from academia to industry, right?

Yeah, I think a large part of that, I think maybe perhaps even something that is not often talked about, especially when moving from industry into academia into industry. Right. Or even staying in academia, is what happens when you start growing up and becoming an adult.

Right.

And need to raise children. I think it was precisely at the moment that I had a child that I started thinking, what does life mean? Maybe perhaps an existential crisis. But in sort of pondering about whether or not I wanted to stay in academia and also what was a very competitive environment in the academic setting that I was in, I started thinking about, well, what would it mean to move into industry and to also open up the options, especially with funding, scientific funding being very, very challenging. And then also realizing too, that maybe for the rest of my life I'm not actually doing science, and that I pro probably end up either teaching, which I do enjoy, but also spending majority of my time reading papers and just writing grants and not actually doing fundamental research. That was a little bit depressing for me. And so I was very lucky. A postdoc that I had worked with who is now a faculty at the University of Michigan, Dr. Aaron Frank, who is a close collaborator, and he worked with me and he kind of asked me the question of, did you ever consider data science? And this was back in 2014, and up until that stage, I actually had never even heard of the phrase before. Right. Even with DJ Patel sort of popularizing that term again, data science. As I looked into some of the job postings that were out there, and I just kind of thought to myself, like, I feel like I have 90% of these skills that people are looking for, and the other 10% I would be absolutely interested in learning more about and having sort of that curiosity. And so that's when I made the change and transition over to industry and started looking for jobs out there.

We'll jump right back into our interview with Shaun Law after a short segment. So I'M back here with Emily Robinson, a data scientist on the growth team at Data Camp, to talk about her guidelines for online experiments and a B testing. Hey, Emily.

Hi, Hugo.

So we've done a bunch of segments on your super interesting guidelines for online experiments, and this is going to be our final one for the time being. I thought maybe you could just give us a rundown again into what online experiments and a B testing actually is.

Sure. So the goal of an online experiment is to measure the impact that a change you make to your site has. How you do this is you randomly assign visitors to your website into either the control, which is the old version of your website, or to your new version, which is called the treatment group. You know, you have this running, you're bucketing visitors this way for maybe a week or two weeks, and then at the end, you compare the behavior of these two groups. For example, maybe you checked there was a registration rate greater in the treatment versus the control. How about the conversion rate, the subscription rate, et cetera. And then if you apply the appropriate statistical tests and you find, say, there's a significant difference, you can be pretty confident that difference is because of the change that you made. Since by randomly assigning people and running the control and treatment over the same time period, that should be the only difference between the two groups.

Great. So we've said this before, and something you've written is that there are many ways that AB testing can go wrong, but most of them won't be obvious. And you writing out these guidelines was a way of dealing with a lot of these issues. So we're here to talk about your final guideline today, which is to focus on smaller incremental tests that change one thing at a time. Could you give us kind of an example of this and why it's important?

Sure. So one example is that is given by Dan McKinley, who is a former principal engineer at Etsy, and he gave a presentation on continuous experimentation. So his team spent weeks working on enabling Infinite scroll for the Etsy search page. So if you just kept scrolling, more items would keep popping up. But when they ran the A B test, they found actually it performed worse in terms of conversion. And their first reaction was it must be a bug. But they found some and the results still remain the same. And the problem here is that when you have this really big change, it's hard to figure out why it didn't work. So in this case, they actually went back and started testing some assumptions behind their belief that Infinite scroll would be better. One of these assumptions Is are more items actually better? But when they changed just the number of items on the search page, they found that while there are more clicks, it had the same number of purchases. Another question is, are faster results better? And it's hard to speed up a page, but it's pretty easy to artificially slow it down. So they tried slowing it down a little bit and found it didn't really hurt anything. If they checked these assumptions first, they would not have invested these weeks of development effort into infinite scroll. And so they learned from these mistakes and instead focused on making smaller cycles of design, develop and measure with a B test and having those add up to a big change.

And this really speaks to another point that we discussed recently on the podcast, which is to have a lot of stakeholders involved in all steps of the process. Like we talked about having a data scientist analyst involved from the very start and the principles of design and needing all those inputs there.

Exactly. And a data scientist also might have been able to say, so this was a case of one really big change. Another tempting thing is to bundle a bunch of smaller changes. So say, redoing the hat of your search page, but also the layout and how the filters work and saying like, oh, I just want to try this really beautifully new redesigned page. It's definitely going to work. And then when it doesn't, it's really hard to figure out why. Because was it just one part that failed? Was it the interaction of the changes? And, you know, a data scientist maybe give you some signal, but they're not really going to be able to split that apart.

Once again, this speaks to another maxim of yours, which is keep it simple, right?

Yeah, keep it simple.

Cool. So, as I said, this is the last segment for the time being on your guidelines to a B testing listeners. If you find this type of stuff interesting and stimulating and something you could incorporate into your own practice, I definitely suggest you check out an episode from last season of Data Frame, which is a conversation with Lucas Vermeer, who runs online experimentation@booking.com and Emily, you actually initially introduced me to Lucas back in the day, which is great.

Yeah, Lucas is great. Yeah. Booking is like one of the biggest companies running online experiments at scale, and they have just hundreds and thousands running at all times. They have a huge experimentation team, so they're definitely experiment goals.

Yeah, there's a lot happening there. And also we've got some courses on A B testing and online experiments at Data Camp, so check those out as well. But Emily, I just wanted to say thank you so much for Joining me once again for such a bunch of wonderful segments.

Yeah, thanks, Hugo. I really enjoyed doing them with you.

After that interlude, it's time to jump back into our chat with Sean. Great. So I want to pivot now to talk about your work at TD Ameritrade. But before we do that, I'd like to speak more generally about data science in these types of organizations. I'm wondering what aspects of financial services and brokerage at TB Ameritrade you think data science can have the largest impact on.

I think the biggest area of opportunity for us. Right. Is personalization. And also to, in that same regard, it is looking at how do we take the large amounts of data that is out there. Right. And to help boil it down to essential information that is pertinent to individual investors, whether it be for our long term investors or even for our active traders. And now, obviously these problems or these types of problems are on different timescales. Right. Where our long term investors might be looking at things that are over months, maybe even years, our active traders could be looking at things at the sort of minutes to hours to days time scale. And so when it comes to that personalization, we get into the realm of things like recommendation system, so a la Netflix and other folks. And so we've been definitely exploring some of that. But even earlier when I started my tenure here, we were thinking about things along the lines of even looking at natural language generation and even natural language understanding. So a lot of NLP type work, understanding when customers call us, what are they calling us about, and being able to take even phone calls and trying to predict what people might be calling us about.

Right. And so in that case, we're talking about conversational software of some sort. Right?

Yeah. And that's also interesting too. Right. So conversation can happen whether it be on a phone call, but it can also occur on some sort of chat platform, whether. So for our customers here at TD Ameritrade, we have something called ted, which is basically an online chat agent. But we've also built on here at TD Ameritrade, a Facebook chatbot as well as a Twitter bot as well, that our customers can interact with. And a lot of that initial work started off on the Exploration Lab team.

Right. That's really interesting. And it reminded me two of my collaborators, Jacqueline and Heather Nolis, who are a data scientist and machine learning engineer at T Mobile, respectively. I might get this slightly wrong, but the message is the same. They're working on conversational software and machine learning AT T Mobile, among other things. And there was a simple problem that people would come and try to log in and have a conversation with an agent, but answer a few questions beforehand. The person, the customer, would initially say, hey, there's a problem with my bill. And the follow up question would be, are you a customer? Because that's the first question that's asked. And of course from the initial statement it's clear that they are a customer. So even solving small challenges like this are incredibly helpful. Right?

Yeah. NLP is an extremely, extremely hard problem. Right. And that's why we have tons and tons of academic researchers that have been looking at this for decades. And it definitely poses a problem for us too, being able to understand the question that's being asked, but also being able to manage the context of that question as it relates to not only the current conversation, but also as it relates back to the customer's journey. At TD Ameritrade, I think the holy grail for any sort of conversational agent is to be able to tie together all of the touch points that a customer has had with us and be able to know exactly what you want without you even having to tell us. Right. That would be a bright day for humanity in general.

Yeah, I always joke, and it's not a joke at all actually, that NLP must be hard if we still use regular expressions.

Yeah. Now you have two problems.

Right, Exactly. So tell me a bit more about, I suppose, the brokerage aspects or the financial aspects. I presume you're probably deeply interested in thinking about time series analysis and prediction, this type of stuff.

It depends, I think on our team, I personally have explored a proof of concept looking at trying to spot patterns within a time series. And one might think that it might be for predicting the financial market, but in fact it's internally for us. It has varying and general applications to look at. Finding patterns of, let's say our server outages or our server resilience and being able to spot that, hey, priority XY and, and prior to some strange event, we noticed that the memory usage was very, very high. Being able to see that certain number of cores maybe were down and spotting those patterns ahead of time and finding early indicators that will allow us to become even more resilient, just so that at the end of the day we're better able to serve our customers. Other alternative areas of application would be to look at even for our customers themselves, what is their pattern of usage of our platform and being able to identify more clearly that are there certain steps that people take before they sort of reach the next milestone of their investment journey. But again, what we've built here, and hopefully we'll be able to talk more about this in the future. But we looked at some research that was actually conducted at UC Riverside by a professor out there that looked at taking a time series and being a fairly long time series and being able to spot from it patterns. And that itself is a very, very tough problem because if you can imagine a time series with 10,000 data points, 100,000 data points, and without you telling it what pattern to look for, can you kind of slide a window across that time series? And at each window, can you tell me what is the top K or top N closest matches to that? And if you think about it, that's sort of like in itself an N squared calculation, depending on how many windows you have. And so it becomes computationally intractable very, very quickly. But this research that was conducted there was able to show that using sort of a smarter method, a smarter algorithm, that you can actually get these exact matches and do something with it. And so we leveraged some of that and we currently have a patent that's filed to apply some of that technology. But at the same time, at TD Ameritrade, we're working to open source some of the underlying code to allow the data science community to start applying some of this to their work, their time to years work as well.

Fantastic. And this is a great example because it speaks to I a huge part of your work which has been a through line here, but we haven't, I suppose, necessarily explicitly stated in that, you know, you think about particular questions, problems and challenges, but you want to draw results from research and work from all types of different industries and different disciplines. Right?

Yeah. And I think that's where again, my background comes into play. And also having pretty stereotypical data science characteristic.

Right.

Which is curiosity that everybody talks about. But in addition to that, what I tend to do is I try to find very niche methodologies that people are looking at. But also going back to what I was saying earlier about listening, I think time and time again, when I listen to people's problems and the problems that they're trying to solve, I start realizing that there's an underlying theme across all of them. And in one case it is, or in this particular case, it is trying to find patterns in the time series without expert human knowledge or domain expertise, necessarily. Right. And it isn't necessarily to do all the work for you, but it at least makes it easier for the human to direct their attention towards certain patterns. And so I think that's one thing that on this team we try to do a lot of which is to hear what people are saying and then trying to connect the dots and then also keeping an eye out for new and emerging methodologies or technologies that are coming in out. But at the same breath, I would say that sometimes there's not even a lot of new things. Right. If you think about something like deep learning or neural networks, those things have been around for decades and so they are now, as somebody has put it's newly interesting. I think that's something that Hilary Mason has said in the past too.

Yeah. So I also recall one aspect of your work is thinking about providing customers with alternative data sets.

Yeah, that's something that's very, very interesting. Right. I think that for investors, they're always looking for some sort of edge, maybe some inside scoop. Right. But at the end of the day it's data that will help drive your decision making or information. Right. Information by proxy to really what you're getting at is data. Right. And so thinking about not only are we reading news, right, are we reading opinion pieces, are we looking at how the prices are doing for certain securities, but also thinking about are there alternative data sets that might be indicative? Right. And again, this is not to say that past performance can help you predict the future, but at the end of the day it is, how do we as a company help provide our customer with the best experience? And that involves providing potentially alternative data sets. And as a research team, some of the things that we do is think about or consider what some of those alternative data sets might look like.

So maybe you can't speak to real examples just for IP reasons, but can you give us a hypothetical example?

Yeah, definitely. And maybe this gives you a glimpse into how sort of my brain and my thought process works.

Please let's enter the brain of Sean.

So maybe a couple of years ago I was literally driving to work and I was listening to NPR out of, out of most things I usually don't listen to music, but NPR was talking about how I think it was at NASA, they were releasing a brand new data set to the community. And this is satellite imagery that allowed you to look at pollution levels across a globe. And then that got me thinking, hey, I wonder if we can leverage this open data set. And so this was called NASA's Aura satellite data set. And so it's mapping high res images of air quality. Because if you can think about it, as an investor, I might be invested in some sort of commodity. So whether it's some it's either maybe soybeans or orange crop yield. And so if you're in a polluted area or have or in an area of high pollution that could affect your potential yield in the future. And so we can marry data, this type of image data, and we can obviously leverage things like convolutional neural networks, which is en vogue these days, along with some historical data about these commodities, and perhaps be able to provide a more holistic view of how the commodity might perform in the future.

That's cool. I really like that explanation and appreciate it because it goes all the way from stating the question or problem or challenge thinking about what type of technology and data that's being collected is appropriate there. And also, you know, the kind of state of the art technique. So that's really cool, Sean. We'll jump right back into our conversation with Sean Law after a short segment. Now it's time for a segment called Data Science Best Practices. I'm here with Ben Skranker, an independent data science consultant. Hi there Ben.

Hi Hugo. Thanks for having me back Today. I am excited to wrap up our running discussion of unit testing and debugging practices, at least for a little while.

So Ben, what's on today's agenda?

Did you know there is an art to debugging?

You mean that there's more than just unit testing and using a debugger?

Yes, those are the place to start. Hopefully unit testing helps you verify code and ensure that old bugs remain fixed. As an aside, when you solve a bug, you should always create a unit test which verifies that the bug has been solved. Today. Today I want to discuss how you should approach debugging. Having a good debugging strategy can help you catch bugs which otherwise could be quite difficult. This is particularly important with distributed applications. I really learned the importance of this when I was writing a large scale simulation for my dissertation.

So what is the art of debugging?

To me, it is a collection of questions I ask to get in the right mindset for hunting bugs, plus some techniques and tools to use. If your unit test fails and a simple pass through the debugger fails to solve the problem, then you have a more complex bug. For example, you could have a situation where multiple threads access a shared resource incorrectly and corrupt it. Step one is remain calm. If you are anxious about finding the bug, it will be harder to think critically about causes. Often you must think outside the box box because something you think is true is not okay.

So what comes after remembering to breathe?

Next, did the system used to work. If so, some change broke it. Hopefully your code is under version control. Then at least you can search through the various commits to find the one that broke it. In general, you should modify your code in a scientific way. That is Change one thing, verify it works, then change the next and verify it works, and so on.

So essentially there we're trying to find the exact change that created the bug, right? Any other techniques?

Another great tool is to explain how the code works to the rubber duck on your monitor. This is better than explaining it to a colleague because often when you start explaining the code you will have that eureka moment and need to rush to the editor. Your rubber duck won't mind if you break off and do this mid sentence. In a similar vein, it is helpful to diagram the system. What are the stages in your pipeline? What are the inputs and outputs at each stage? Did you leave out a data cleaning step, Forget to handle outliers, or rescale your data to get the correct magnitude for outputting results?

Those are all great tips, Ben. Anything else spring to mind?

Another classic technique is divide and conquer. Take a big complex problem and keep breaking it into smaller components until you can understand it. This is especially helpful if the bug is in a library or vendor application. Keep hacking out code until you have the simplest possible case to reproduce the bug. This will make it much easier to identify the source problem and get help. Obviously do this in a branch.

And what if none of this works?

If all of this has failed? I like to list all possible causes of a bug, no matter how crazy. Then I check each case. If none of them are the cause, then at this point I must think deeply and question assumptions. Something I think is true clearly isn't.

Any final tips? Ben?

A last tip. It is often helpful to add logging to a system instead of printing diagnostics. Logging is how you capture runtime information. Like the black box on an aircraft. A good logging module allows you to adjust the amount of information and control how and where it is logged. This is especially important in distributed systems where it is difficult to debug across threads and nodes. Here, logging the right information and keeping the system interfaces simple is crucial. In Python, the logging module is a great place to start.

Thanks Ben for joining me once again and discussing the art of debugging your code.

Thanks again for the chance to share some hard won wisdom with the data framed community.

In other news, we played an April Fool's Day joke earlier as this episode came out on April 1st. I'm not really an AI, it's just plain old Hugo. Now back to our regularly scheduled programming. Time to get straight back into our chat with Sean. What I'd like to do now is we've talked about the types of questions and impact that data science can have in TD Ameritrade. I want to know more about R and D. The fact that you don't necessarily build scalable infrastructures or end of the line products, but you build proofs of concept rather than actual products. I'm wondering what a proof of concept looks like and what's the distinction between them and prototypes and products.

Yeah, that's a great question. So the way that we view sort of this, this pipeline, right, is that product is something that tends to be, you know, put into the hands of some end customer. And now that customer could be anybody, right? That could be an external customer, but it can also be some, an internal associate here at TD Ameritrade. Maybe something that will help improve our internal process, something that will help a customer make better investment decisions. But for us, we're at sort of the opposite end of that spectrum, right? At a proof of concept. Concept. And at that level we are literally trying to prove out whether or not there is potential. And that means that tends to mean that there's a lot of risk involved or at least perceived risk because we're literally standing in front of a dark cave. That's sort of the analogy that I have where we're standing in front of our dark cave. We have this chintzy flashlight that we can only see 10ft in front of us. We know that there might be light at the opposite end of that tunnel or that cave, but we really don't know until we start venturing in versus if you're in production, you probably are pretty sure that not only is this what the customer wants, but that you have a scalable solution. Or maybe in the case of machine learning models, you have something that has relatively good precision and recall.

Let's just zoom in on and maybe this is what you're getting to. Let's zoom in on what it means to have potential. I mean, what's the sniff test or the rule of thumb? Or how does something have potential enough to go to the next phase?

For us, it is being able to have a reasonable internal use case that we're exploring along with the underlying technology or methodology and realizing that this is hopefully truly a hard problem that we're trying to solve and that if we're able to provide a minimum viable proof of concept. So MVP in this case is for minimum viable Proof of concept, then that might have enough legs to transition into what we're calling a prototype. And the way that we view this is that a proof of concept concept again is showing the potential that, okay, there's something there. So maybe to give your listeners an example is if I'm building a machine learning model that's maybe a binary classifier of some sort. And what I don't know is even where the data is to answer this problem. That's part of that risk that we're taking on. And so we have to explore whether the data is internal, whether we even have to collect the data, whether the data is in some third party vendor API that we need to connect with with, and then once we bring in that data, how much effort is it to actually get it whipped into shape to be used in some data processing pipeline? And then maybe afterwards, are we doing some analytics on it, Are we building a machine learning model and in the case of, let's say, a binary classifier, if we're building something out of the box, are we getting better than a coin flip? Now, we're not looking to target, you know, 99% accuracy because this is, this isn't a Kaggle competition. Right, Right. But what we're looking for is at least something better than 50 50, and then that will give us some level of confidence that there's perhaps some signal amongst that noise. And that's when we start thinking about moving it towards maybe a prototype where if truly the goal here is to optimize the model performance, and that's one thing, but perhaps 50, 50 or at least 60%, let's say accuracy is enough where previously we had no process and with no process in place, your accuracy essential. Right? That's sort of one way that we look at it.

Yeah. But of course the joke is that if you have a binary classifier and your accuracy is less than 50, then.

You just flip your predictions multiplied by negative one.

Yeah, exactly. So you also have a nice example involving the Amazon Echo Alexa that maybe you could tell us about.

On our team, actually a few years ago, we were exploring the idea of, even before things like Chatbots became very, very popular, even here at TD Ameritrade, we started thinking about communication. How might a customer want to interact with us? And that might be through, let's say, an Amazon Echo. And so one of our teammates here, Brett Frittori, built out one of the earlier proofs of concepts where not only can you interact with the Alexa, which is the assistant on the Amazon Echo, but you can Log into your account and be able to query, what is my account balance? What is this particular stock trading for? Or what is the price of this company? Or even pull some relevant news articles that might be relevant to you. When we start doing that, we start realizing that maybe this is one form factor. But is there opportunities for us to think about maybe a headless system that is device agnostic? Everybody carries around their personal mobile devices, but not everybody necessarily owns an Amazon Echo. Or maybe people own multiple different devices, such as Google's Home Assistant Assistant. And so one of our other team members started to think about, well, how might we make something that is headless? And so that opens up a whole realm of research discussions. Right? Because if you think about it, the first thing you do if you're going to roll your own is you have to think about, well, how do I capture the audio coming from the customer, assuming that they want us to, and then from that audio, how do we transcribe it from audio into text? And then how do we start parsing out that audio and thinking about what is the actual intent of the customer? And that gets to the realm of natural language understanding. Once you understand what the customer wants, you have to then go and retrieve the relevant answers. And that answer can be pretty complicated. You touched on it earlier about the context of a conversation. If somebody is already a customer and they're already authenticated in through our systems, then that experience is completely different from somebody who is unauthenticated.

Does the minimal viable proof of concept need to kind of demonstrate a proof of concept for each of these components of the pipeline?

You got it. That's where it's important to have good scope and that it's actually okay for you to fall out of each of these sections if things don't work or we find that it's actually extremely, extremely challenging. And now that doesn't mean that we completely abandon it. It might be open to further discussion or it's something that we might even consider rolling over to a more dedicated team. And so we've definitely had proofs of concepts on our where members have followed that proof of concept into prototyping and then also out into production where they became the tech lead on those individual teams.

What types of things can the proof of concept stage miss that's then caught in the prototype stage or even later. Top three things.

Yeah, I'd say the top three things would be first thing is scalability. Unless the proof of concept was looking at a technology that claims to be able to provide scalability. We try not to be bogged down by scalability because oftentimes that can squash projects from the get go. The second, the second thing is in the data science realm is model accuracy. We're not out of the box necessarily interested in getting the best model, but at least from a data standpoint, is there enough to provide a reasonable model? Or maybe from a methodology standpoint, if there's new methods that come out, such as some new deep learning architecture or maybe even cat boost or something like that, we'll test it out. And really the goal there is to show that the method itself itself works right beyond what is being published. And the last one is, the last thing that we try not to focus too much on initially is real world application. Of course, it's an ebb and flow. There are times when we definitely want to have a real use case in mind where we go out and build something, but at the same time we also, to the point of finding a time series pattern matching methodology, we might not necessarily have a use case in mind at the get go.

Okay, great. So scalability, model accuracy and real world applications. Fantastic. So we're going to wrap up in a minute, but we've kind of dived into a lot of different interesting questions that you think about how you approach answers to them, process and a variety of techniques you're interested in. But I'm just wondering what one of your favorite data science techniques or methodologies is.

Maybe this is a longer answer, but maybe from the interpretability standpoint. Right. I've always been a big fan of linear regression, but with like an L1 norm regularization. I've studied that and used it on occasion. And I found that that not only provides good interpretability, but sort of constraining the coefficients that come out of it can be very, very powerful. And I've seen some fantastic applications in the realm of predicting the potential energy of some protein states as well as looking at extracting background noise from videos, all using this type of approach. But I think I mentioned it earlier, some of the work that I've done on the time series pattern matching has really opened up my eyes a lot, lot because it's non traditional and I recommend that people look out for that git repo or reach out to us if you're interested in the future.

Great. And we'll link to it in the show notes as well. So do you have a final call to action for all our listeners out there, Sean?

Yeah, what I would say is in working in sort of data science and science for a good number of years. I think people tend to, especially in the marketing world, they tend to over glamorize, especially nowadays AI and artificial intelligence and machine learning. But I think it's actually very important for many of us practitioners to shape data failures. And so even at our PI data meetup, we always encourage people to not only point out where you succeeded, but also where things didn't work, because that's equally as valuable. And I just wish that there was when I was in grad school that there was a journal of things that didn't work that would have been helpful.

I couldn't agree more. And I'd love a journal of negative results. I think there is actually a journal of negative results, but we definitely need a lot more negative results published, particularly in biology, where I came from. There's such a reproducibility crisis.

Absolutely.

I love that. So remember everyone, data science and AI are you can learn techniques by working on problems step by step and problems that interest you. Sean, it's been an absolute pleasure having you on the show.

Thank you so much for having me. Hugo.

Thank you for joining the conversation with Sean. A deep dive into how the Exploration Lab at TD Ameritrade thinks about and implements building proofs of concept, all in the name of research and development. Built on a team with a wide array of skills, from mathematics and biochemistry to user experience, design, cybersecurity, front end, back end, mobile, ar, VR and AI, the Exploration Lab moves swiftly from simple hypotheses with high risk of failure to team review and then to research. Phase one example involved customer interactions through an Amazon Alexa, which involved a vocal interface, natural language understanding or NLU and text to speech, among many other moving parts. And such a proof of concept requires proofs of concept for each step in the proverbial pipeline. Wait, that isn't a proverb, it's a pipeline. Now, don't forget that proofs of concept in such R and D initiatives are not intended to catch scalability, model accuracy, or real world performance. These are caught in the prototyping stage or even later. Now, next week I'll be speaking with Angela Lee about spatial thinking in data science. Angela is the R Spatial Advocate at the center for Spatial Data Science at UChicago and also runs R Ladies Chicago Cargo. We'll be speaking about what spatial thinking means, particularly in an age when so much spatial data impacts us all. From Google Maps to Uber and Lyft to public policy involving inherently spatial questions. So many questions can be framed spatially, from urban planning and transit policy and government to placement of ads. Spatial data is everywhere. But when should we be thinking spatially? And what types of tools exist to deal with these questions? Join us next week to find out. I'm your host, Hugo Baun Anderson. You can follow Data Camp on Twitter @Datacamp and me Hugo Bown. You can find all our episodes and show notes@datacamp.com community podcast.