[AI Access] Applied Analytics from End-to-End

Sam. Hi everyone and welcome. My name is Sandhya Simhan. I'm the Director of marketing here at DeepLearning AI. Welcome to our event series AI Access. If you'd like to check out our previous event on integrating design and technical innovation in AI versus products, you can find a recording link in the description box. Machine learning represents a powerful set of tools and techniques to make sense of complex process problem domains and challenging problems. But despite its power, it's rarely the whole solution to a challenge. To explore a set of those challenges and demonstrate how machine learning, distributed data engineering and a product oriented mindset can result in impactful results, we are excited to have Zachary Hanif, the Senior Director of Machine Learning at Capital One, join us today. Currently focused on leading teams to develop transformative platforms, data and analytic capabilities for Capital One's card business, Zachary is passionate about modern analytics and the infrastructure and platforms supporting and democratizing them. Before Capital One, he developed analytics and capabilities to secure and protect enterprises and consumers, focusing on cybersecurity, fraud and anti money laundering. Before inviting Zach to the stage, a friendly reminder for our audience who have registered the slido ticket. Don't forget to log into our Slido page to submit your questions or upvote others. You'll find the slido access link in the reminder emails we sent you. And now, without further ado, let's welcome Zach.

Hi everyone. Thank you.

Hi Zach.

Hi.

Thank you for joining us today. Well, thank you for having me. Before you start your present. Oh, I'm sorry, it seems there's a little lag before. Thank you for joining us today. Before you start your presentation, I'd love to do a little quick three question lightning round so the audience can get a few insights on how you've built your career in AI. So first off, how did you get started?

So I got started really a very long time ago. While I was still in my undergraduate doing my undergraduate degree, I got involved with data and analytics as applied to cybersecurity and I loved every second of that and it just kind of built up from there.

Awesome. Next, how do you and your team stay up to date with frequently changing machine learning techniques?

I think everyone has a different way of doing this. Some people really like asynchronous learning. So picking up papers, journals, things that they are capable of reading. Some take a more formalized approach, taking courses either online or in person. And other groups do it purely through kind of the act of working on material and the act of experimentation, working with the technology in and of itself, playing with it. Turning it into a game is some of the best ways to learn. So I think everyone's always got a little bit of a different approach. Myself, I do a lot of reading.

Sounds good. Different modes of learning for different learners, very much. And last but not least, what is some of the best advice you've received as a young machine learning engineer over the course of your career that you'd like to share with our attendees today?

I think it's that you shouldn't be afraid to fail. Right. And that's interesting advice because I think for much of our lifestyle, especially today, we're told that failure is inherently terrible and inherently bad. And I think that kind of loses a little bit of the beauty of computer science and a lot of the work that we do in this industry, which is we are an experimental science, we try stuff and sometimes it even works, and that's great. Now, don't get me wrong, I don't think you should fail for the sake of failing either. And you want to manage all that, all the real world considerations there. But I think there's a real value in not being afraid to try something and fail. And that can just lower the friction to trying experiments, trying new things, talking with people, and then learning from those experiences.

There's a great quote that I'm forgetting who said. But it's that we often forget that computer science is a science, that it is inherently a game of hypotheses research, hypotheses, testing, validation, and then repeating. So I think it's a good thing for most machine learning engineers and aspiring machine learning practitioners to keep in mind. Now we'll go ahead to your presentation. The stage is all yours.

Fantastic. Hopefully I'm able to share the screen here and nothing goes sideways on me. There's always something weird about technology when you do that. So, everyone, thank you for coming today. The title of the talk is Advanced Analytics. From end to end, introductions are always a little awkward for me. And so one of the things I've always done when giving talks is try to start with this one particular story. When I was doing my undergrad, I had a mathematics teacher who taught combinatorics. And on the first day, the professor entered the room, placed a pack of markers on the table, and said, with the kind of conviction most people reserve for talking about things like gravity and physics, that I am here on Earth to teach mathematics. And by way of introduction, that kind of a statement has stuck with me ever since. And so in the more local sense here, my entire career has revolved around collecting, working with and managing data, performing analytics, performing systems that utilize it, that collect it, that work on it, that sort and organize it and make those insights available to some kind of end user. I have focused primarily on machine learning based analytics throughout my career, but I wouldn't say that that has been the sum total of it because as I will get into over the course of this talk, as you try to bring machine learning into a production environment, make it a part of a product, do something with it that is able to affect the world in a positive way, you need to work with the stuff around it too. And so I'm currently at Capital One, I'm working on production considerations of things like data models, feature computation inside of our card line of business. I have been focusing for a couple of years now on bringing machine learning capabilities into the market. And so for a little bit, and this is a little bit of a change in my recent career path, instead of focusing on development time considerations, which is what the majority of my career has been focused on, I've been recently almost exclusively focused on the production side of the lifecycle. And previous to coming here, as was said a little bit earlier, I led teams inside of Capital One and other places to build transformative machine learning capabilities from beginning to end, from design time to productionalization. And I particularly have enjoyed my work inside of adversarial domains. I'd like to think that I've learned some things worth sharing and hopefully I'll be able to talk about them today. I'll warn everyone that I do write really dense slides, so it turns into a little bit of an eye test. It's for the benefit of future participants who might not have the benefit of either time or audio. I know some people in even some of my learning groups watch YouTube presentations of talks at very accelerated speeds so they can see the slides and how people react. So this is kind of based off of that. Should be a lot of fun. Before we get started, I've got one disclaimer that I have to make my experiences, all of our experiences are all different and it's moderated by the context around the problems that you're solving, the culture and technical expertise around your problem, and that's present in your company and industry. Kind of the temporal effects. When are you solving this problem? When you are looking at it based on where everyone else is around you. And so while I hope that some of my experiences can help you in your own endeavors and I'd like to, I think they generalize well. Please don't think for a moment that I'm standing up here saying this is the objective truth in all situations because nothing could be further from reality. And then two I try to avoid talking too much about specific technologies, but when I do, I'd ask everyone watching this now or in the future to remember that they're not advocations by Capital One who is my current employer. Anything that I'm talking about is really just a mention of my personal experience and opinions. And honestly, you can apply that statement to pretty much everything in this talk. It is an amalgamation of lessons that I've learned over the course of many companies, many domains, many problems, many challenges, and so not something that should be interpreted as oh hey, this is exactly what Zach's current employer is doing. So great. Let's start with kind of the first statement that I have, which is the context in which you are solving problems will inform everything you do from your initial design conversations all the way to production. My experience has been in applied machine learning. I am not like many of the luminaries in our field who are developing new theorems, who are proving direct branches of mathematics applicable to machine learning. I apply those lessons that they have learned, developed, and have shared with the rest of the community. I am not directly a theoretician myself, and so as a result I think of everything from the perspective of developing it to solve a discrete problem or set of problems, applying that in a product or service, and bringing that service to an environment where it can leverage itself against the challenges at hand. And so there's kind of three things there that kind of distill the lessons that I've learned, or at least several of the lessons that I've learned in doing so over the course of my career. It is that one analytics and machine learning does not happen in a vacuum. You need to understand your domain specific assumptions, failure conditions and responses that two when building your analytic capabilities, developers should have joint teams by skill set. So that means you're pairing statisticians, data scientists, machine learning engineers, software engineers, designers, modelers, all of it together whenever possible, and that you're able to ensure that they're engaged throughout the entirety of the development lifecycle. I've had interesting experiences in worlds where you have one group of people like say, software engineers, build a system and then kind of throw it over a wall to the data scientists or modeling teams and then kind of walk away and never talk about that again. I don't think that's a great methodology. I like seeing teams and I like working on teams where you have joint skill sets Together because you get a lot more than the sum of the parts. When everyone is working on the same problem store at least has direct visibility into it. And finally, three, I strongly advocate teams going through their own machine learning lifecycle, building machine learning teams or working on a product, using machine learning to not only focus on the algorithm, but to invest heavily in infrastructure, data and interfaces. And if you're going into production, if you're building a product or service, I almost advocate that you invest in those three things in that order. And I advocate to teams to be very, very deliberate in what you build into your pipeline and your product and what work you take on. I think overall that as modeling teams mature, they will move from artisanal modeling output effectively. Hey, we've worked on one model, and it's the thing that goes out to a world of productionalized capabilities effectively, where you're building these things instead of artisanal scales at industrial scales. I think the industry as a whole is seeing energy transition in this direction, making machine learning work scale dramatically. And I think we're seeing this huge wave of technologies that concern themselves with the infrastructure around modeling capabilities. I am super excited about all of this, particularly the idea of infrastructure supporting machine learning because it empowers automation, it lowers debt, both technical and business, and it speeds development. I think it's critical for our field to grow to the next maturity level. And I'm really glad to be here today to talk about at least just a little sliver of a fraction of that. Saying all that more simply understand your problem domain and how it affects and is affected by your analytic capabilities and your surrounding product and or pipelines. Control for debt by building teams that represent multiple skill sets, specialties and domain experiences. And then finally, when spending your energy and resources, you will get a great return on investment if you invest in your infrastructure, data and interfaces. And I would say in a biased way before anything else. All this varies based on domain context, but it's a pretty decent rule of thumb that served me well so far. Model developers, I have found, and development teams focused on models feel often like they've got to consider a large number of variables directly. And so they feel that things in the machine learning set of machine learning concerns are things from efficacy and algorithmic considerations to considerations like data access, how to serve the thing, what their control planes are, and how to deal with monitoring of the system on the whole. And that feels overwhelming for teams, especially when they're first starting. I think, however, that it is easy to lose sight of the fact that Machine learning teams are often a small piece of a much larger whole. I've stolen this from the paper Hidden Technical Debt in Machine Learning, which is fantastic. And I actually think it encapsulates really, really well kind of my philosophy here. Your machine learning code, the statistical artifacts you build, the models that you create, are really one small component in the entirety of the pipeline that governs what you're capable of doing and how you're able to continue moving forward and serve your capability in a scalable, efficient sense. And if you leverage those things around you, you will save yourself a lot of time and energy and radically reduce technical debt. Applied analytics are affected by their domain context, as are the systems around them. Let's make things a little bit more concrete. A lot of my modeling experience comes from working in adversarial domains or domains where as your model improves, there's some other force out there that is capable of interacting with and responding to your model. It's kind of similar to the Red Queen's race, which is an analogy that's used to inside of Lewis Carroll's Alice in Wonderland or through the Looking Glass. And there's a quote in it that says it takes all the running you can do to keep in the same place. And in an adversarial domain that's very true. Think about it from the perspective of cybersecurity. As you a defender gets better, your adversary, the attacker, is heavily incentivized to react in response to you, try new things, find new exposed surface area and evolve their approach to try and evade your ability to detect them or stop them. And we see this in a lot of fields where machine learning is being applied today. Cybersecurity fraud, kind of any environment where there's someone you're able to where you're trying to avoid a particular behavior or de escalate risk, where that risk is being driven by human activity is a really clear example of this. And one of the things that I'll say here is that machine learning might not be necessary to perform in these spaces. We certainly did all these things before machine learning was a tractable system of approach for much of this. But machine learning brings huge piles of benefits. To be able to consider high dimensional data and adapt to change these domain requirements, inform your product and therefore the model itself. So let's talk about cyber network defense for a second. You've got to defend an unknown number of services with unknown vulnerabilities against unknown attackers, while still supporting your users needs. Especially if you're trying to do this in an environment that is already up and running, like for example, an environment inside of a large enterprise that exists today. You have to work in concert with human analysts and responders because the insights that you're able to generate need to be acted on. And most cybersecurity organizations staff their team with humans. And then finally, your adversary is incentivized to react to your efforts with speed and find gaps in efficacy that can lead to, let's just call it a materially worse user experience. If your model is trained in a vacuum, some of these concerns might not be clear. If you simply train your model off of like a static data set that someone provides to you with no external context, you will probably end up with a model that performs well on that data set, and you will probably also end up with a model that does not accurately or well represent the world that it would be applied against. And these considerations are really important. Obviously those considerations are things that are the clear challenges. You need to have something that changes quickly. You need to be able to filter boundaries and events inside and outside of the model. Your design considerations need to consider the human experience from the outset. For example, how many alerts is too many or too few and how do you want to engage those humans and your model Adjustments should probably be automated if possible, especially if you're in an adversarial space that changes quickly or dynamically. And so you can make learning your domain specific context a part of your model's goals. Or you can ensure that your non modeling functionality is domain aware and is capable of enumerating things like failure scenarios. That brings me the next layer down here when talking about your domains and how your domain specific qualities are important to designing your model. One of the things that I encourage teams to think very hard about, that I have seen teams sometimes forget, are your failure conditions and your response to those failures. There are failure conditions within any analytic capacity. At some point, malformed, malicious or irrational input will find its way into your pipeline. Not all analytic capabilities can be local and network partitions will happen to the best of us. Upstream and downstream partners can exhibit strange or non compliant behavior because in many organizations, especially larger ones, the pipeline is not a a isolated system unto itself. It depends on myriads of other systems and other teams that are all operating on their own operational cadences and have differing SLAs. Your system could expose patterns that habituate users to a particular behavior and a future change that you build in could break that workflow. Or worse yet, your partner services are not perfectly aligned with your release Cycle and changes might be not upwards or backwards compatible, especially if there's some kind of intent breakdown. And these challenges are present in systems of all sizes. But failure conditions don't scale linearly and you need to consider how your system is likely to fail, what your possible breakpoints are, and most importantly, how you will respond in those situations. And the goal is to ensure that failures are infrequent and failure responses are aligned with domain needs. And that's why this is really important, I think. I've seen teams, when they start thinking about failure scenarios, think about it in a domain agnostic way. What happens if the service goes down? Well, we'll send an alert, we'll restart the service, we'll perform auto scaling. It's a very technical based approach, but if you think about that, that's really only one slice of that pie. It's only one part of the overall scope. On the whole, I think that there is an element where you want to be able to think about what do we change in the event that there's a technical failure and what do we change in the event there's a domain failure? And so graceful failure environments look like this, where an elevator has broken, excuse me, an escalator has broken and it's temporarily turned into stairs. And not quite as much like this, which is a tweet that went around earlier this week where an individual named Rachel True posted a screenshot saying, hey, I've been having trouble with Apple icloud because of my last name, which is an input variable problem. And I don't mean to call out Apple here by any stretch of the imagination. It's just something that kind of was topical and came up recently over the course of the last 48 hours kind of idea. And it demonstrates one thing. It's hard from a software perspective to think about what happens when your systems fail. And the introduction of complex domains and complex models can cause failure to occur in more subtle but no less meaningful ways. And that increases the importance that I think individuals and teams have on building out infrastructure that lets them monitor for failure, respond to failure, anticipate failures, and most importantly, rationalize about those failures inside of environments that might be both technical, as we're seeing in the screenshot, and domain specific, where failure is a word that is defined based on particular constraints inside of the domain. I question, so the question is, how do you do that in a practical sense? As your system gets more complex, this obviously gets harder. And I've referenced domain failures here a couple of times and that's where it gets really tricky, especially in the area of machine learning. Incorrectly calculated input features, for example, can lead to correct operation. No system has crashed, no error screen displays everything from a technical perspective, continues to work appropriately, but it can still have dramatically different decisioning outcomes. Right? If your input data is bad, your output data is probably going to get a little weird too. And an example of this can be very trivial. Let's say, for example, you have some kind of classifier that takes as input data the height of the individual who is interacting with the system. If you've historically been measuring that height in inches and suddenly someone provides you that height in centimeters, you've obviously entered into a scenario where there's going to be a model failure, where your model is going to give weird looking input because you've suddenly changed the boundary conditions under which it must make a decision. And so how do these concerns enter into the system? They can be introduced during handovers between teams with differing skill sets. They can be introduced when teams turn over or change between like and therefore lose local knowledge. They can be introduced at boundaries between interconnected system components. And failure conditions compound as the total state that developers need to consider exceed the collective knowledge and experience. I think we've all probably worked with models, or at least systems that are so large and so complex, no one person can think of the entirety of that workflow all at the same time. And that's when stuff can start to creep in passively that can lead to situations where a user might think to enter in their height in centimeters instead of inches. They might be allowed to do so, or at the very least, that isn't detected. An investment of infrastructure, I feel, can form the guardrails that allow you to scale and remain robust to change without increasing your overall cognitive load. Infrastructure is a really tricky topic, especially these days. It's unlikely to win any major awards by itself. And with apologies to Randall Munro, that's kind of why we're here today. That bookkeeping, the things that allow you to offload, I'm going to call it tedious work to make sure that you're able to do things more safely, more securely and without having to worry about. Did you kind of like not carry the one in performing some series of complex operations? The goal of infrastructure is to enable repeatability across the pipeline and reduce efforts for adjusting components of that flow. You want to look for robustness to change. Your data is messy, your problem is evolving, and guardrails let you iterate faster because you're not paying attention to what often can become a Rube Goldberg machine of a pipeline. Three years ago, I gave a talk very similar to this that talked about kind of the ways that certain teams inside of the company I was working at were responding to the reproducibility crisis in science. This is a picture I stole from Nature that discusses this as science as a whole. So biology, chemistry, physics, all of it. But I don't think we're immune to the reproducibility crisis. I think there's probably a lot of us who have tried to reproduce a paper that's been published and have had challenges in doing so because the paper leaves out steps and the data is hard to access. It's hard to be able to reproduce things even in a field that's so easy to join in and participate in as computer science. And without reproducibility, what kind of a science do we have? The point of science is you test ideas by experiment, which means you need to be able to repeat them. And so we thought at the time that we wanted humans focusing more on determining if their models were good as opposed to the raw act of generating them in and of themselves. Very recently, Capital One publicly released on GitHub a system called Rubicon, which is an answer to that case. This really talented, really amazing team built this system to make it really easy and trivial to store and retrieve model inputs, outputs and analyses to file systems, either local or S3 or other stores of truth that allows you to enforce and agree with best practices to make sure that you can log every single experiment that you did. And it's kind of an example of a piece of infrastructure that makes the overall process of model development safer and easier. And now this is certainly at development time and there are implications going into production. But I started this talk by saying there's a lot of production areas that I wanted to talk about today. So let's see some of those other areas that infrastructure can improve. I think that the industry as a whole and the community around open source inside of machine learning is reducing a lot of that friction going from dev to prod. But prod considerations are distinct from development design requirements. When we're talking about supporting infrastructure, latency enters into a picture in ways that it didn't for a developer. Certainly you want during model development, your system to be snappy and responsive and not introduce friction as you build and develop things. But you definitely also realize very quickly that, hey, during the act of model development latency, for example, a 3 millisecond SLA that you might have with a customer is not as immediately important and so it can be forgotten about or a secondary consideration that you don't remember until later on in the process. Iterative flexibility inside of production is not quite as necessary. Data access patterns and systems may well change depending on where you are and the kind of environment that you have. Monitoring becomes a more diverse effort because you've got a larger set of things to monitor and keep an eye on. And then finally input, validation and recovery definitely has to be performed. Production deployments sacrifice flexibility for performance and efficiency is the statement that I would make. And I'm sure there's someone in the audience who is aching to be able to raise their hand and say, hey, I disagree with that. I live in a world where my production environment is just as flexible as my development one. Those certainly do exist. That's probably very true, but I would say that it's not the standard case across all organizations because many organizations, due to the fact that industrial scale machine learning is becoming is kind of an emerging discipline. Not everyone has made it down that maturity curve yet. And so there's lots of tools that are available to make model serving easier. We're putting a lot of energy around that as an industry and decreasing the friction in the promotion process from dev to prod. There are fewer systems that are available to ensure that you have ongoing things like data quality, well managed feature orchestration and other things like that. Inside of a heterogeneous environment. Data quality is critical and it's deeply embedded inside of your domain. You've got something that is a constraint that can be evidenced by your data, but is not necessarily inherent within it. The thing that I ask everyone to take away from this talk is this. Do not trust your data. It will sooner or later lie to you. And if you're in an adversarial environment like say, cybersecurity, it will do that intentionally. You're likely to rely on data for a sufficiently complex model or a sufficiently complex process, which your model is a part of, is to turn around and ensure that you've got a way to handle data of differing quality levels that you might not have direct control over. Data quality covers a kind of a wide number of dimensions. Five that are really well known and well understood are up here. I won't read through all of them because that gets tedious very quickly and I don't want to bore everyone. That's why I create the dense slides. But they are accuracy, completeness, consistency and timeliness and validity. And then finally you have two more which I have found show up in production environments. Pervasiveness, which is how often do we duplicate the same data in multiple places, and temporal mutability, which is to what extent does the data describe a phenomenon that naturally drifts over time, and where are we accounting for that drift? I think that last one's really important. There are multiple definitions of the same thing in many cases that can be difficult to manage in an effective way. And this starts to begin a discussion on the concept of lineage. Lineage records things like the origin of data.