#91 Building a Holistic Data Science Function at New York Life Insurance

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com. You're listening to Data Framed, a podcast by Data Camp. In this show, you'll hear all the latest trends and insights in data science. Whether you're just getting started in your data career or you're a data leader looking to scale data driven decisions in your organization. Join us for in depth discussion discussions with data and analytics leaders at the forefront of the data revolution. Let's dive right in.

Hello everyone, this is Adele, Data Science Evangelist and educator at datacamp. When we talk about data maturity on the podcast and the process of becoming data driven, we often lose sight of the sheer amount of effort and work it takes to build a strong data science function within an organization. Building data teams, creating connection with the rest of the organization, evangelizing the data team's work, training others on data literacy and building a data culture is no easy feat. But today's guest approaches building high impact data science functions with the clarity few have. Glenn Hoffman is the Chief Analytics Officer at New York Life Insurance. He's an experienced senior executive in insurance and financial services who currently leads the corporate 50% data science and AI function at new York Life. He is responsible for the foundations relationships with many internal groups and a great variety of projects and also leads their Data Science Academy through internal education program for all New York Life employees. Throughout the episode we talk about how he built the team at New York Life Insurance, the different skills they optimize for delivering career pathways for data scientists, building MLOPs and model governance teams, how to build relationships and market the work of the data team, the ins and outs of the internal Data Science Academy, and much more. If you enjoyed this episode, make sure to rate, subscribe and comment, but only if you liked it. But now let's dive right in. Glenn, it's great to have you on the show.

Great to be here. Thanks for inviting me.

I'm very excited to speak with you about your work leading Data and New York Life Insurance, how to organize effective data teams delivering Impactful data science use cases within life insurance and much more. But before, can you give us a bit of a background about yourself and what got you to where you are today?

Let me start with the current position and how I got to that, and then I can go a little further back if needed. Yeah, so I've been at New York Life a little over five years actually, even before coming here, I was watching the life insurance industry for a bit and was sort of excited about it. And then it seemed like the right moment about five years ago to join the industry and do some data science, because I mean, life insurance is a fairly traditional industry, but now over the last few years it's been ripe for change. And the nice thing is it has a great variety of interesting problems that we'll talk about more and we can build the first solutions. Right. So New York Life is a pretty big place. It's one of the Fortune 100 financial companies, so there's lots of challenges and most of them we tackle for the first time. So I think that's really exciting to me to be at a place where you can be truly innovative and build first solutions for really interesting problems. So that's kind of why I joined the company. And since then I've been a function in the team a little further back. I mean, I've been in data science and what we used to call statistics for a long time. I started out as a stats professor a long, long time ago, did five years of that and then joined industry and basically did every job. I was a hands on predictive modeler early in my career, eventually got more and more responsibility and then probably over the last decade or so I've been building data science functions for a few different companies.

That's really great. And you mentioned here the exciting nature of working in data science and life insurance and I'm excited to first discuss that.

So.

So I'd love to first understand the lay of the land when it comes to the use cases and applications of data science within the life insurance industries. What are some of the exciting data projects you've overseen at New York Life Insurance and where does data science and AI really provide value for life insurers?

First of all, my team, the center for Data Science and AI at New York Life, we are here to provide data science and AI expertise, solutions, education and worked with a variety of business partners to essentially build data driven solutions that make decisions in all kinds of processes. So that's the idea. The idea is to create value for the company, mostly through data driven automated Decisions that function in real time or in batch and business processes. I'll give a variety of examples of that and why they are exciting, at least in my view. There's a lot of different things we do. We support a whole bunch of different areas in the company. So for example, the first one is our distribution system, right? We sell life insurance through agents and we have built solutions for agent recruiting. We've built solutions to predict agent productivity. So it's exciting from a data perspective because it uses a lot of behavioral data and some transaction data. From a perspective of algorithms, because this is actually neural network models that we do on that front, then we have underwriting and for life insurance, underwriting means mortality prediction. So that is and triaging of various data sets. So this means medical data, right? So from a data perspective, exciting because it's a great variety of medical data and from an algorithm perspective it's survival modeling, classical and machine learning type. So that's cool. A third example might be Covid modeling, right. The last two years, as you might imagine at a life insurance company, we've had a pretty significant project trying to model Covid mortality. We've worked with medical, doctor and government affairs people. And that is obviously very exciting. Very current data, very interesting data. And from an algorithm perspective, we did hierarchical based pandemic modeling, which is very, very sophisticated statistical modeling. Then there is marketing modeling that we do. There's fraud modeling. Fraud modeling is another very exciting problem because there you're looking for a needle in a haystack. The huge overwhelming majority of transactions are perfectly fine, no problem at all. They should happen. But every once in a while, in a very small minority of cases, there's somebody trying to attempt to fought. And there was a whole other set of techniques that we use to try to figure that out. This is just a handful of projects we have going on right now. There's obviously much more. But that's what shows the great variety and excitement in those projects, both on the algorithm side as well as in the data side and certainly on the business problems that we're trying to solve.

That's awesome. And you mentioned here a few use cases that create new value, such as COVID19 modeling to use cases that accelerate value by reducing costs and generating more efficiencies. So of course delivering on these use cases requires a high performing data science team. And this is something I'm excited to deep dive with you on and really break down. So I'd love it if you can walk us through what you found to be the hallmarks of a successful and impactful data science function, especially within large and complex enterprises like New York life insurance.

I guess I start with saying it is the third data science function that I'm building in my career. So I learned a lot from all the mistakes I made the first two times and I think we did a lot better this time around. I think there was three big aspects of putting a successful data science function together. First is obviously talent. So hiring great data scientists and great data science managers and leaders is important. But then also data scientists don't operate in a vacuum, so also hiring a bunch of other skill sets to make it work. So I'm thinking of machine learning, operations engineers, data engineers, project managers, model governments, a little bit of training and development. So I think all these are important talent to have to make a team successful. The second is platform and technology. We set up a platform both for model development as well as probably even more importantly for model deployment into production. So we have all the regular development platforms, the Python environment, the R environment, geospatial environment. But then we really spend a lot of time over the last three years, I would say, on creating a deployment platform that allows us to easily put every model we built in production, both in production on our side as well as in production at the business partner that's, you know, using the models every day. So that's the second one. The third one is relationships, especially in a large and complex organization, like new life relationships inside the company to our various business partners and other stakeholders. So we spend a lot of time and effort and communicating pretty consistently and frequently with our business partners, both in the creation and designing of projects, execution of projects, and then post project, you know, the use of our models and products.

So I'd love to unpack this with you, you know, given the diversity of the roles and stakeholders on any given data project or team. How are these different roles organized? At New York Life Insurance, do you have a centralized data function, you know, like a center of excellence or an embedded model where these different roles are embedded in functional teams or something in between?

From a corporate perspective, the function that I lead, the center for Data Science and AI, is a corporate function, Right. So we support almost all of the company with data science AI. So that's sort of centralized. From a data perspective, there is a corporate data team and then there's a few individual data teams that are closer to the business or the function. Right. So I have a little data team in my organization that takes advantage of what the corporate data team produces, but then does the last Mile of data processing if you want that we need. And then within my team I have a specific machine learning operations team. So that has a leader who is an experienced engineer and he's got like I believe seven engineers on his team that are completely specializing machine learning operations. I have a project management office that is led by a manager and have several project managers on that, have a change manager on that and then we have a development team that basically does a lot of training and community building for my team as well as the rest of the company on data science. So the nice thing, I mean we have some scale. My team is about 55 people if you count, then it's probably about 75. It creates a necessary scale to have career pathing for all these skill sets on the team. So let's say a junior data scientist that might join the team or even a mid career data scientist that joins the team has many options both from the type of work they do in terms of what area they've built models in and the company as well as a career progress. So they become really senior data scientists that works on more complex stuff. They become a manager, they can become a manager of managers all the way up to my spot. Right. So we have a good career track for everybody on the team. We also take care of continued education, we do tuition reimbursement, we allow people to go to conferences so they can top up their skillset.

That's great. And you mentioned here creating development tracks for a lot of the talent on the team. When do you start thinking about creating these pathways and when should data leaders building data functions start thinking about these pathways and what are the lessons that you can share from your experience?

I think this is sort of important when you think about you have a centralized team, you have small teams in different places. You need to have at least one sort of central team that has critical mass to enable the career passing, to enable the funding for platform and to enable all the support that you need. I started thinking about that right from the start when I put the team together. I assigned my team, or at least for the most part in this way right from the start. There are a few functions we added later as we gain scale, but the general concept is there from the beginning. I think it is important because if you're joining a five person analytics team somewhere, then you do have to wonder what are you going to do two years or three years down the line.

That's definitely true. And harping on the MLOps team and MLOps functionality you've added in the Data team at New York Life Insurance. This strikes me as relative to where the industry is at, as a sign of a mature data science function to have dedicated talents towards deployment. So can you share the process of building such a function and what were some of the best practices you learned along the way in building a MLOps team from scratch?

You know, I think it's been important for a while, but now we have much better technology that actually makes it a little easier to make this a reality. And that has evolved tremendously over just the last few years. The importance for me and the reason why I put so much focus on it, I see a lot of data science team out there that build really interesting, exciting models and then never deploy them into production. And data scientists get frustrated because they built something that nobody uses. And business gets frustrated because they don't get any value out of the data science team that they built. Right. So I wanted to avoid that completely and say the focus is on productionalizing. We want to build great models that we can have trust in and that are diligent and sound, but they have to be deployable, otherwise there's really no point. So pretty early on, as soon as the first models got to a stage where we worry about where are they going to go, started building this mlops function which sits in between the data scientists and the pure technology folks. The mlops function helps the data scientist works with the data scientists every day to build models that are ready for production, both from a code perspective, because, you know, engineers can typically write a bit better code than data scientists can, but also from a data perspective, is the data that's being used ready for production in that form and what are the concerns that are going to come up later when we productionalize? And then we also invested in platforms. So now we have a Kubernetes based deployment platform where we can promote code directly from Python into the platform and probably a little bit of code polishing, have these things be ready for production. And then we build an API from our production platform to any production system in the company. So our models can be used in real time by any production system in the company. And that was really the key in creating value relatively quickly from everything that we built. So I think it makes greatly satisfactory for data scientists because everything they build is being used by the company in production every day. So I think that creates a certain sense of satisfaction. It's great for the business. So when people ask me what kind of value are providing, I can point to a dozen or more models that are in production that people are using every day. So it's not a hard question to answer.

Definitely. It must be so exciting to show the ROI of a data team. And I'd love to talk about building MLOps teams for hours. But given the topic today and discussing your work, leading data, Data Science and New York Life Insurance, I'd be remiss not to talk about the data culture component when building a high impact data team. You've been very outspoken about the importance of transparency in education when it comes to the data team's work and impact because it ultimately means both buy in from decision makers who own different lines of business, and practitioners who are on the ground and predominantly relying on their own subject matter expertise. So I'd love to first start off by understanding how you went about getting that buy in and partnering with these different audiences and getting them excited about the potential of data science.

Yeah, that is a very key thing that we spend a lot of time on. To me, it's a lot of communication, a lot more than you might think in a technical function. So we have what I call sort of a multi level communication approach. So no matter who you are on my team, no matter what level you're on, everybody has a set of stakeholders that we actually define and we have an exercise about that every once in a while where we say, okay, these, in my case probably like 30 or 40 people. In the case of a junior data scientist, maybe eight to 10 people. Those are the relationships you have to maintain. Those are your stakeholders. You have to talk to them often and you have over time make them understand the value of data science for their business problem. But also you have to understand their business problem and you have to understand what they worry about every day. Because communication can really only happen when both people understand a little bit of where the other person is coming from. So we have this sort of multi level strategy. We get together frequently on the side of my team to actually strategize about who communicates with whom, what are we communicating and how are we basically getting buy in on what we do. The other principle that I have is I don't disintermediate my data scientists. Data scientists do talk directly to business partners. Engineers do talk directly to business partners. There's no MBA translator in between. I don't believe in that. It does, however, mean that data scientists do have to learn how to communicate fairly well. There's pros and cons to each of those, but that's the model that we have done. And I guess the third thing I Will say is first with communication and multi level and often sentencing. No, it is a mediation of technical people. And then the third is we don't go away and do our own thing and then come back with a solution. So every project has weekly project meetings. Everything we do, we communicate to stakeholders right away and get their buy in. Often even on relatively small decisions on a project such that by the time you're halfway into the project, it is not our project, it is their project and they speak about it. And when we go to like a status meeting in the business unit or something, I prefer our business partners to present the project and not us to present the project because that cements the idea that it's really their project. So by the time you come around the end of it and a decision gets made on how to deploy it and how fast to deploy it, it's their project they want to deploy. No longer what we say. And that's the best case.

That's such an awesome approach. And following up here, how have you been able to instill this mindset with data scientists on your team, especially the more technically minded ones, and gear them to be relationship driven and value driven and not just focus on technical aspects of their work?

You know, it's constant reinforcement and it's not right for every single data scientist. Right. So we try to recruit the people that are both technically very strong, but also have the desire to solve business problems, the desire to communicate, the desire to teach sometimes. Right. The desire to sort of advocate for their craft, so to speak. So recruiting has something to do with it. And then of course we do a lot of training, do a lot of soft skills training, communication training, PowerPoint training, and then there's a lot of on the job learning. Right? Learning from their manager. Obviously, you know, all of my managers are very strong in that. So people can learn from the managers, they get feedback. It is something we focus on and, you know, that's how it happens.

Another key component of creating a data culture as well is by ensuring that there's a community of practice around data and that folks can feel like they can get involved. So can you walk us through the different ways you address this and how you got the general population at New York life insurance companies excited.

There's a lot of sort of community work. So I have two people on my team that do nothing but training and community for the rest of the company. So on the training side, we created this thing we call Data Science Academy and it is a training program for really anybody in the entire company who has some interest in data science. What we have done is we've put sort of curated learning tracks together, mostly using existing online courses that we put together into a track. We also created some of our own content from our projects that we do and we have learning tracks anywhere from like on the high end, sort of the one year or ten course coding based data science learning track, if somebody wants to make a career change in becoming a data scientist, all the way to more of like a business style non coding, multi course track to a get to know track where if you just want to watch a few videos and look at a few articles about data science, you can do that. More recently I actually taught myself a half day data science course for all the executive officers at New York Life. And we got about those are very high level and busy people. About half of the executive officers actually enrolled voluntarily into a data science course. So that's, you know, the Data Science Academy is one of the most popular learning programs in the entire company today. So that creates a lot of branding for us internally. It also creates people who are interested in data science at the various different business units that we want to collaborate with. And it's always good to have that. It's always good to have somebody to talk to in other units that have an interest in data science. It makes it a lot easier for us to engage. So that's one thing. The second thing there is what we call the data science community. So out of all these people that have some interest in data science in the entire company, you know, we created a mailing list and we have a lot of events. So we put on seminars, lunch and learns, we bring in external speakers, we interview people inside the company and that is really exciting. And in the days of Zoom meetings over the pandemic, we have an average of like 300 people logging on to Zoom Call. That is a technical data science seminar. That's a pretty good audience. That part is working really well. And then of course for my team itself, we have a lot of post technical or non technical training as well. So that's understood. But this sort of branding effort is more for the entire company. I think it's important, especially at a place like New York Life where data science is just sort of in the first five years of development.

That's fantastic. And I love the degree of personalization that you have in these different learning pathways. There's always a learning track for you no matter where you are at the organization, but also how you've paid attention to executive training to generate that buy in and digging deep on the executive training component. These are folks who are really busy. They have a strong gut instinct that they developed over decades. How do you create a learning program for such an audience? And what does success look like for you in this case?

We originally thought of planning it actually before the pandemic hit and it was supposed to be in person in a conference room for half a day, you know, with some snack breaks and all of that. Well, then Covid hit and that didn't work out. So we had to re engineer the entire program for a virtual experience. One of the things I also should mention that we also use in the executive track we use for other purposes is for all of the large projects. After the project, we actually create a video and the video will have people from my team speak. It will have our business partners speak, might have somebody from technology speak. And it's like a three to four minute kind of video, enough hard to watch and you can show it everybody. And that's about. So we used the videos also in the executive officer training. We used some general materials around what is data science, how it is applied in various industries. And then I brought a lot of examples from New York Life data science, which is really relevant to them. And to your question, what does success look like in a training like that? So first of all, if you've done trainings on zoom, I guess the first criteria is that people keep their camera on and they'll attend. So that's actually not as trivial as it sounds. The second thing is, are they asking a lot of questions during the course? But the third thing that's probably the most relevant is are they coming up with their own ideas of data science projects? Right. So after getting an idea of what data science can do, do they come with their own ideas at the end of the course or doing a discussion in the course and then follow up on those ideas after the course, there's a couple of meetings to explore new ideas that they conceived after learning a little more about data science. So we had probably 20 different ideas coming out of these training sessions that they were interested in. That's how I would measure success of a training program.

That's such a great way to think about it because if you're able to create evangelists out of your partner executives, that will make life much easier for you as a data leader and harping on the upskilling for the broader population. What are the tools and skills you've prioritized and how has the upskilling program move the needle for you? When it comes to creating a common data language in the organization.

So the skills sort of vary depending on the audience there, right? So I think for executive level folks, or even like maybe director level folks, if they are in the non technical areas like sales or HR or those types of folks that we just want to create a general understanding of how to recognize a problem where there may be a data science solution. So understand enough about data science to recognize a problem that might have a data science solution. And then for a little bit more technical folks, it would be giving them a sense of what data science solutions are looking like. And even for mid level folks, giving them an idea of what does it take to build a data science solutions, what are the types of people that need to be involved? How early do these people need to get involved before the data is all screwed up and all that, and really enable them to recognize our skill set as data scientists to be different from maybe a business analyst or from an actuary or from an engineer, and recognize when they should get us involved and recognize the benefit of getting us involved. Then there was the other cohort of people that are thinking about a career change to data science for those folks, we have to teach them real data science. You know, that is also something we can do.

So we talked about the skill element, the culture element, and to a certain extent the platform element. But something that cuts across our discussion so far is the importance of building relationships. So you've partnered with a lot of different folks within New York Life Insurance. How do you manage these relationships and who are your main stakeholders? And how do you ensure that the data team's roadmap is aligned with the business's priorities?

Yeah, okay. There was a lot of stuff in there on the relationship side, basically. I mean, everybody on the team has a set of stakeholders. They maintain relationship spreads. Right. I mean, for myself, I probably have regular meetings or lunches or something. There's, I don't know, maybe 40 or 50 different executive offices in the company just to kind of understand what's going on in their unit, you know, what are their priorities, where could I potentially help or my team could help and stay on top of what's going on in the business so that we can detect opportunities as soon as they arise. For folks that reporting to me, they have a large set of stakeholders also and they want to be a little more tactical and say, okay, what projects are you starting? What's coming next year? How can we help? What's the status of current projects we're engaged in? Is everybody happy? Are you happy? Are your people happy? Keep that going for more individual contributor types. On my team, it will be the stakeholders of a particular project and make sure they talk to them, not just in official meetings, but maybe, you know, have a lunch with them, have a one on one meeting with them to just make sure that we're hitting the mark, we're on the right track, they understand what we're doing. There's no gaps in knowledge or understanding, no gaps in goals. You know, that's kind of the practical way that we do that on the communication side now, on the planning side, that's, I think, the other part of your question. There needs to be budget alignment. When I say deployment, it's nice to have a platform to deploy all the models, but deploying models is not free. You also have to have the budget to deploy them. In our annual financial planning process. I don't only have to worry about my own budget, that's complex enough, but I have to worry about the budget of every one of my business partners too and make sure they have budgeted for the deployment of our models into their environment. So that will actually happen. So there's a lot of coordination going on during the planning process and the budgeting process to make sure that we have to budget on the tech side, we have to budget on the business side to actually get models deployed in the coming year. And that's very important.

There's definitely a lot of complexity in managing these relationships. And speaking of complexity, I'd love it if you can also walk us through the challenges in working with data within an industry that is highly regulated. And how do you ensure that you're consistently innovating responsibly?

That's very interesting and I guess more relevant topic every year. I mean, I want to make maybe two points on that one. One is, I mean, regulation is there to protect consumers, right? So I mean, New York Live as a company is very much aligned with that. In fact, we're not a market traded company. We're what's called a mutual company. So New York Live is actually owned by the policyholders. It was essentially owned by our customers. So hence our philosophy is very much to do right by our customers and that is, at least in principle, the purpose of regulation as well. So we appreciate well thought out regulation and we're quite happy to follow that and influence that in the right way. It's very much aligned with our company. So that's the first point. The second point for engineers and data scientists is the compliance with regulation is not boring. At all. It actually creates a lot of interesting analytical challenges in its own right. So when you think about disparate impact testing for protected classes, that creates a whole new set of algorithms that you have to develop to do this intelligently. Because disparate impact is not sort of a black and white thing. It's complex. Disparate impact of a model. Well, a model is multivariate. Which pieces of that model are actually creating or not creating some potential disparate impact? How do you analyze that in a multivariate situation? There's a complex modeling algorithm behind it. Where do you draw the line? It's not a qualitative concept. In practice, it's quantitative. Where do you put a line between some correlation that might indicate disparate impact versus not? So? It creates a lot of interesting analytical challenges. I'd say that the head of my model government team is probably one of the most interesting positions on the whole team.

That's a great way of framing it, where constraints add a new layer of complexity and intrigue for different types of data science projects. So can you walk me through the process of setting up a model governance team and the lessons that you can share with other data leaders here?

We set up a model governance team probably about two or three years ago, once the organization was mature enough to meet our own headcount on that. It's what I call the first line of defense. I have my own model governance team, first line of defense. Then the company has a corporate model governance team. That's the second line of defense. Every model gets validated on my team. So every model that gets built gets validated by people that had no involvement in building. That's independent validation. We look at data validation, we look at model validation. And then of course, we look at any kinds of regulation or legislation that this may be subject to, depending on the use case and the data that's in it. So the leader of my model governance team has to be a very good data scientist who actually understands all the algorithms and the data, but also is directly connected to our government affairs team and the company and the legal team to understand regulation, legislation, not just of today, but of tomorrow. Because these models we built, they should be in production for many years. So we have to look at, you know, what is being proposed as legislation in the state of Ohio in the next year or something like that is also regulated. So we don't just have one regulator. I mean, the federal government is also a regulator, but we also have 50 other regulators. Right. In every state. So that makes it Pretty interesting job.

That's really fascinating. Now, Glenn, as we are reaching the end of our episode, I'd love to ask you a few questions before we close out. The first one is what would be your advice for any up and coming data scientists fielding chall choices between joining tech or more traditional industries such as finance or insurance?

There is sort of the misconception that I want to debunk a little bit, that if it's a tech company, it's exciting. If it's an insurance company, it's boring. That's not true at all. What you should look at, what is the specific job and what kind of thing are you going to work on? Because if they make 95% of their business follow from advertising, 95% of a data scientist is going to work on is maximization of advertising leverage. So an individual job might be focusing on optimizing one single search term for the next year. A lot of these large tech companies have done this for a decade or more. So you're building the 17th version of that model, whereas at a place like New York Life, you're building the first version of a model that has never been tackled before in a variety of different challenges. The company is old, 175 years old. But the challenges we tackle from a data science perspective, we're often tackling for the first time. Look at this specific job. Don't just look at the company, look at what you're actually going to be working on. Are you building the first model, the second model, or the 15th model in that area? What kind of interesting data do you get to work with? What kind of variety of models and other data science solutions are you going to build? Those are the people you're going to learn from. And then obviously the most important thing is the data science function set up for success, meaning all they are deploying all their models into a business and actually get the benefit out of it. Because if they're not, doing that probably means that the existence of that data science function will eventually be questioned.

And given the exciting use cases of data science and AI in insurance that we just discussed, what are some of the trends you're particularly excited about that will impact the insurance space?

Yeah, I mean, like you said, we can talk a long time about mlops. So I think mlops is a huge trend and we're just at the beginning of that journey and it's already made such a big improvement in the way data science functions in reality. I mean, now we have good platforms that you can deploy models on those Platforms will probably get even better. There'll be more of an integrated way that integrates even better with the model development. The thing that we're actually investing a bit of money into now is what happens post deployment of the model in terms of software solutions. With the model in production. It's running now, you should monitor that model. I have a monitoring function that we put in place in the last few years. Every model now gets monitored both for things like data drift and score drift. If something goes wrong with incoming data, somebody gets loaded right away. But before we make a bunch of bad decisions and then also reporting. So we want to write a model. After the model gets deployed, our business partners want to know that they're actually going to get the value that we promised. So we have to do a bunch of reporting on that production model. We'd like to not write custom code for. All of that would be a lot of work. So we've been looking at software and actually we made a software decision that helps us monitor and hold on models in production. So that's all part of the mlops process. And I think mlops Engineer is probably one of the hottest professions right on the same level as data scientists.

That's great. Now, Glenn, as we close out, do you have any final call to action before we wrap up today's episode?

Like some of the things I said, if you want to kind of learn more over time, definitely feel free to connect on LinkedIn with me. I'm always happy to provide material to the community. I post pretty actively on LinkedIn, so that's a way to stay in touch. Also, I would be of a miss to not mention job opportunities on the team. So I frequently have openings on the team on all the different skill sets that I mentioned. Data science and labs engineers, data engineers, project managers, model governance at many different levels. We also have a large internship program that we run every summer, and recently we started what we call an associate program. So starting this year, we actually hire several people with a bachelor's degree in a technical area, and then they spend three years on my team, rotating through a few different jobs and at the same time doing a master's degree for which we pay the tuition. So that's yet another great way to join the team and become a data scientist.

Thank you so much, Glenn, for coming on Data Framed.

Thanks so much for having me.

You've been listening to Data Framed, a podcast by Data Camp. Keep connected with us by subscribing to the show in your favorite podcast player. Please give us a rating. Leave a comment and share episodes you love that helps us keep delivering insights into all things data. Thanks for listening. Until next time.