Productionizing AI at LinkedIn

A lot of us get fascinated by technological advances in AI, right? The fact that we can ask questions out of an AI agent and express a meaningful, almost human like response when it comes to actually putting these systems into production. I want to say that AI in some sense is an engineering discipline, but I think we are very much far away and it's for a reason, right? It's not because we are unwilling to do so, it's just that the inputs are not as deterministic as some of these other systems, right? So when, whenever you have a change in your data distribution or something happens, then how do you put in, quote, the right unit and regression test to make sure that you're going to be able to catch some of these errors. How do you keep track of the number of people? And I can easily tell you the number of instances where you know, a team deploys a feature and then there are 20 other teams with, unbeknownst to you are starting to use this feature and the next time you make the change all help breaks through.

Bandwidth for change Log is provided by Fastly. Learn more@fastly.com we move fast and fix.

Things here at Changelog because of Rollbar. Check them out@rollbar.com and we're hosted on Linode Cloud Servers.

Head to linode.com changelog this episode is brought to you by Digital Ocean Droplets Manage Kubernetes Manage databases, spaces Object storage, Volume block storage Advanced networking like virtual private clouds and cloud firewalls Developer tooling like the Robust API and CLI to make sure you can interact with your.

Infrastructure the way you want to.

Digital Ocean is designed developers and built for businesses.

Join over 150, 000 businesses that develop, manage and scale their applications with DigitalOcean. Head to dot co changelog to get.

Started with a 100 credit again. Dot co changelog. Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@chain.com community and follow us on Twitter. We're Racticalai FM.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist at SIL International and I'm joined as always by my co host Chris Benson who is a principal emerging technology strategist at Lockheed Martin. How you doing Chris?

I am doing very well today, Daniel. How's it going?

It's going great. It's a nice cool day here and so no Complaints over the weekend. The Internet's been crashing at my wife's business and it's because there's been too many. The access points couldn't handle the number of devices. It wasn't so much of like a bandwidth issue, it was like a access point. So we just went full in and like I ordered some people might be familiar with ubiquiti or unifi network appliances and that sort of thing. So I got like a new Gateway Plus 48 port switch, plus a network video recorder and four cameras and a bunch of cable and all that stuff and two new access points. I think the. Something ridiculous. You can have like 500 people or something on these access points. So might have gone a little bit overboard, but maybe, you know, it was, it was really fun. It was like a nice different sort of tech thing that I'm not qualified to be interacting with at all, but was fun.

Yeah, yeah. The weather's been really nice. I'm enjoying the fall weather. Things are cooling down. Spent the weekend outside as much as I possibly could doing that. And then we're into this week. And just as an incidental note, this is the GPU technology conference week for Nvidia. So been kind of watching the announcements and some of the talks on that. So lots of cool new things for us to talk about going forward as people adjust to this year's announcements. But I'll leave it there for a point where we're actually going to dive into that.

Yeah, definitely. And, and people should check out our episode number 106 where we had Will Remy on talking about the GTC conference and some of the things related to that. So if you're at gtc, that, that would be a good one to, to listen to as well.

There you go. It's all virtual this year, though, you know, it's a virtual gtc.

Yeah, it is all virtual. That's good. Yeah. But I'm sure at GTC people are virtually exchanging some, some LinkedIn requests to connect back and forth and of course, we're all used to that at this point, and that's part of our lives. And they're doing a lot of cool machine learning and AI stuff at LinkedIn. And we're really excited today to have Suju Rajan with us who is Senior Director of Enterprise AI at LinkedIn. Welcome, Suju.

Hello. Hey, Daniel. Hey, Chris. Thanks for having me here.

Yeah, great to have you. I know I could look on your LinkedIn and look at your whole background, but for the sake of those listening at home, could you kind of give us a sense of your background, how you got interested in AI and machine learning and eventually ended up where you're at right now.

Wow. Goes back a very long way. Right back to my undergrad where I had a professor and I was taking a class on image and signal processing with him. And one of my undergrad senior year projects was on using bidirectional associative memories for recognizing digits. And it actually worked very well. But I remember compared to these days when the scale of the data is huge, it was like this tiny computer in which I had my matlab code running out. All of these different BAM and EBAM architectures and whatnot. So that resulted in my getting admitted to UT Austin under Professor Jyoti Kosh, whose lab was doing data mining. Kind of dates me now, but data mining was what it was.

Do people still label themselves as data min or doing data mining anymore? I don't know. Maybe. No.

Sadly I was there with you. It's not cool anymore. Not as snicker.

They make fun of us. Still data mine. Yeah.

But I don't think they'll ever go away. But yes. Let's not call ourselves data mine.

Okay.

I just want the listeners to know that once upon a time, data mining was in fact cool. And we wanted to do that.

But that's.

Yes. And we had NSF funding.

Exactly.

So at UT Austin again was a very interesting application. It was to analyze satellite images. And it was very hard to get labeled data for these images. So you would have like these marshy lands in Botswana and all of those places. And you had to identify what was the sort of vegetation that grows. And. And we were collaborating with the center for Space Research and they'd actually fly out people to go walk and record that. Hey, this is the sort of vegetation that is growing and whatnot.

Yeah. Talk about annotation.

Seriously. Right? No mechanical duckers to fly out to Botswana. But over there my thesis was on knowledge transfer because you kind of wanted to learn from seasonal data to some extent so you can extrapolate to satellite images in the future. Also from related geographic regions as well. So you don't basically to bring down the cost of annotation while maintaining good performance. So that was what I did. Had a very fun time. Converted into an internship at Yahoo. Yahoo Labs. Yahoo. Research. Lots of different names, but it was a fantastic nine years as an individual contributor. Applied Machine Learning. Scientists call it what you will, but got to work on a whole bunch of different machine learning applications. And I think it Was right at the point where ML was kind of taking off, Big Data was taking off. MapReduce was sort of starting to come in. But when I started, we didn't have it yet. In the beginning it was still all of these single models that you would build to build classifiers better, keyword clustering for sponsored search targeting for advertisements and whatnot, right? So had a lot of fun. Eventually it became larger and larger data. Now we had to do news clustering within a few hundred milliseconds and so on, so forth. We had to have all of these models that were tens of twenties milliseconds, if I may, to be able to figure out if someone's going to convert for an ad or read a news article and so on. So had a lot of fun doing that. Eventually switched into managing a group that worked on again, personalization across a whole bunch of Yahoo products. This was on the apps on video recommendation, News recommendation really got a good sense for what would work as a consumer facing machine learning product. How do we measure interactions? Following which I went to Critio where I headed the critique AI lab. This was my full deep dive into computational advertising. And now if you talk to anyone about machine learning for advertising, they immediately think of CTR prediction and hey, it's kind of done, right? It's sort of already there. There's nothing interesting going on in that space. But given the amount of money that is spent on advertising, there is a lot more that's got to do with the causality of the models, right? Because the people who are funding these campaigns want to know that something useful is coming for that money that they are spending. So how do we show that our models actually work was an interesting phenomenon. And more than anything else, I was working at this company which was sort of like a demand site platform. So you're kind of listening to all of these publishers who want to put ads on their different pages and you have all these advertisers that you're synchronizing with. But if you've ever like had a deep dive into that world, there are so many, many, many players. There's this waterfall model that they have, right? Like the first party publishers want the first dibs. If they cannot do it, then it goes to the exchanges and so on, so forth. But all of that entails a very, very, very, very tiny window in which you need to sort of make that bid and then assemble the ad unit to be shown and so on, so forth. So it was a very interesting time where you got where I learned Quite a bit about the real world constraints. Right. It's not as much about the technical complexity of the models, but hey, can it work in this specific constraints in terms of the latency in which you have to operate? Okay, so that was Kreo AI Lab also got to show that there were lots of cool problems in computational advertising. So we established the lab as a pretty strong presence in Europe. And off late, a little over a year now, I've been at LinkedIn that I head what we call Enterprise AI. So think of all the business facing applications LinkedIn has, one of which is Talent Solutions. This is where recruiters come to LinkedIn to look for candidates and we also have all of our members who are trying to find the right jobs. So how do we make sure that this marketplace works well? Right, so that's one aspect of it. Of course there's the advertising aspect of it, the sales navigator notion where folks who want to pitch their cool new products and services to LinkedIn members, how do we make sure that it's relevant and there is value on both sides? So these are some of the NAI for all of these different business facing applications is what I'm working on right now.

Super interesting. So you've definitely always been at the forefront of these different fields as they're really, you know, have so much momentum. Does it seem like the sort of current hype and influx of people into machine learning and AI? Does it seem like, you know, faster paced or more momentum than in the past to you? Or does it just seem like a sort of new area that, you know, people are putting focus on and switching from, from other things?

For better or worse, we became a cool area to be in. Right. There's.

That'S funny, a lot of media.

Coverage on what AI can unlock, but it is true, right? Maybe 10 years ago it was sort of behind the scenes, nobody really talked about it. Hey, why do you see the results that you do? Why are you being shown the ads that you have was not really a part of the discourse. But I think all of these interesting innovations in terms of, I want to call it the PR events that some of the big companies sort of set up helped the field get the recognition that it perhaps deserves to have. Right. Given how integrated some of these models are into our daily lives. You know, the fact that whether I get a home loan or not is perhaps has a machine learning model behind it. Right. And it's important for folks to, to be aware of it. I would say it's a combination of both. It feels like A new space because there is a lot more recognition of the fact that it is very integrated. Right. And data driven decisions and AI models are never going to go away from now on. So how do you want to think about this field from a societal standpoint, I think deserves conversation and that's how we make sure we don't make the wrong choices. And at the same time, open sourcing has become a big deal, right. With TensorFlow and PyTorch and all of these larger companies, quote, democratizing it as they call it. Like, hey, get onto AWS and you have the SageMaker and the whole lot of it, just give us your data. Right. And the fact that it is becoming a lot more accessible means that a lot more folks are also interested without having to. You know, in the past it used to be, hey, you had to go to this lab, you had to get a PhD, you had to publish papers before you can even think about machine learning. And you know, the barrier for entry is a lot lower these days, but given the ubiquity of the application, I think it's important for everyone to understand what AI can do and the implications that it has. So for the better. But I wish we didn't have as much media attention.

So I guess, and I wanted you sort of answered the thing I was about to ask you and that is, is kind of as since you did your PhD and you did one in machine learning. And just to put things in context, you know, it was unusual, I would argue for lots of people to have machine learning PhDs at the point where you did in 2006 and you were done with that program and stuff. And obviously you've seen this change. Did you ever have any inkling where this might go in terms of the evolution? You've talked a little bit about the change over time, but just your perception of machine learning as you came out of your program versus your perception of it now. What really strikes you? Like what were the big things that when you do look back, you're kind of like going, wow, I never saw that coming. Or I told you so, I told you so.

So the fact that a lot of problems, I don't think it's fully there yet. Right. I think the biggest thing folks somehow forget is for any machine learning or AI to work on, at least this is my personal take, you have to have enough data to make it be meaningful. Right. And otherwise it's handcrafted rules. You do some tiny things here and there. Of course you could overfit like heck dnn, but this is, But I think what changed, I don't know if it is necessarily a change. The rise of all of these companies like Facebook and Amazon and you know, if I may, the Microsoft's of the world, right, The Googles and whatnot that managed to get as much user data and hence were required to deploy machine learning models. Did I predict the rise of these platforms as much and that they would aggregate these massive amounts of user engagement that it would be required? Perhaps not, but Google was sort of taking off at that point when I was doing my PhD as well. And Yahoo was one of these forefront companies. But within Yahoo, for instance, the fact that hey, from everything from news recommendation to figuring out what tags to put on your email to what ads to show was driven by a machine learning model. So that aspect was always the case, but that there would be these few centers where it's just going to explode and the democratization aspect of it, that it is also good for these companies to open source code there, not necessarily what is running in production. I know for a fact it cannot be the same models because we are struggling to adopt some of these frameworks when you're outside of Google and whatnot. But in that sense opening that up I think was a change that I didn't necessarily see coming. Like you would think that hey, this would be a secret sauce, this is something that we kind of want to keep internally in our company because this is the thing. But that there is so much learning and sharing of what machine learning is, that is as a field, the accessibility of it and the opening up of it to more and more people that I didn't see coming.

You know, just to make a point that you just said there, if you put that in context, as you were describing that I was thinking back about the time roughly that you were getting out of your PhD program, we were doing MySpace. And if you think about the evolution of social media from the days of MySpace as the first major platform that did not survive in a meaningful way to where we are today with LinkedIn and Twitter and Facebook and stuff, it's quite a change. In terms of the data aggregation, as you said, it's a completely different world.

And even how we shop, how we get information, how we look at videos, this much acceleration in how humanity lives packed into such a short period of time, sometimes I think I'm lucky to be a part of this. And you're always excited to see what it looks like. But on the other side, I guess with anything that gets accelerated, then comes all the challenges. Are we thinking about this. Right. And, and it's good that we've started asking these questions, but there are no clear answers. That's why we are here. Right. So the answers are also interesting to think and ponder and make sure we are doing the right things on.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog. It's better.

So, Suju, we've talked a lot about the AI industry in general and you know, interesting shifts that have, have happened and all of that. And I'm sure some of those are even visible, you know, within LinkedIn. I was thinking while we were talking, like, you know, how many people have switched their title from like data mining person or data scientists or like data analysts to like AI engineer or AI, you know, whatever. So that's super interesting. But I think more to the things that you're working on personally, I'd love to hear, particularly you, you brought up the some ideas around recruiting and that sort of thing. Maybe before we jump into the specific things within LinkedIn, could you give us a sense of how machine learning and AI are kind of starting to influence recruiting in a more meaningful way?

Sure. So historically, how would recruiters hire? Right. It came down to your network, of course, it was a physical network. You probably had your Rolodex. You'd go through contacts to say, I'm hiring for these candidates and so on, so forth. Or you would engage in some external job sites and whatnot. Right. But the burden then is I think this is sort of underappreciated to some extent. Extent. There are a lot of noisy applicants that might come through and you do not want to, as a recruiter, send the noisy applicants up to your hiring manager. That's not going to set you up for success. It also limits the number of candidates that you can give your bandwidth to. Right. You can't talk to 100 people in two weeks in any meaningful way to close them. So for sure, as there are more and more jobs that are not at this current point in our existence, but in general as more and more jobs.

I would say we're living in an anomaly. Yeah, that's probably true.

Taking that outlier out of the, hopefully outlier out of the picture, how do we help Our recruiters be have to spend their time on, let's say those candidates that are necessarily worth spending their time on, right? Because it's not just because you call up someone, they're going to immediately, you know, drop their job and move over. Right. You see, still need to work on the storytelling, as our recruiters call it, to convince them that this company is worth it. So if we could in some form or fashion make the hiring of our members be that much more efficient, I think that is the change that we are trying to drive as well. There is also a lot these days for sure. And LinkedIn is also trying out some of these interesting elements where, you know, maybe it is not all about going on something site, right? And proving over and over again going through the same leet code exercises over and over again to show that, hey, I can do this job, right? So what if you could have your credentials in some sense assessed by a platform, right? So we have all of these skill assessments that you can take, which I'm sure you've seen in some of these other places like hacker rank, decode and whatnot. Right? But think of it as your credentials that sort of travel with you. Right? And if you could, then not have to prove yourself on these same elements over and over again. But what really matters? What is a differentiation for this recruitment company or this particular job that would make you that much of an ideal fit? So it's a lot more about, given the enormous number of candidates that have their profiles online, how can we make it that much faster for the recruiters to hone down that smaller set and then spend their time in converting them? The same is true on the seeker side as well. There are lots of non traditional roles now. There are lots of options to switch carriers, right? You're not like a carrier employee anymore. People take these very interesting paths. And so how do we sort of help our seekers understand, here are all these career trajectories one can have. What are the skills that you require? Say if I wanted to switch to becoming a PM instead of being in the AI field for design or some such thing, right? So what would that path look like? So helping our members also be aware of, here are the opportunities, here are the skills that are required and in some ways even tying it up with LinkedIn, learning to say, here are the courses that you could take if you wanted to switch gears. I think that is where we want to think about recruitment moving forward. Right. Technology is interesting, it keeps changing. So can we automate the reskilling and the up leveling of our members numbers and continue to show to them all the opportunities that come with a new workforce, new marketplace. So I think that's where it's headed. It's also going to be a lot more data driven. A lot of our hiring managers and whatnot want to understand the landscape. Right. Like hey, I'm trying to hire this particular profile in say, Lafayette. Right. So what is my probability of getting someone with these skill sets? Do I need to broaden my search? What is the typical salary band that is being paid in some of these other places that I'll have to compete with and maybe even this world after we all come out of this Covid situation. Right. What does the future of remote working look like? What are the pay scales that you'll all have to think about at that point? So it's also a lot more of trying to understand from the broader trends that are happening in the industry. So it's not as niche as it used to be. Right. These tools are being made available in a data driven way. So yeah, hopefully that answered some of it.

It's really interesting to think about, like you were saying, the path that someone takes to a certain position and the changes that go along with that. Yeah. Because there's so many unique trajectories into particularly into machine learning and AI into those positions. You've got people coming from science, people coming from engineering, people coming from even other places, philosophy, liberal arts, economics, business. And so there are so many sort of unique trajectories. And I often get asked when I'm like doing a workshop or just talking to someone, doing like mentoring or something like that, like what are the steps I need to take like between where I'm at now and where like this position that I want or have in my mind, what's the sort of pathway there? And I of course have my own bias opinion about that. But it is really interesting to think about, you know, mapping out the trajectories and helping a person understand from where I'm at now to where I want to be. You know, how do people typically get there? Where did they jump off from and to. That's super interesting to me. And it also makes me think sort of like a graph structure sort of coming up. I think somewhere in I heard or I read about, you know, LinkedIn's knowledge graph and I was looking through a couple of the links in, in one of your blog posts with the papers and I saw some graphs. So yeah, I. That seems like a very interesting relationship where you've got like People who work for companies, who are in places, who are in industries, and there's of course, time in there and all sorts of things. So it seems like there's a definitely, even a unique data structure that's going on with the LinkedIn data. Is that correct?

Yeah, I think LinkedIn has been very open about calling it the Economic graph. Right. So how do we see the whole job space evolving? And to your point, yes, it starts with industries and companies. What are the skills that our members are acquiring? What are the sorts of educational qualifications people are getting, what are the titles that we have? And sort of standardizing all of this and normalizing it, if I may, into an internal representation so that, you know, I can look at, say, Chris's trajectory and then map it down.

Oh, gosh, oh, don't do that to me.

Or figure out, hey, how can we learn broadly that, hey, this is the movement in mass in terms of it's no longer, I don't know, things that used to be in Matlab anymore. Right. It's more of these other sorts of tools that are becoming that much more popular and being able to make sense of all of this data. Right. As you can imagine, not everybody writes the same things in their job postings or on their profiles. So how do we, if I may, project it down, even though that's not exactly what we're doing, but into a common vocabulary that we can drive products and machine learning systems across. LinkedIn is what we do. And besides internal LinkedIn, the economic graph is also used in lots of different forums. Right. To give insights into what are the fields that are hiring. Here are the top things that candidates or members are interested in. Here is how skills are changing across the world, and so on, so forth. And sometimes you also get very interesting insights in terms of behavior, how women behave, how men behave when they approach a job and whatnot. Right. So it unlocks a whole bunch of value. And I think that is the most remarkable asset that we have purely members sharing our journeys and career journeys and of course, from the marketplace of hirers. How is the job landscape changing, how are the skill demands changing and so on, so forth.

Yeah, so that's super cool, I guess, to continue building on kind of what you're doing, because I love where this conversation is going as you've kind of laid out how you're looking at the problems that you're trying to solve going forward. If you start thinking about where AI technology is today and these types of problems, where are you matching up various AI architectures or even approach in general to solving problems that LinkedIn and LinkedIn's customers care about. Where does AI fit into that large set of tasks that you guys are addressing?

Well, so I want to say in almost all product services that we have just given the scale of the data, right? So we're talking 700 million plus members any given day, 14 million to 20 million job postings even in my small domain, right? So there are things like hey, do we show job opportunities to people? How do we rank things on the feed given the number of people users to show up and so on and so forth. But maybe just talking about the talent space itself right now, making sense of the these member trajectories as an example. And again, while one view could have been hey, let's just pass encode all of these features and try to build some sort of a simple model which says hey, I'm not going to apply to this job posting. And of course that is restrictive. Then how can we not have to do with these sorts of standardized categorical features? If I were to look at the entirety of the text that the job posting has, can we leverage some of these recent, I don't know, fine tuned versions of Bert, say the internally but option to give us some of these semantic meanings because of not all job postings are created exactly the same, even though they might be talking about exactly the same jobs, right? So putting all of these into play basically to at this point we are leveraging, if I may, more deeper understanding of member career trajectories if I may, plus whatever is available on the job posting sites to be able to do more than am I just going to click on that particular job posting. It's even more of am I going to apply if a recruiter is going to reach out to me as an example, am I going to accept that recruiters connection and to get to that notion of understanding what the job posting is really about. And we've taken it a little step to further in our recruiter facing projects. Whereas a recruiter you perhaps don't need to give very, very explicit signals in terms of, you know, here are all these Boolean operations I want to do on the member places to get me that narrow set. But just identify a few of the candidates that you're interested in and behind the scenes we are able to then say hey, this is the role that the recruiter is looking for given their interests that have shown. So let's organically start pushing some of these matches, if I may, which we call recommended matches down to our Recruiters. So that's something that we're working on. Underlying all of these, I'm not going to say are incredibly complex models. At some point you could even set this up as an RL problem, right? Like say, hey, Sujo's benefit or whatever, the thing that she wants to optimize in her life is maximize her career potential, whatever it is, right? Or economic potential or some title potential or whatnot. Now, given that is my long term strategic play, what are the series of short term actions that I need to take to be able to almost be a coach to a member's career trajectory is my personal dream of where I wanted to take this thing. But the interesting part is LinkedIn is just one view of it, right? In the sense that I can see that, hey, Sesojou is applied to this particular, particular job post, which it sort of moves off of LinkedIn and we get very delayed feedback, if I may. So maybe after four months, if I'm an active LinkedIn user, I change my profile and then we, hey, hey, this actually worked, right? And now we are able to get these sorts of positive signals. So coming up with all of these proxies and trying to make our AI models be smart enough where at least we are efficient, right? From a matching process is where the current focus is on. But is there scope to do 20 times more? Definitely yes. And this is where we want to take the team.

So I guess as you look at all this possibility and all the things that you can apply different AI architectures to, it made me wonder as you were talking, how much crossover is there? I realize that you guys are owned fully by Microsoft and yet LinkedIn remains its own brand. How much crossover is there between the AI teams at LinkedIn as a subset of Microsoft and the various Microsoft AI teams as a whole. And just to ask, you probably don't have anything you can announce yet, but we're all interested in the fact that GPT3 just came under license from OpenAI. And do you foresee some interesting use cases going forward with GPT3 and LinkedIn?

Well, I'm not going to comment on the.

That's why I, I was giving you the out right there. So I recognize that.

But.

But I just got to say I can certainly think of some interesting, interesting things to do there.

Exactly right. I think the biggest benefit of all of these models, especially that in terms of how we want to represent, I don't know, the text that humans generate, right? Or even images or even how we want to think about knowledge overall in some senses. But the cost of doing that of course is non trivial. Right. So in, in these cases, can we go this fine tuned approach, sort of a route where we start with these pre trained models, then get it into the LinkedIn specific context and then see hey, end to end. If you were to optimize for an outcome of an hiring, are we able to see more benefits? This is an active piece of work that is constantly ongoing in LinkedIn. So trying to make sense of these glove embeddings. At some point, point it was legal if I may. Then there is but. And then there is Lee but that we are trying to do. And of course as this space evolves and the heavy lifting is done, maybe once or twice and then it's more about adapting these architectures to your internal use cases and seeing what the benefit is. I think it makes a ton of sense and it seems also wasteful in some senses for all of us to be doing these massive trainings over and over again that piggybacking on some of these methodologies to see what is the value that it unlocks for sure, right. I think the dream case is for us to show up at LinkedIn and say, Hey, I want to get to be X in the next two years. Help me right now. Can we make use of everything that we understand about our members journeys about the skills that they have acquired, you know, in some interesting world. If we could also, if the companies for instance, are willing to participate. Right. Like I want to up level my talent to this kind of a place. Like I want to upskill all of these folks. Now how do we have these personalized recommendations for folks in the career trajectory space? Can we learn globally and say, hey, this is what a career transformation looks for someone in the automobile industry, Right. I think it's definitely doable in some.

Ways, but I need some help here, you know.

Okay.

So I would love to get into a few of the, you know, practicalities of some of this work at LinkedIn. I've read a bit through some of the information about the, the, you know, recruiter systems and all of that. It seems like there's kind of maybe a couple of different areas that have been drawn out in what I've read. One of those is kind of the search side of things and representing candidates. And then the second side is more of a personalization side of things. I was wondering if you could touch on those two and if the representation side of things, like if you have a sort of, are you developing kind of embeddings of candidates within the graph of LinkedIn. And that's what you're leveraging in part to maybe pre compute that and get search done. How does that tie in?

Sort of building off of the standardized version of us looking at the data that we have. Right. So Daniel, exactly to your point, we do have a notion of what a member embedding needs to look like. And the data for LinkedIn comes from our LinkedIn profiles. Right. So in this sense, understanding that, hey, this particular member has a in that space sort of looks like people who are in this technical industry in this location. And here are the sorts of jobs that they kind of apply to as much as we can represent them in that number embedding space. Likewise, we have things that are on the job embedding side as well. And someone asking for a data miner, maybe because they were under a rock, is also still asking for any scientist. So how can we sort of bring that job embeddings view into it and in some senses be able to scale this? Because if you think about where there was an earlier question of what are the different surfaces in which we have AI, right there is the search aspect of it. Then we also organically recommend jobs to people. There is something that we call instant jobs where you want to be the first to apply. So members actually sign up for alerts to say, hey, the moment you see a job come into our system, then send me an alert right away. So you need this to be high, high, high precision as an example. Then on the flip side, on the recruiter search as an example. Right. It's all about matching members to your team jobs. So we sort of try to build this notion of what we call these two tower embeddings which represent the member side and the job side and bring that together along with more near line and real time features to sort of then personalize it to what that particular member is also looking for to do. If I may, our matching. Right. Of course, if it's on the search angle, then you also have the query context that comes into play. On the recommendation side, it's a lot more just based off of your activity signals and so on so forth that we try to infer intent to be able to make those recommendations. But to your point, yes, on the underlying surface we do have these embeddings and from what we are going to be writing a blog post pretty soon about at least this particular data layer in our talent solution systems.

Awesome.

Yeah, thank you for explaining that. And for listeners that are maybe new in the space as well, when we're talking about embeddings, we're talking about, you know, you could think about representing a candidate or member, in this case as a, as a series of numbers. You could encode them and you could define how those numbers should be yourself. You know, that you could encode that in a certain way. But you could also say, well, I may not be able to, you know, pick the right encoding for this candidate for my task. I'll let my model learn how to encode or how to represent this certain entity or the set of data, in this case, a set of data about a candidate and that sort of learned representation or learned set of numbers that represents, in this case, the job or candidate we'd call an embedding. So thank you for mentioning that.

Kind of curious as we've been talking about this and within the context of LinkedIn is very good if you, if you look at its, its long evolution, you know, from the very beginnings when it was basically just an online resume. And over the years you've developed that into really finding the person in that resume and showing those different aspects. Do you have any thoughts on, you know, and I don't, I'm not pushing you in an exact direction, but any thoughts on kind of how you find that personality, those soft skills, beyond just, you know, good at presentations kind of thing, you know, that are not strictly like a hard skill, like doing Python, but finding the person and trying to fit that personality type into different types of cultures and jobs? Because I would imagine that that's pretty complicated and that maybe any thoughts on how AI might be able to be used or anything that y' all might be able to share that?

So LinkedIn per se tries to, what we do at LinkedIn tries to learn from career transitions even. Right? So that's something that we have as E signal that goes into our recommendation models as recently to say, hey, this is the career trajectory that this person has went from here to here to here. In some senses, you could start thinking of how long did it take for this person to get there to say, hey, this person is like an outlier superstar, or this is like an average sort of a career path that, you know, could be something that the recruiters may be interested in. But you are asking an excellent question in terms of, hey, how can we factor in more latent aspects of what a member brings to a table, right? Not just that, hey, look at my Python proficiency score, but this person is really, really motivated, executor of tasks kind of a thing. Not yet. This is not something that we are able to leverage right now. But again, what we are trying to do in some of our recruiter facing projects, we call this our recruit assistance connect. This gives LinkedIn an inside view into how the candidate is sort of progressing through their different interview funnels as an example. Right. And now maybe from understanding and at some point, if all the interviews are moving to video interviews, we already have debuted a notion of what we called video introduction. And this to your earlier question, also makes use of Microsoft technology to sort of try to figure out, hey, how is the today it is with the intention of giving feedback to the candidate to say, hey, speak with more confidence or you know, you had all these ums and ahs and so on, so forth. So. So to enable the candidate to prepare further right now, with the permission of the candidate. And of course the company which is gathering all of this data can be at some point then learn from all of these behavioral signals. But I think it has to be more than just your resume at that point. Right. You need to have all of these other data points flowing into the system. So we are making smaller steps, but still within the world of what is being collected for the purposes of recruiting. But I can visualize a case for someone wanting to say hey, then, you know, in some very ambitious world, if we could then say hey, how is this person interfacing within the company? Right. How quickly are you able to start collaborating with your teammates? How quickly are you able to be up and running in being able to get something shipped to production and so on, so forth, Right. Does this data exist? Of course it does exist. Right now, are we. Do we think we'd be able to drive the right decisions? Do we think we'll be able to build the right models to tease apart the Latin signals to make sure. Again, the idea here should be to want to do something to propel everybody forward, right. And at the expense of nobody else. In some. In some, which I would we do it. Well, I need to think about this a little bit more. I want to say we want to get there. And more so more than anything else, that is inefficiency in the interview process, right. Like, hey, I've spent 20 years working and how are you going to judge whether I'm going to be good for my job? Of course there is some skill and art to it, but it's also exhausting that you have to keep taking and talking about these credentials, establishing this credibility over and over and over again every time want to switch something else. So how can we sort of package that into something more than assessments to say, hey, here is this candidate in my mind, I call it a credit score of sorts, if I may to that you can sort of carry along with you. But we are ways from there. But I think the building blocks are being put in place so we take the pain out of recruiting. Sorry, it was a little bit of fiction and a little bit reality.

I liked it.

No, it's good. I forgot. Appreciate the answer there. As I was asking it, I thought, I hope I was imagining if anybody was thinking about this it was going to be you guys. And I imagine you guys are going to be leading on this given your position. So thank you for tackling a tough question for me there.

And I have kind of a selfish question. I think from your perspective, it seems like you've spent a lot of time transitioning a lot of bleeding edge AI technology into the enterprise, into practical usage, into production usage. Even at LinkedIn, you know, I'm looking at some of these blog posts about the search and recommendation and recruiter product and there's links to, you know, research papers and you know, to some degree you're, you're tasked with bringing that, you know, bleeding edge research stuff into actual practical usage. And I'm just curious about your perspective or any recommendations that you could provide to other people out there who are wanting to use some of these, you know, maybe new AI technologies or new models that are published in terms of like the tech debt that comes with them and the things that you need to like, think about as you're transitioning some of this technology into an actual practical use case.

Thank you for asking this question. I think while a lot of us sort of get fascinated by, if I may, the technological advances in AI, the fact that we can ask questions out of an agent and express a meaningful and maybe almost human like response when it comes to actually putting these systems into production. I want to say that AI in some sense is an engineering discipline, but I think we are very much far away and it comes it's for a reason, right? It's not because we are unwilling to do so, it's just that the inputs are not as deterministic as some of these other systems. Right. So whenever you have a change in your data distribution or something happens, then how do you put in, quote, the right unit and regression test to make sure that you're going to be able to catch some of these errors? How do you keep track of the number of people and I can easily tell you the number of instances where, you know, a team deploys a feature and then there are 20 other teams with, unbeknownst to you, are starting to use this feature and the next time you make the change, all help breaks through, right? So how do you sort of have that discipline? And of course many, many simple things that we possibly do haven't spent enough time on in terms of feature lineage, when something goes haywire, how are we available, able to quickly monitor, alert, track, roll back to a previously versioned system? How do we recover from that particular point of view? Do we keep going back to reduce, if I may, feature bloat, right? Because it's very interesting as a machine learning researcher to keep on adding, here's some set of new features, here's some set of new features, but at some point you want to sort of take a step back and say, hey, holistically, how much have I improved, improved this particular platform, right? Have some of the changes that we have done reversed, some of the other changes that we would have pushed out otherwise? And even within a field of metrics, right, there is one layer of things that maybe we are pushing in the right direction. But then when you take a step back and say, hey, but what about long term engagement, is this something that we have been able to holistically measure? So the more the field progresses, I wish a lot of us were lot more diligent and cognizant of what it takes to maintain a machine learning system in production. Especially if you're supporting a large organization right there, there are lots of customers and the outputs of your model could be used in many interesting ways. So how do you have the right contracts in place, right? And say, at least in some machine learning teams, the idea is, hey, here is this nice thing that the PM comes up, then you quickly work on it, things work well, everyone celebrates and then you move on to the next thing. Then now who maintains, understands or even figures out, hey, has this model drifted with time? And as much as I want to say AI has been democratized, people who are not familiar with what to expect out of the system don't even know what a rollback for this model should look like or how do you even diagnose this thing. So there's been some good work from folks like in, in TensorFlow with TensorFlow X platform and these sorts of paradigms where we talk about AI metadata, which we need to be very religious about, which we need to start having health assurance and model monitoring and these sorts of things on. But as a discipline, as much as we want to train our folks in terms of all the latest and greatest in DN and tech, I wish we also had courses which talk about machine learning, model maintenance and the tech debt in schools as well. Because as this is going to just grow larger, it's going to become that much more important.

Yeah, and it sounds like, you know, maybe, at least at this point, hopefully that will change in the future. I certainly hope so. But it sounds like maybe in the interim it's up to us who are maybe leaders on, on teams or leading certain projects to really insist and, and ask hard questions around this and anticipate into the future what we might need to monitor and you know, where sources of bias are and what happens over time when we need to trace back what happened with a certain model. Do you have any other practical sort of tips for maybe team leaders who are trying to push their team in this direction and are maybe struggling?

Well, I want to maybe plug a paper. Google has definitely the machine learning depth of machine learning that, sorry, is the paper that came out a few years ago. But more recently they also released a framework of machine learning tests course which basically is a very simple thing which looks at, you know, the minimum score you could have gotten along either your feature monitoring, feature health, about your model monitoring and so on, so forth, which sort of gives you a simple rubric, if I may, to figure out how good is your system, right? Is it a house of cards which is going to collapse the first, first time something goes on, or is it a robust thing that can be deployed in production? So I think even adopting such a framework and most of these things perhaps are not even automated, but if you have been in the weeds and you understand how your model has been set up, you should be able to come up with this simple checklist to say, hey, yes, I think we are going to be robust to something catastrophic happening because it will. There's no reason to think that anyone's going to be an exception. That would be a first start to even figuring out with respect to, if I may, the health of something that is being deployed in production. Now this is like 101 if I may like the basics of what you need to keep in mind. But Daniel, you also mentioned something even more fantastic which is about there is like model biases and data biases in the data and then there is all the choices that are models make, right? Like because we were going after a particular product objective. Now what is the longer term impact that this may have overall in our user base? Is it going to be unfair to some segments of our population that's an even bigger ethical fairness question that I think is becoming more and more pertinent these days. But it's interesting, right? In one hand we are talking about like pure health and just monitoring of the system and on the other hand we are in the crossroads where we are even having to think about societal impacts, which I think speaks to the rocket ship speed at which this field has grown, but something for everyone getting into AI to be aware of.

Yeah, yeah, I think that's a really good perspective as we kind of come here to a close. I really appreciate you kind of bringing us in on a couple of those practical things because obviously you've developed a lot of great systems that have made an impact over time and it's great to get get that perspective. We certainly appreciate you joining us on the podcast and taking time out of your busy schedule to join us. And of course, you know, all you people out there on LinkedIn right now, make sure and post this episode when it's coming live on LinkedIn and you know, get that in the LinkedIn knowledge graph for all time. But yeah, thank you so much Suju for joining us. It's been a real pleasure and I hope we do get to meet at some type of networking event in, in the future in the real world. So thank you so much.

Of course. Thank you Chris and Daniel for having me. It was very nice talking about the future and the present of everything that people. Thank you.

If you enjoy practical AI, we would enjoy a 5 star review on Apple Podcasts, a blog post in response to something said on the show and or a recommendation to a friend or colleague. Those word of mouth recommendations really do make a difference. Practical AI is hosted by Chris Benson and Daniel Whitenack. It is produced by Jared Santo with music by the mysterious Breakmaster Cylinder. Thanks again to our partners who support this show's existence. Since shout out to Fastly Linode and Robar. That's all we have for you today. We'll talk to you again next week.

Um.