#357 Data-Driven Workforce Analytics with Ben Zweig, CEO at Revelio Labs

You could use AI to apply to, you know, 1,000 jobs a minute. There's a little over 5 billion job postings that we've scraped to date, about 90 million active at any moment.

It's often really hard to find the right job just for you. At the same time, hiring good candidates is also incredibly tricky. Jobs have a dirty data problem where job titles are often meaningless. It's difficult to know what other people do all day at work. What even is a senior data evangelist?

Consulting firms are now Hiring lots and lots of AI engineers and very few entry-level consultants. That is a huge shift. There are approximately zero companies that use ONET for their internal taxonomy.

Today's guest is Ben Zweig, the CEO at workforce intelligence platform Revelio Labs. Ben cut his teeth as a data scientist trying to do people analytics at IBM before realizing his models wouldn't work due to bad data. His response was to found Revelio, and he's just written a book called Job Architecture about how data can help almost every aspect of work.

I think a lot of Bayesian statistics is very difficult to use AI for. I really do think that we could see a world where labor markets are as sophisticated as capital markets, and that's a very exciting world.

I love it when data makes things better. Let's find out how. Hi, Ben. Welcome to the show.

Hey, happy to be here. Thanks for having me.

Yeah, great to have you. Now, I get two really common questions around getting a job, and one of those is, People are trying to get jobs, say, tell me it's really difficult to get hired. And then people who are doing the hiring tell me it's really difficult to hire good people. Why are those things simultaneously true?

One way to think about it is that hiring is a two-sided market. So, you know, in, let's say you are shopping for some fruit and you go to the grocery store, you pick out an apple and you say, this looks like a good apple. You pick the apple, the apple does not have to pick you. But you, you shop around for a job that, that employer also has to pick you. You're shopping around for, for a kid that they also have to pick you. So there's like, there's this like two-sided match that needs to, that needs to happen. And so it's always gonna be more complicated. Same with like selecting a university. Like you can't just pay money and go wherever you want. They also have to select you. There's a lot of markets that are kind of two-sided in nature. It adds some complexity, some friction, um, because you have to, you have to match based on some criteria. Now, what also makes that difficult is that there's some types of matching that, um, that are fairly straightforward. Like if you're, if you're looking for like, you know, a kidney or blood or whatever, you know, you have to have the right blood type. And if you have the right blood type, it's like either it's binary. It's like yes or no, it's a good match. But if you're selecting a university, let's say you want the right specialization, you want the right location, you want the right, um, you know, culture, vibes, whatever, you know, there's a lot to look into. And with a job, it's even more complex. There's a lot of dimensionality. You know, you have to, you have to be the right fit with the company. The position has to align, the timing has to align. There's a lot of complexity to work through, so it's hard to find the right match. And there's a lot of discretion in just, you know, interviewing and all that. So I think it's always hard just in principle. It's just a hard problem to, to match for. And there's not a lot of information. So, you know, individuals don't really know what it's like at a company. Companies don't have a lot of visibility into how well someone will perform at that company, what they really want. But I think it's getting harder because, I mean, for a variety of reasons, but because, you know, that matching process used to be more, more kind of costly in a way. So if you, if you were applying for a job, you'd write a cover letter, you'd do all this and that. Now there's like a lot of, you know, you could use AI to apply to, you know, 1,000 jobs a minute. And, uh, you know, the, there's not like a lot of, um, signal in like who applies for a job. So. You know, people feel, you know, employers feel kind of bombarded by a lot of applications and they need to select that, select their candidates using technology. So can't really, um, you know, evaluate cover letters to the same degree. So, you know, there's, there's a lot of noise in this, in this matching process now, which has made job search and match a lot more complicated in the last, couple years. So it's tougher now than it really ever has been.

Yeah. So it's interesting. I mean, you talk about like matching, so it sounds a bit like dating, right? I mean, there's a whole website called like match.com or whatever. So yeah, it's getting the right person to the right job.

Yeah. And it's hard to date. It's hard to find, it's hard to find a romantic partner, probably even harder.

Okay. So, uh, I guess the question then is how do you do better matching? So how do you make this process more efficient?

I think it starts with, it starts with having the right information and then goes into having that information be categorized and ultimately evaluated. So in terms of having the right information, one thing I'd say is that, you know, job seekers don't really have a lot of information about what jobs are even out there, what they're like. You know, people select into their occupations based on their parents, parents' friends, friends' parents, you know, like, like based on their, their network, which is very anecdotal. I mean, there's university career centers, which are close to worthless. They don't have a lot of information. It's basically like, how do you format your resume? So, so it's, it's really not, you know, I think job seekers don't have enough information and which is a problem because This is a very high stakes, very important set of decisions that people are making. On the employer side, they also don't have a lot of information. So, you know, you can, you can analyze someone's LinkedIn profile, someone's background, but these, these tools are not very sophisticated. I mean, so some of that is what we're doing at Revelio Labs. We're providing information to employers to do more, you know, smarter talent intelligence, smarter sourcing for candidates. So, you know, trying to, trying to get out there and, and help employers find, find candidates in a smart way. But that's really only half the story, you know, job seekers also need, need a smarter way to do it. So, so, you know, having that information is important, but then there's also categorizing that information. So let's take the job seeker's perspective again. It could be that first of all, there's a lot of language that people use to describe jobs. That is not so well understood. So you or I might know that when one person says lawyer and another person says attorney, those mean the same thing because we like kind of know the deal. And we might know that in some specific domains, but then there's other domains where it's actually really confusing. Like, you know, in engineering, like sometimes, you know, someone says DevOps, another person says CI/CD, and you're kind of like, well, are they really doing the same jobs? Like maybe in this context they're really kind kind of the same. So, so, you know, there's, there's cases where you might have a lot of language that is synonymous. So you need to, you need to understand like how to, how to bucket people, how to, how to make categories so that you're, you're actually categorizing similar jobs together. The flip side of that, which is a little harder, is sometimes you have a job title that can actually mean two different things in two different contexts. Maybe you have a product manager at Facebook, which is really an engineering lead. Maybe you have a product manager at Barclays, which is really a client success manager. And, you know, they use these titles in an uninformative, cavalier kind of way. So, I think in order to solve this problem, you really need to understand the content of each job. A job is kind of definitionally, foundationally, a bundle of tasks, a bundle of work activities. So, you know, and there's not a lot of good data on, on work activities that's, that's standardized. So, you know, creating structure and taxonomies for work activities is a difficult thing to do and, and isn't widespread. It's not, it's not easy to come by. Um, but that really is the way to, to find out what jobs are the same job. So that's the challenge. It's a solvable problem, and something that, you know, I spend a lot of time thinking about, but really, I think that is the core of the problem, you know, to get common taxonomies of work activities and of jobs so that you can, you know, understand what you're looking at when you're looking at someone with a job title, with a certain amount of experience, when you're looking at a job posting. To be able to categorize that into, into, into things that you do understand. And then once you have that categorization, I actually think the rest of it's kind of easy because then, then you can do match with, with the right, with the right information. So search becomes easier, filtering becomes trivial, and then the rest of the match process is just interviewing. Um, which, you know, isn't, isn't so easy, but like is, is, is about making sure there's the right, like, vibe check, culture fit, um, which is an easier problem than, than just making sure you're actually— you actually are coming in with the right expectations, the right definitions.

Okay, certainly, uh, what you're talking about, people saying words and it ought to be obvious, but actually you don't know what's going on— this is very close to my heart. Like, as a Brit living in the US, it took me a really long time to work out what arugula was. It like in Britain, we call it, we call it rocket. So it's like, it's kind of like, so, rocket, arugula is rocket. They're the same thing. And very few people know this. You've got to live in both countries. So I understand like getting confused about words is like a really common thing. And then people ask me like my job title, I'm a senior data evangelist. What is that? Very difficult to explain. Yeah. Podcaster works I love the time. So I love the idea that, like, job titles are kind of meaningless in a lot of the cases. You need to work out what they are. And you kind of went deep there in the philosophy of, like, what is a job? It's kind of a bundle of tasks. Can you give me an example of, like, how you define, like, the boundaries of a job? Like, I guess, for if you're a data analyst, like, are there a— what— because the skills aren't going to be exactly the same from one data analyst to another. How do you go from, like, these are some skills to— this is a real job.

Yeah, yeah, it's a great question. So, so I would actually not start with skills. I, I would, I would ignore skills in, in this, in this question because I think, I think skills are really attributes of people. They're really not attributes of jobs. I think that the job can really be broken down into, into tasks, and the person can be broken down into a series of attributes, and skills are, are one set of attributes. But, you know, it could be that the reason why I don't like to, to think about skills and jobs as being in the same kind of space is that you could have a job transform. You know, jobs change all the time when skills really don't change. Someone's in a job and the next day they say, hey, a client needs something. You know, who's gonna step in? Who's gonna do this work? Um, we need someone to kind of take over this project, you know, It's gonna be different than what you're doing today. Who can step in? And then you do it. Doesn't mean the skills have changed, but the job has changed. So, so that, that happens a lot. I, I think it actually happens like way more than people think. Very often someone comes into a job and that job is really a collection of responsibilities, a collection of things. And then, you know, 6 months later, a year later, 2 years later, they're doing totally different things. So, you know, even if someone has the data analyst title, um, maybe they find themselves, you know, morphed into something different, something new. Um, but I, I would say, yeah, I mean, what are the borders? You know, it's, it's a really tough question. I mean, let's say, let's say you come into a job. When, when you, when you have a job, you have a collection of responsibility. So let's say it's some distribution. Of activities that are your responsibility. Now maybe you have, you know, 20 different things all with different weights, you know, so it's, you know, it sums up to 1, let's say. So you have that distribution and no one's gonna have the same exact distribution that you do, but you know, it's gonna be close enough. So, so basically if you look at this distribution of activities and, and you represent that distribution as like a vector, let's say. Then you find out like, okay, you know, who, who else, you know, what are some other jobs? What are some other people with the same job title where, where that, you know, distance is, is low? And you say like, at some point you gotta, you gotta make some hard cutoff to say, okay, we are going to consider those to be a category. Because there's a, there's, it's a lot more convenient to think in terms of categories and taxonomies. I mean, that's like how that's just how our human brain works. We are natural categorizers. So, so we need to discretize this in some way. And by definition, discretization is about throwing away information. We are throwing out variation at some point. Now, you could have a very granular taxonomy and you could say, all right, you know, at some, at some level, you know, the difference between like a data analyst and like, you know, a BI engineer are like different, 'cause they're different. But if you are in a, you know, broader function, let's say you're in the finance department and you're trying to analyze like, you know, the breakdown of roles, you don't really care about that fidelity. You just want, okay, data people here. And then even broader, you're just like, all right. Technical people here. And so, so there's, there's this hierarchical structure of how you categorize things. And at that broader level, of course you're losing a lot of information, but that is fit for purpose for whatever, whatever the downstream, the downstream use case is for that practitioner. Let's say, you know, someone running finance for the op, for the organization, trying to report things to investors. Like they, they really don't care. About the difference between like, you know, BI and data analysis. Um, but like if you are in talent acquisition, let's say, and you have to recruit people to fill a certain, a certain opening, you have to understand a lot of the nuance between, you know, this role and a similar role. So I think these boundaries are fuzzy. And, but we, we, you know, we have to like make a cutoff at some point and where that cutoff gets made should be more a function of the, the application of these categories rather than like coming in and saying like, okay, this is like this, you know, um, this is like sufficiently distinct by some objective measure. I think it's subjective in nature.

Okay. This is amazing. I'm starting to imagine like different ways of like clustering things now. So I think we can probably nerd out on this, right? Very hard.

Yeah. Let's, Let's do it.

But, well, no, but maybe we'll save that for later. First of all, I want to talk a bit about like the benefits of this. Suppose you do have better data on like what a job is. You've got like these, or these better defined categories of jobs, like who benefits from this and how do you sort of achieve those benefits?

As an economy, as a society, there are basically two factors of production. There's labor and capital. And, and that's like, you know, a little bit of like econ. Econ, you know, basics. And the way we allocate capital is relatively scientific, pretty sophisticated because we have an entire sector of the economy, which is about the science of allocating capital, and that is the financial sector. And we don't have a science for how we allocate labor. And, and, you know, it is very ad hoc, very anecdotal. Obviously there's differences. Like we mentioned, labor markets are two-sided markets. Capital markets are not, you know, they, they are one-sided markets. A little, it's a little simpler. But another, another major reason is that we have armies of people in the world. We have millions and millions of people in the world that categorize data manually for capital markets, and they are called accountants. That is All they do, that is what they do. They, they are, they are manual categorizers and there's a bunch of rules. There's like generally accepted accounting principles. There's the Financial Accounting Standards Board. There's like governing bodies which dictate how certain things should be categorized so that, you know, people can make allocative decisions with respect to capital. Now in labor markets, we don't have those categorizers. I don't think we're going to. Wake up someday and find millions of people saying like, you're really this, you're really that. Although, you know, people have tried, you know, there's, uh, there's like efforts to do that. Um, but you know, now, now we have LLMs that can do that. So, so, you know, that, that's, um, you know, now it becomes possible to kind of leapfrog, um, where accounting has grown over the last 100 or so years. Um, but, but yeah, who benefits? I mean, ultimately, the, the end benefits are about allocations of capital. It's about matching between firms and employees. So that, that is really like how, how labor gets deployed. Now, that's like a little too abstract. I think there's, uh, there's other kind of intermediate, um, places where, where there's value. So, so one, one function I would say is, um, uh, there's a function and you know, every large organization is called people analytics. That is the analytics function for talent management, I'd say. I mean, I think that is like not the proper definition, but I think it's true enough that, that people analytics is really about analyzing internal data, um, to find out how to improve, uh, you know, operational performance, uh, you know, of the workforce of a company. Um, and to do that, they have to figure out where things are going well, where things are not going well, diagnose problems. And at the, at the core of all of that, it's about groups of people. It's about groups of employees. You know, employees have to be grouped somehow. You have to say, you know, like they wanna say, all right, we have, you know, a very high attrition rate of engineers in our Denver office. You know, that, that is, that is something they, they want to be able to say. So they need to know who's an engineer. Who lives in Denver? You know, these, you know, there's categories. Um, you know, who's in a certain business unit. So, and, and these categories are very often not good. You know, they are, they are categorized, you know, in weird, bizarre ways. I, I was involved in a project like this at IBM for, for years, and we were very constantly struggling with categories that were. Redundant or made no sense or weren't well-defined. And then the downstream analysis would be suspect because at its core, the categories weren't correct. Um, even more direct examples could be in like compensation benchmarking. You wanna know what to pay people and, um, and what level they should be at. And, you know, in order to do that, you have to know what are similar jobs. You have to, you have to define a job the way that the market defines a job. You wanna benchmark yourself to the market. Uh, there's other groups of talent intelligence. We mentioned sourcing, you know, you gotta find out like who to hire, where are there, where are there enough people? You know, what are supply-demand ratios? Um, it's a proxy for like market tightness. You know, there's, there's so much analysis that happens within a firm that uses these categories and, you know, it's a lot easier if the categories kind of make sense. Um, but then on the job seeker side, there's also a, a tremendous amount of benefit. Like you may wanna know, okay, if I start in one job, You know, let's say I start in a given occupation. Where do I go from there? What, you know, what are the exit options? You know, what are the follow-on occupations? It sounds like a simple problem, but it doesn't really work unless you have, you know, categories of occupations. Or they may want to know, you know, should I develop in this certain skill? And we mentioned skills are, you know, key attributes of people. You know, these are the inputs. Into completing these work activities. You know, is it worthwhile to invest in this, in this input? Depends on what outputs it creates. Um, and what are the associated, like, you know, salaries or career trajectories, um, for those. So getting a handle on like what, you know, what are these skills? How are they bucketed? You know, what do they lead to? You know, these are, these are questions that I think could just produce a lot more sophistication in how people, you know, allocate their own time and how firms allocate their resources.

Absolutely. The way you say it, it just seems like a really obvious thing that if you want to pay people the right amount, you should have some way of comparing jobs. And if you want to make sure that, like, you know what's happening with your workforce, you should know, like, what job everyone is doing, and it should be somehow comparable between different teams. And I guess Even with people outside your organization, you want to compare with your competitors and things like that. But so why is the state of data so bad then?

One, I can only really speculate, but one historical bit of context is that accounting never really considered human capital as very important because our accounting standards were developed in the age of railroads and like industrial, you know, businesses. Where the value of, of a business was mostly a function of its capital stock. And there's been a lot of analysis more recently. Um, most notably, um, Baruch Lev wrote a book, um, The End of Accounting. There's Capitalism Without Capital, you know, so the, the economy has, um, become a lot more intangible, a lot more based on knowledge work and human capital. And the, and the state of accounting, the state of categorization has really not caught up with that. Hopefully that's starting to change, but, but it's, you know, it's slow. There's another, there's another, um, reason that I think, I think this has faltered. Um, and that is due to the government taxonomy of occupation skills activities, something called O*NET. So O*NET is the, is, you know, is the US government's, um, taxonomy of occupations and things surrounding occupations. And O*NET, um, is done by surveys. Um, you know, they, they collect, you know, experts in employment and have them watch people and take notes with a pen and paper on like what they do with their time. And, and, you know, they collect 5, 5 observations a year on each occupation. You know, it's like really, um, not digital. And they, and they have developed a taxonomy that is useful for very broad analysis. Like if you are analyzing, you know, macroeconomic trends, you'll be able to tell the difference between legal professionals and manufacturing and nurses and, you know, like things that are very obvious. And so it's, it's used in like the public sector. It is not used in businesses. I, there, there are approximately zero companies that use ONET for their internal taxonomy. And it's really because they, they haven't gone deep enough into the idiosyncrasies of each business. So, you know, we, we did a recent project with NVIDIA where, where we were helping them with their, their taxonomy of occupations and work activities. And for NVIDIA, they don't care about the difference between doctors, nurses, therapists, um, social workers. Like, that's not their domain. That, that, that is completely irrelevant to them. But there's one part of the space where, you know, they really, really want to know the difference between like different types of hardware engineers. And like, you know, hardware engineering is like probably like 60% of their workforce and there's like different types of systems within that. So, and I don't, I don't even think hardware engineer is a category in ONET. I think it's just engineer. So if they use that, they'd have no visibility into the, the distinctions that matter for their organizations. And NVIDIA is a little bit of an extreme case in being like such a niche type of company, but really for every company, they have, they have a lot of distinctiveness in who they hire and how many people they hire. So kind of using a universal taxonomy that doesn't go very deep is, is just not sufficient. For them. So I think, I think that is, is a little bit of an unforced error. Like, I think, you know, this could have existed if, um, if there had been more, more resourcing toward this taxonomy, more granularity, more adaptability. So I think there's a little bit of like historical reasons why, why this hasn't been done. And I also think that the, the state of data in this domain is, is all text. Like, you know, when we think about work activities, we can get those from the responsibilities section on job postings. We can get those from the bullet points on resumes, and that is free text. People write whatever they want. There's typos, there's this and that. It's not like a dropdown where you say like what you're doing. And it really wasn't possible to get a lot of value from that data until LLMs. So we started analyzing that in 2017, 2018. And like, that was really like right when like Word2Vec and Doc2Vec were getting popular. Yeah, it wouldn't have been possible really in, in 2013 if someone had tried this, I think they would have not been able to.

Okay. So, uh, yeah. Uh, I love that the technology has finally come together to the point where we can actually build all this stuff. I guess, at a reasonable sort of, yeah, level of effort. 'Cause I was thinking about, I mean, you mentioned like NVIDIA and they've got very specialized roles. I mean, even at DataCamp, it's like a, I'm, we're a relatively small company, but like we've got a lot of different roles around variations on course creator. And if you go to OwnIt, it's like, oh, teacher, content creator, something like that. And yeah, that's—

You're certainly not gonna find data evangelist.

Absolutely not. No, which is actually a problem when I have to like put my name on like US immigration forms. Like, what's your job? Well, data something. Yeah, not sure. Yeah. Okay. So actually, I'd love to talk more about data careers. So I mean, because you've got this wealth of data around like what skills are being used and what jobs. So what are the most common sort of data science or other skills that are in demand at the moment?

Data as a field has progressed so quickly. I mean, faster than really any other field. I mean, when I started, when I started in as a data scientist, this was 2012 and my title was quantitative strategist, which is, I mean, this was at a hedge fund, so you know, they, you know, I guess like it was just in the quant category, but it was really data science. And I retroactively relabeled that data scientist, data scientist. But, but then at, uh, then went to IBM and it was like, you know, advanced analytics consultant or whatever, whatever it was. And then later got retitled data science. So, but then at the time, you know, data scientists were really basically statisticians. I, I, I think, you know, I came from a social science background, you know, I'd been an economist. I'm still an economist. That doesn't— can't give that up as much as we'd like, but that stays with you. So, you know, I think a lot of data scientists were quantitative social scientists, you know, running regressions, doing a lot of causal inference, you know, really, really like, you know, doing statistics, you know, basically in memory, like, like, you know, I, I had never touched SQL before getting to the corporate world. And, and that was, you know, obviously now I wouldn't like look at a candidate if they didn't like, you know, have experience with, with, you know, large datasets. So data has gotten a lot larger. So, so, you know, just like the, the history of it is, um, is progressing, is progressing quite a bit. There was, there was a, a moment where that became a lot more about supervised learning. That, that has, I think, gotten much more commoditized. BI has taken, taken over a lot of, a lot of space. But now, I mean, the, the big trend now is, is AI engineers. And, and that, that is, you know, people that can, I think of it like people that can vibe code stuff. And that is not so easy. You know, I think peo— you know, If you believe what people, uh, you know, post on LinkedIn and other places, it seems like it's so obviously trivial, but it's quite, it's quite complex. Like you have to, you have to know how to work with different systems. You have to know how to stitch things together. You have to know how to like, you know, debug, how to move things to production. There, there's kind of a lot of MLOps stuff that, that's involved in being a kind of, you you know, AI engineer and, and having it work with data is, is also not so simple. You know, LLMs are really built on, on text. Having them, you know, analyze tabular data is not, is not trivial out of the box. So, you know, there, there's a lot to figure out, but there's a big demand for that. Consulting firms are now hiring lots and lots of AI engineers and very few entry-level consultants. That is a huge shift. So that, that's like, you know, I think they, they are kind of leading the charge in, in this kind of role transition. But I, I think that is what a lot of data analyst roles are sort of morphing into.

Absolutely. I mean, certainly AI engineer, I mean, it's, it's the breakout career of the last year or so at least, cuz yeah, AI can finally generate code fairly reliably. And so yeah, even like without, uh, decades of coding experience, you can still make some progress, build some software, and yeah, make use of this technology. So, Pasi, Yusuf, I guess related to that, like maybe the most common question I ever get from our audience is like, what kind of careers are most sort of robust against being taken over from AI? I mean, you mentioned like junior consultants, it's like, it's a tough position to be in. Are there any things where you think is a little safer?

There are some things that, that technology is just not anywhere close to that are completely unrelated. Like, you know, if, if you want to, you know, if someone's like busing tables like that, that there are no robotics that have that sort of, you know, nimble capability. But I, in the, in the data world, it's a little hard to say. I mean, you know, in terms of roles that are robust, it's tough. I mean, I, I would say there are some like subdomains within, within data analysis and data science that seem a lot more robust to me. So, and, and, you know, you know, technology could make fools of us all. So, you know, this is like, as of, you know, whatever it is today, but I think, I think a lot of Bayesian statistics is, is, is very difficult to use AI for. And it really comes down to, to the fact that like we you know, there's a lot of, there's a lot of judgment calls in how you, how you design a model that works at scale and also works in very small subsamples. So, um, so I'll give you an example that, that we're personally working on. Um, you know, we, we analyze a lot of, um, online profiles, online resumes, and things like that. And when someone updates a transition, it's not always when they when they made that change. People don't update transitions right away. Sometimes they'll wait 3 months and then retroactively restate it. So we know there's some lags in how people report transitions. Fine. And we can observe that distribution by looking at multiple snapshots of data. And that, you know, there's basically like 2 sources of lag. There's user lag and there's like scraping lag and all that. So, so you have to like, you know, get a little crafty on how to combine these distributions. In an informationally efficient way. And then you have to create a model, a nowcasting model that, that gives you a, a prediction of like inflow and outflow rates that you would see in the absence of this lag. So you have to like design this counterfactual. But then what makes it more complicated is that you can't just apply it as overall distribution. You also have to make sure that it's, um, that it works for very tiny companies. Um, so, you know, let's say you want to analyze a 20-person company, you know, they're going to have super spiky inflow and outflow rates. And, you know, if you, if you apply some like blanket, um, blanket distribution, you're going to create some wild, wacky scenarios, or, you know, and, and like you're going to get some results that are very embarrassing. And the flip side of that is maybe there's some large company, maybe you want to analyze this at Walmart, and they have like their own idiosyncratic seasonality and whatever. Or, you know, or maybe, you know, some layoff announcement or something. So there, there's like, there's times when you want to be more general and times when you want to regularize toward, toward a population. And the, you know, the balance of those things is not real. There's no right answer for how, how you wanna balance those, those things, especially since there is no ground truth. You know, we don't know, you know, what the hiring rate of a company is this month. They don't report it ever, and even if they did, they'd only report it in retrospect. So, um, so we have to, you know, we have to make some call that, that kind of balances different needs. And that, that is something that I think takes a lot of, you know, like you have to really know the data you have to really know like how these populations interact, what is the real like source of variance. Yeah, it's just complicated. But I think this like subdomain is something that I think is just not how I imagine, you know, AI systems will ever behave.

So I have to say, when you started to answer, I was getting a bit nervous how you were like, well, it's tough to keep a job in data. But actually, yeah, you mentioned Bayesian statistics. I mean, I feel like It's still one of those slightly underappreciated fields of statistics, very important. And more generally, it sounds like these are kind of soft skills you're talking about. You need to be able to make value judgments, you need to be able to do critical thinking, you need to be able to translate between whatever your domain is, in this case people analytics, to some kind of statistical model and then going back again. These all seem like important skill sets to have.

Yeah, yeah, totally. I mean, even just like understanding how to communicate certain things is, you know, we're still gonna need human communicators, not just evangelists, but also evangelists, but also just like explainers of like how a model is working, how it's thinking. And so being able to sort of do the diagnostics and like trace at every step of the model, is it like behaving the way we expect it to behave, is something that we constantly are struggling with and trying to do a better job in. And that's something that is not going to go away. I actually think, you know, since LLMs, we're hiring more in on the modeling side of data science and more on the MLOps side too. I think the roles where AI is diminishing our labor demand are probably more kind of roles in front-end work and maybe scraping and things like that. Engineering where there's a lot of replicability. So, so I think the distinction with data is that, you know, every, every problem, you know, every data analysis problem is really different. You really have to like study the domain, the problem very intimately. But if you are creating a web app in React and you want to have some widget that does whatever, like that could look like every other dashboard on the internet. And, and like, there's a lot of examples like that. Now, obviously, like, there's still a lot of interesting, cool work to do on the front end, but like, there's so much of it that can be automated more easily.

Absolutely, yeah. So if you're doing the same task over and over again, you've got something repetitive, then that's prime candidate for automation. If you're doing these kind of, I guess, more boutique things, like something more artisanal, which I guess a lot of this sort of data science modeling work is, then that's not going to be automated. All right. And so you mentioned you've been hiring for some roles more, like the modeling roles and Ella and or so, uh, or modeling ops kind of stuff. Um, in general, like, do you have advice for hiring managers? Like, do you need to start looking at different profiles?

For different profiles. So, so I think we, we've had a lot of success in hiring, um, people from PhD programs. Um, usually for whatever reason, it happens to be in like either economics or physics. I don't know why those are the, the areas that where we've had the most success. Doesn't have to be, you know, I've worked with a lot of great, great data scientists from like, you know, other things like, you know, political science, chemistry, whatever. I think the reason why I think those domains are particularly strong is that they are, they're, they're areas where you can embed a lot of theory. So, so I, I, I remember I interviewed one person from, um, from a degree in math. In mathematics and he does PhD in mathematics and I was asking him like, well, what's his dissertation in? And he said something about like, mixing liquids. And, you know, so the extent to which, you know, like let's say there's an oil spill, like there's some diffusion of that oil into the ocean. And if it gets mixed, like if you stir something, then that diffusion happens faster and trying to, you know, quantify like, you know, how to manipulate diffusion rates. And I said, you know, that's really interesting, but like, how is that different from like, whether, you know, let's say you had had that dissertation, but your degree was in physics and not mathematics, like, how would it be different? And he said, it could be written by someone in physics, but what physicists would do is they would actually plug in the parameters and they would say, well, we know the density of saltwater and we know the density of oil and we know the, the like parameter for gravity. And like, you know, we have all these parameters that we're just gonna like plug in and, um, and then it becomes an easier problem. But a mathematician would, would create a generalized solution and they would say, you know, you could plug in anything and, and it's a more reusable abstract thing that, that is, um, generalizable. Use it in space on different planets, and you could use it on some alien planet that will never exist, you know. Yeah, you can, um, yeah, you can extend it to, to all sorts of, like, you know, different environments. And I, I think, um, I think very often— and, and economics is the same way, you know— there are constraints on, on models based on what we know about the real world, you know, like, like you have, you know, you can't have, um, you know, certain things go negative or whatever, you know, there's, there, there's things that we can, there's ways that we can manipulate models to make them a little bit more informationally efficient in an environment where, where we have small samples. And I think that actually does describe the real world in most cases. I mean, we have, you know, many, many petabytes of data, you know, we have tons and tons of tons of data, and yet we are still trying to squeeze out information efficiency for small subsamples. I mean, I think Andrew Gelman said like, there's no such thing as big data. You know, there's only small data that, that, you know, um, once you drill down deeper, you realize, you know, your datasets are actually quite small. And I, I think, I think the, the, the comfort with, with being able to kind of like customize a model for, you know, the way that you understand the world is, is something that, that comes in handy. So I think those backgrounds are great. I also think master's in data science programs are excellent. Um, you know, the, you know, I'm based in New York. There's, there's one at Columbia, there's one at NYU. They're, they're fantastic. I mean, people graduate from these programs and they know how to do everything, you know, that they, they are great engineers. They know all the tools. They, they are generally really smart. And it's also, I mean, it's a tough market for them. So I think you can actually, like, as an employer, you know, it's a good time to be hiring.

Absolutely. Yeah. Yeah, I feel like more companies should be able to take advantage of this. I mean, it's particularly when you've got, like, people who are very highly qualified and then, okay, you give them a job. If you've got a highly qualified data scientist, they should be doing more than just maybe building a dashboard, you know, take advantage of them. Super. So, uh, earlier on, uh, we were planning to nerd out later in the show. I think it's time. So, okay. Uh, you've got basically a whole internet worth full of job posting data, loads of LinkedIn profiles or whatever, like, uh, details of people. How do you go from all that like messy sort of web-scale data to I've got, um, a set of jobs in a taxonomy?

So let's say, uh, let's take job postings as an example. So there's, uh, a little over 5 billion job postings that we've scraped to date, about 90 million active at any moment. And, um, and so, so, you know, one is like, we have to scrape that, we have to collect it. Um, and you know, that has to be every day, you know, we have to get, you know, start dates, end dates and all that. But then there's sections. So sometimes you get some metadata from these postings. That could be job title, geography, company name, post date, things like that. But then there's, there's sections within the, the text that can vary by posting to posting. So there could be, you know, a paragraph on information about the company, a section which has bullet points of skills or qualifications, a section on responsibilities, you know, maybe a little paragraph about benefits or whatever it is, you know, so you have like different different sections of these postings. And, um, and of course, you know, we want to categorize them. So, so there's a few different categories that, that we, that we create, which really comes from like different, um, it's, it's really, it's really based on like our theory of how, of how jobs are structured. So, um, one is occupation. Um, so, so that, that is, that is one way to categorize jobs. Another is seniority. And by the way, a title actually contains two pieces of information. It contains information about the occupation, contains information about seniority. Not all the time, by the way. Sometimes you have someone whose job title is director. I hate when that happens, you know, but, uh, but, but sometimes they don't have any occupational information. They just have seniority information. Or sometimes you just have occupation information and no seniority. Sometimes they say marketing. You're like, well, you know, is that a junior marketing analyst? Is that chief marketing officer? You know, like we don't really always know, but, um, but despite that, usually a job title contains information about, um, about occupation and seniority. And then there's skills. So that's another. And then there's work activities. So, so those are the way we kind of categorize the jobs. Of course, there's also categorization. In things that aren't related to jobs. We have to categorize companies. So, you know, someone says they work at Zoom. There's like 11 companies in the world called Zoom. And, you know, if we just take the modal company, then everyone will get classified to Zoom Communications. And, you know, if there's like Zoom Consulting or Zoom Moving Company, then they'll have zero people and that's no good. So, so, you know, we, we, we need to have some like probabilistic categorization, um, to companies. Um, then of course we need to categorize geography. You know, someone says they're hiring in the New York metropolitan area or the broader New York, you know, like that, that could mean a few different things. Sometimes they'll just say Upper West Side, and then you have to know that that's like in Manhattan, which is, you know, and like there's complexity there. So Manhattan is a county, it's a borough, which is a county within New York City, but very often counties are broader than cities. So, you know, it's, you know, so sometimes there's like some many-to-many match. So there's like some complexity in how to categorize things there. But if we just think about the, you know, in the job space. So we kind of have made a call to, you know, start with activities. And then cluster occupations before, you know, sorry, once we already have activities. Because we would like to define an occupation as a bundle of activities. And then skills is also done in its own, well, actually skills is also done after activities because we'd like to use skills as an input into completing work activities. So, but it really starts with activities at the center of it all. So we, we get that from, um, so from, let's say from the job postings, we parse these different sections and the responsibilities section. That is, that is the, the corpus of, of activities. There's, you know, 5 billion postings, maybe like 3 billion of those have activity sections in English. We, we only train this on English so far. So we train it on English, we infer it in other languages. So there's like always a difference between like your training and your inference. But yeah, so we have, call it 3 billion or so paragraphs. Each paragraph has like, you know, let's say 5, 6 sentences and maybe sentence has like different chunks. So let's say there's like, you know, call it 15 to 20 billion, like, activities that, that we say sentences that describe an activity. Then those need to be embedded. So, so we need to have an embedding of those activities. Um, so, so we, we use the, a kind of BERT-based model to, to embed those activities. Um, choosing the dimensionality is of course, uh, uh, you know, a, a thing to do. We also have to decide whether we want to control for industry. You know, there's a lot of like, things, you know, decisions that have to be made to get that embedding. But once we have those embeddings, then, um, then we need to cluster those activities and label them. Um, so that's, that's a big challenge. Yeah.

All right. No, uh, that's interesting. So you got like giant dataset, it's messy. So basically, uh, my sense is you clean up, try and standardize it in some way, and then you're taking out the duplicates, and then you've got some sort of, I guess, smaller, more standardized data set you can then work with to feed into your model. So you're about to talk about modeling.

In some way for us, like the model is the categorization. I think like we just happen to be in the space where we are really, you know, our company, Revali Labs, we're a data company. Like we're not really trying to make prescriptive recommendations. We're trying to deliver data that can be, you know, nice and neat and embedded into a traditional analytics workflow. But, you know, yeah, the, the, so when we create a taxonomy, that, that kind of is the model in a way. Like, like we are, we are really, you know, there's a lot of submodels within this. You know, obviously there's the embedding, there's the, you know, garbage detection, deduplication, cleaning. Parsing, you know, all that. And then there's the, the, the sort of, then there's the clustering and that has to be like agglomerative in some way. Then there's the labeling, which is a whole other can of worms. And that really wasn't even possible until generative AI. So that's, uh, and, and that's like getting, getting better as new models come out. The labeling's really tough, especially because, you know, with job titles, there are plenty of duplicates. Like you could take, you could say like, oh, for this given cluster, like what's the modal job title? And that's already like a pretty good, pretty good way to go. But for activities, there's 12 million sentences and no duplicates. No one writes the sentence in exactly the same way. So, so yeah, I mean, being able to synthesize all those and label them, it really requires some generative AI.

Actually, I'm curious about that because you're working with primarily text data and it seems like generative AI LLMs are the standard modern approach for dealing with text data. Do you ever make use of any more, I guess, traditional natural language processing techniques? I mean, you think about like topic modeling and named entity recognition, all these things that were like really popular like a decade ago. Are they still in use?

Yeah, yeah. So I would say, I would say with, with named entity recognition, not really, because I think LLMs are are really just like, you know, like, like if you, if you put in a paragraph into ChatGPT and say like, do NER for me, it'll do a really good job. So, so I think it's like usually like overkill and like probably expensive and slow. I mean, it is expensive and slow, but it'll be so good. And so, so I, I think like NER is not really necessary if you're dealing with a small enough dataset. Topic modeling less so. So topic modeling, we actually do still use more traditional topic models. I mean, I think they're, they're like, we don't use LDA, which was like the kind of gold standard topic model, you know, pre-LLMs. But now there are kind of like, you know, transformer-based topic models that kind of work the same way as, as, you know, as LDA, but, um, but they're just like generally a little stronger. Um, but functionally the same, but, you know, internally a little different. But for those, yeah, we, we use those. Um, I mean, I'll give, I'll give you an example. So we, we, we collect a lot of data about employee sentiment. This could be set from sites like Glassdoor, Fishbowl Blind, et cetera. So, so where, where people post employee reviews and we want to find out like, what do people like about this company? What do they not like? What are the topics that are emerging? And so we need to, you know, extract topics from text. I mean, it's a perfect, like, textbook example of, of using LDA. So for that, we, we, we had extracted topics that typically show up in these reviews. I think we got like 47 topics or something. And that, and now, you know, we have to, so first we have to create those using topic models, but then You know, we have to infer topics based on every new review that comes in. And that's, that's, those are the categories that, that we use. I mean, it's, it's completely traditional. And of course, you know, we, we use like, you know, regex and cleaning and what, you know, all the, all the normal, you know, all the normal basic, uh, you know, NLTK, uh, stuff. Um, But yeah, I think there's still value in the traditional NLP sometimes.

Okay, that's interesting. I guess like once you do have these giant datasets, then yeah, working with LLMs, it's computationally expensive. And I guess like there's a dollar price attached to that. So having these sort of older techniques that are a bit cheaper to run, it still has some value there.

Yeah, I mean, I'll give you an example, like for translation, we cannot do translation on our descriptions from postings using LLMs, using like generative LLMs. It's just too expensive. Like it just, um, they would do really well. We really, it would be prohibitively expensive. I mean, it would cost us like a million dollars a month or whatever, like some tremendous amount of money. Um, so it, you know, maybe someday, you know, like I'm sure some businesses are able to afford that. But yeah, for us, like we have like more traditional ways of like detecting language and doing more like, you know, substring, you know, translation. So, you know, that's a constraint that we have to kind of deal with just because LLMs are expensive.

Absolutely. All right. So before we wrap up, I'd like to know, what are you most excited about in the world of people analytics and labor analytics?

It's a great question. I'm excited about all of it. I mean, I think, uh, I think, you know, it's, it's, um, I think a lot about categorization. Like, I think that is, you know, I mean, mostly because I just wrote a book about it and, and I, and I want to, you know, like, it's hard to not obsess over something when you're like deep in the weeds of it. But also, I really, I really do believe that that the core of our difficulties in, in labor markets generally are about a lack of the ability to categorize and standardize and compare. Like, we, you know, analytical HR, you know, the, the part of HR that analyzes data does not compare across companies. It's just not really done. There's no benchmarking and there's no sense of how, how you compare to peer companies. And I think that's like, you know, a little sad. You know, I think, um, Michael Porter, who's like the father of modern strategy, had said like strategy standing on one foot is about differentiation, like end of story. And you know, that, that is like, you know, in a nutshell, like, like what it means to be strategic. And that's not being done in labor markets, which is, which is unfortunate, but I think it could be done. So I really do think that we could see a world where labor markets are as sophisticated educated as capital markets. And that's a very exciting world to me. So I want to help, you know, help build toward that, you know, lots of hard problems to work on, of course. But, um, but I think that's exciting and I think we could see that in our lifetimes.

That's quite a compelling vision, really, like the idea that we're optimizing for everyone getting their dream job.

Yeah, exactly. I mean, that, that, that is like, there's so much human value in that. I mean, this is like how everyone spends their time. Like, what could be more important than that? You know? And, and like, whatever cause someone cares about, whether it's politics or climate or this or that, like, that is all downstream of people, you know, doing their job well and having good matches in, in how, in, in how people work. So yeah, I think it's, I think it's, you know, helpful for everything.

Everyone gets a happier working life. We get, we make more money. Uh, it sounds like a dream come true.

Wonderful.

And, uh, just finally, I always want more people to learn from. So, uh, whose work are you most excited about at the moment?

Great question. I, I, so I think there's a lot of, um, there's a lot of work being done in like the economics of AI. So, so how will AI affect labor markets? And there's some, there's some really interesting thinkers in that. In that space that I think are, are very exciting. I mean, I'll make another, I'll make another plug, uh, for, for two more podcast recommendations. So one is, uh, called Justified Posterior, um, hosted by Andrei Furatkin and Seth Benzell. Really wonderful researchers, great, great podcast, just talking about like papers and like how, how, you know, AI is affecting the macroeconomy. And they, and they have, you know, really interesting guests and talk about interesting things. There's, there's lots of interesting work done by, you know, David Autor, Daniel Rock on these, you know, these types of things. They're great. And the second podcast I'd like to plug is my own. It's The Economics of Work, but it's, you know, I think we're really, you know, I think the goal is to try to like find people who are thinking deeply about labor markets and how they operate. You know, there's so much of just to complain a little bit about labor economics. There's so much research on like, oh, how can we more precisely estimate the returns to education? And I think there's such, there's such a big opportunity now to think more foundationally about like, what are the mysteries of the labor market? Like, how can we improve search and match? Like we spoke about, like how, how can we analyze the, you know, vulnerability of jobs to AI and. How remote work is transforming the world. Like there's, there's so much interesting stuff to think about. So I don't know. I mean, there's, there's like a, there's not so many people, but there's, there's enough that, that really are thinking deeply about these big questions. And I, I think it's exciting. It's the right, it's the right time for that.

Absolutely. So economics has this bad reputation as the dismal science. So it's like people who are trying to make it a bit more fun and go, go for it. I like it. Uh, looking at the mysteries of the labor economy. That seems a useful thing to look at. So making it a bit more exciting.

Yeah, I think so.

Wonderful. All right. Thank you so much for your time, Ben.

Thank you. Yeah, thanks so much for having me.