Bob Pulver Live on Shally's Alley

All right, hey everybody, you are live on Shelly's Alley, the show that puts out the fires caused by your burning questions. You like that one, Bob? My guest today is Bob Pulver, a former IBI— yeah, a former IBM, uh, think tank person and expert in AI IQ, which is AIQ. Um, I invited Bob today to talk a little bit about responsible AI in, in TA and in human resources. Um, Bob actually has a show, a podcast here in the Work Defined network, or I don't know, is it network, uh, distribution? What is Work Defined? How do I explain that?

Well, officially I call it the Podcast Network, but like to think of it as a syndicate.

The syndicate.

Sure, let's go with that.

The Kaiban, or— anyway, so if you go to head over to Word Defined, you'll see a list of podcasters. I've recently been picked up by them and I'm being distributed there as well. And Bob has a long-established show there. All about elevating your IQ for AI. So, AIQ. Anyway, and that's how we ended up connecting him to the show today. Bob, you've positioned yourself at the intersection of responsible AI and human resources. But as I go out there and I travel, and I know you recently went to the Transform conference, which I I want to ask you about as well. I've noticed that AI is increasing in use in recruitment, but it's also starting to raise alarm bells around algorithmic bias, something a lot of folks probably aren't aware of, intersectional bias. Can you share a scenario from any of your audits, you know, privacy, names removed or whatever, a scenario from any one of your audits or advisory experiences where AI-driven recruitment significantly disadvantaged a specific group through some sort of intersectional bias or something like that? Can you give us an example of something that you encountered that you've obviously not naming any clients, but the situation or the conditions?

Well, I will tell you, first of all, Charlie, thank you for having me. And your whole opening there, you're— this is something I need to strive for. I gotta get— I gotta up my game in my own, uh, show because I love the The opening sequence there. Um, the— I guess some of the things that I've seen, or where people are more, um, they're not thinking deeply enough about what this even means, right?

They just push the button like, this is cool, let's do it, and they push the button.

Yeah, like, we gotta do— we gotta do AI, we gotta do AI, everyone's doing AI, right? We're never gonna get through all these applications, so we gotta sort of offload some of that. But so a lot of it goes back to, like you were talking about, this AIQ concept, which I think about as AI literacy and readiness. And so there's a lot that goes into that. And some of it is, as you were alluding to, the principles around how do we be as responsible and fair as we can as we deploy AI, as we build AI. So we've got to put more pressure on the vendors in this space, which I just saw quite a few of those at Transform, but holding them accountable to be more sort of trusted, right, is a whole other set of questions that we can start to ask of these vendors. But in terms of responsible use throughout the talent lifecycle, so, you know, TA and beyond, as we think about, you know, pay equity, as we think about promotion opportunities, even learning and development opportunities, are part of, you know, someone's, you know, career potential and livelihood and compensation, all these things, as far as the EU AI Act, they would say that these are, you know, high-risk situations. And so we just need to be very thoughtful about how we, you know, trust these vendors, how we understand what data we're using, to train these systems because the output, if you mentioned algorithmic bias, if there's bias in the output, in the training data, yeah, then obviously it's gonna learn from that. It doesn't necessarily know, you know what—

It doesn't know what bias is. It just knows, yeah. Exactly. Isn't it true that the way that these AI systems are trained, they just have a natural tendency towards dominant categories. So when something is dominant, when something repeats, the themes that repeat the most often, it sort of takes that almost like the popularity is the truth, like the popular content is the true content, even if it's not necessarily the best content. Is it true that AI basically has a built-in bias towards dominant classifications?

Well, I would say in large part that's a fair observation, but I would also say that, you know, if we think about our historical decisions, you know, human-driven decisions, you know, it's a result of that historical bias, right? So even if you get a solution that has been effectively trained and it's been audited to make sure that the adverse impact is mitigated, once you add other new datasets that maybe, you know, haven't been sort of, you know, cleansed and there's bias in that data, now it's going to be trained as it goes through reinforcement learning and other types of training with new data, right? All these LLM-based tools and all these AI tools continuously, you know, absorbing and ingesting new information, right? So if there's bias in that new data, So let's say you trust a new vendor, a new ATS, maybe they've gone through an independent bias audit, you take it in and now you say, oh, you know what would be even better is if we took some of our historical performance management data.

And that might be right, okay.

Yeah, then all of a sudden it starts to pick up these new patterns. It starts to think maybe I should be prioritizing white men over, has some, you know.

But see, that's exactly my question, right? It's like, like if you were to ask an AI today, the way the general, you know, generative AI is right now, if you were to ask the AI what is the best flavor of cookie, uh, you know, or what is the best cookie, it's gonna give you an answer based on all of the data that it has and which flavor of cookie or which type of cookie is the most talked about, the most written about, the most popular one. So it'll say, you know, chocolate chip. But it is ignoring, for example, in, you know, trying to bring this back to recruiting, it's ignoring cultural bias, right? Like if you were to ask an AI that was raised on an Italian culture, then the favorite cookie there might be, you know, the wedding cookie or, you know what I'm saying? So like, isn't it true that basically we are essentially training them to recognize the majority which no matter what you do is always going to negatively impact the minority. So, you know, it's going to say chocolate chip and ignore snickerdoodles or whatever.

Well, I think part of this, as you think about what, what's the objective of using the tool, what new insight can I derive, you've got to understand what are the right parameters. So, shall you know this, when you're creating a, you know, a Boolean string to go do, um, outbound sourcing, you've gotta increase its own, the tool's intelligence by saying, look, I'm looking for this, I'm looking for this person. So you're not looking for the best tasting cookie, which is subjective into itself, right? I'm looking for a cookie that's got this texture and this, whatever, low, maybe it's gluten-free, maybe it's whatever, right? So the more specific you can get, in training it for exactly what you look for, then it can sort of adjust its weights for those different attributes of the cookie or the candidate.

So what you're saying is that we can essentially educate that, we can counterbalance that proclivity towards dominant traits by specifically saying, we don't want to pick out dominant traits, we wanna pick out these traits.

Yeah, I mean, if you're having, if you were, um, let's say if you were having people over, over for dessert and you needed to get more than one type of cookie, you'd want to know what people's, you know, preferences are, what their allergies are, what their, you know, things like that. So, so you've got to get, you know, pretty specific, which is why even if you're using AI to like create a job description, you want to look it over and not completely trust it, right? You want to look it over and say, okay, what about this, what about that, or, you know, maybe that should be in the must-have category. As opposed to the, you know, nice to have, you know, a bulleted section and things like that. So we've got to just be careful about injecting our own biases as we calibrate some of these tools ourselves, right? As you know, some of these sourcing tools or ATS tools, you'd say, you know, show me, show me more candidates like this, less like this, or whatever. But, but it would be more helpful to say exactly why, right? So a lot of times I see these tools where they might say Yes, maybe, no, and then give me a reason for the no. But I don't see a lot saying, why did you say yes? Or why is this a maybe? What's preventing it from being a yes or a no?

Right.

So, so that level of fine-tuning and calibration, I think, can go a long way. But you've got to recognize that historical data is riddled with bias because humans are biased. Entities and they make biased decisions, whether we realize it or accept that or not. We've got subconscious and unconscious biases.

Right, right, right. And I also don't want to, I don't want to throw bias. Yeah, right. I understand that we have, right? So there's historical bias, there's bias that we literally train the tool with. So the data may not be biased, but we are, et cetera. And I also do want to clarify that I'm not trying to throw bias under the bus here. I mean, we do need bias biases. I think bias is an important part of human existence in that it helps us recognize patterns. Some patterns may be unhealthy for us, so it helps us avoid dangerous situations. Like, for example, if you're a driver of a car and you have a bias towards angularly moving objects, it's going to save your life because you're going to avoid a car accident, right? So, bias is necessary. It's just being aware of the bias. And that's kind of what I'm What I'm getting to, have you seen, can you point to a scenario from your investigations or audits or experiences where there's been an unintentional, like give us an example of an unintentional outcome that disadvantaged a particular group, whatever group, because of the implementation. I'm not looking for like a person to blame or anything, just a scenario of like what could go wrong. Give me an example of what could go wrong.

Yeah, well, I think there's the protected categories that we all know that have been protected from, you know, just basic civil rights, you know, legislation going back, you know, 50, 60 years. There's newer biases where, you know, you could check for age discrimination, you could check for disability discrimination. So, you know, just speaking in generalities, I would say a lot of it is, you know, can we definitively determine that— not— I think to your point, there's, there's the, um, there's the attributes themselves, right? Like this person is Asian American, African American, you know, and then there's obviously gender, and then there's the intersection of those things. So, you know, African American you know, females were at a significant disadvantage for this type of role. So we see that. These are some of the ones that certainly make headlines, right, where we're only hiring, you know, male engineers, or we're only hiring—

Right, because they're so obvious. But that's, I mean, the intersectional bias is, I think, is partly where the really big risk is, is in that unknown, you know, because we know that we have a propensity for bias in gender and race. And we're looking for that. But we're not looking for the unintentional intersectional bias, which I'm happy to explain. Have you encountered a deployment where nobody expected to find this kind of bias and they found it?

I haven't. But to be fair, I mean, I haven't done— I learned how to be an auditor and I got certified to do audits. But that's not really what I do. I'm more consulting and advising companies on both the vendor side and the buy side. So TA teams on—

So you haven't seen the train wreck, but you know how to prevent it.

I haven't seen the train wreck, but based on my relationships, not just with vendors who have gone through audits, but with some of the auditing firms. So I have relationships with a couple of the AI governance, AI assurance.

You're helping people not build trains that don't derail, basically.

Yeah, well, it's like, let's get the message out. Let's, let's evangelize, you know, this being responsible by design and some of the responsible AI core principles around, you know, can you explain what this algorithm is doing? Can you be as transparent as you can about how you're using it, why you're using it, where in the spaces that you're—

can you opt out? Why should you opt out? Yeah, okay.

Yeah, all the, all the requirements that go with that, right? Like, once you go through the audit, you're supposed to actually publish the the results. So if you're not publishing the results, why is that?

There's no transparency, right?

Did your lawyer tell you not to publish it? Did you— does that mean you failed the audit and you are exhibiting adverse impact?

That's okay, we found it, so we can fix it, right?

If you find it, you can fix it. That's why I'm bullish on, um, you know, this, this direction that we're going, uh, because a lot of people say, well, you know, human At least human bias doesn't scale, right? But the problem is you don't even know where it exists. Only now that we're having this conversation are people starting to think twice about like, oh, I wonder if, you know, oh, well, I always interview and I don't care about skin color and I don't care about gender and I don't care about these things. But you can say that, but unintentionally, you may be exhibiting certain bias. Maybe you're biased because this person went to, you know, a rival, you know, school from you, right? So there's certain attributes within a profile, whether that's from a resume or from a, you know, a screening or, you know, an AI-powered interview or something like that. There's certain things that, like, some of this information should be, you know, obfuscated so that it doesn't trigger human bias. You don't need to see their name or where they went to school to know what they studied and the roles that they've had. But there's bias for people, you know, just like in, in college, or for when you look at someone, yeah, recent college grad, you might say, oh, I want— let's, let's, let's do recruiting at, you know, these Ivy League schools or these specialist, you know, schools where we know we get good engineering candidates. Yeah, you're, you're overlooking all of these potential candidates with incredible potential just because they didn't go to one of those, you know, high-profile schools.

Yep.

And we see the same thing with, with companies, right? Like, where, where do you work now? Or where did you, um, where did you work before? And, and you see bias towards, you know, the Googles, the Netflix, you know, the, you know, the FANGs of the world. Oh well, yeah, we know the engineering culture there, we know they'll fit in great. Here. I mean, that's for sure, right?

Yeah, they definitely, they definitely are. And the fact that they work there means that they're really good at what they do, just automatically, right? Yeah. Um, I, I want to unpack intersectional bias a little bit. Can you, can you go dive a little bit deeper into what is— for the uneducated, what is intersectional bias and why does it matter? How is it different than what everybody knows as gender and race bias?

You mean the combination of some of those?

Is that what you mean? The combination of multiple known biases together doesn't necessarily signify intersectional bias. It's almost kind of like looking back. It's like looking backwards. For example, I'll just offer one example. There may be bias in the system against people with extended gaps of employment. In other words, if it looks like they were not employed from 2022 to 2024 or something like that, and that adversely impacts their resume score in the algorithm ranking and they get ranked lower because of that gap, the intersectional bias here is, um, and I don't know exactly what would cause that, but if you were looking at that and that affects that population, there could be bias that's being undetected because, for example, you're discriminating against caregivers. If someone was a caregiver for 2 years, they found themselves in a position where they're the only one in the family that was able to take time off of work to be able to provide care for a family member. You may be discriminating against race or gender without even knowing it. You may be discriminating against veterans without even knowing it. There might be a gap on someone's resume for 2 years because they were in the Guard and were deployed. And now you're discriminating against them because of the gap, not because they're a veteran. But the result is you discriminated against veterans. And those are the ones that we don't really know about. To me, that's the biggest danger because we can't train the AI to distinguish things that we ourselves don't, you know, when you look at that gap in the resume, a good recruiter asks the question, what is the gap about? And the answer is deployment. And the recruiter goes, great. That's, you know, I don't need to know. I don't need to ask about your your discharge status or your gender or whatever. I just know that you had to serve your country. I get it. Let's ignore that gap in employment and move on. But that recruiter looks at that and goes, this person is not qualified because they haven't been working in the last 2 years. That's—

yeah, you know, yeah, no, this is a really, really interesting, um, scenario. I do— first of all, I do empathize with these people because I've had people reach out to me in that exact situation. Like, this job market is, is brutal, and I've been out of it for 2+ years.

Yes, I'm getting kicked to the curb just because of that, right? Yeah.

Um, and then some of those other things, um, almost— it's not that they were not getting hired because of what was protected, they're not getting hired because of what isn't even protected and hard to I feel like some of the use cases— so I mentioned like this AI governance and AI assurance, you know, platforms, and there's a whole growing sort of category of software vendors that are in this space. But even some of the more advanced ones that I'm familiar with aren't necessarily diving into this unless someone explicitly signed them up to do like a special project to look at it, which is why that's pretty obvious.

If we don't know that, we don't know that we're not going to tell an auditor to look for it because we don't know. Yeah. You know, so you don't go to an auditor and say, can you please find things that I don't know anything about?

Well, it's one of these, it's one of these, um, like unknown unknowns, right? Like, because if people don't raise it, this is part of the problem with the, um, the whole, you know, bias, um, and holding, you know, some of these companies accountable, um, in, in the first place is it takes a complaint to the EEOC to even you know, get anyone to, to go and look at it, right? So, um, so in this case, I feel like it's one of these sort of special, special projects at this point where someone would have to go and say, you know, we're really curious for, um, to look at this intersectionality, and, and can we look at all the people that we've potentially rejected because of that gap which in itself is something we need to look at. But let's take a look at all the people that, um, that had that gap in employment, um, that could probably be, you know, reasonably explained. And let's look at what other, um, you know, categories were further disadvantaging, um, because of that, right? Because of my social economic status, I can't afford to put them in a home, I've got to take care of them myself, or whatever the circumstances are. Because I mean, as a Gen X guy, I mean, my parents are in their 80s. I got my daughter in high school and people are trying to care for multiple generations, not to mention their own career and livelihood. And so it's a tough juggling act. But some of these I feel like are still very much under the radar, but I mean, it's an important point to bring up. And I wonder how many people are going deep enough to discern that.

That was what I was asking you.

Yeah, well, one of the things that I think is interesting is, and concerning, is that I see a lot of organizations that are underfunding like their people analytics and talent intelligence teams. So if you don't, invest there, then you may not have the tools and the resources to even dig into some of those issues if you even if you wanted to. So that's definitely a concern. And then I'm obviously concerned if people are just writing any kind of rules into their system to automatically knock out, you know, a resume or an application because of that gap. That's obviously Um, I don't think that's intentional.

Yeah, I think when that happens, it's a systemic, um, you know, when you tell a computer to find a pattern and it will find the pattern, and it just might not be, um, it might not be a pattern you expected, right?

Yeah.

And that may be a good discu— I mean, that's how, that's how these machine learning and AI models are identifying new cures to diseases. By, by sort of extrapolating and identifying patterns that were never visible to us before as humans, because humans can— we're great pattern recognition experts in our own right because it helps us stay safe in the concrete jungle and in the regular jungle. But, um, we can't absorb the kind of information that, that a machine can absorb, you know, terabytes and, and exaflops of data. So analyzing that data, you can find these patterns that humans couldn't find or just haven't been able to. Yeah. But that's exactly right. We might find patterns that we completely didn't expect. So I guess the point is, if we train the machines to ignore those, then we're kind of asking for it at that point.

Yeah. Well, I also think about the way that some of these models are, are trained, and I think about, um, you know, if you put enough weight on these other characteristics which in them unto themselves are biased. Like let's say you put too much weight on, you know, that they went to, that they have a 4-year college degree or they went to a college that's listed on, you know, US News and World Report's top, you know, 50, you know, liberal arts colleges or whatever it is, whatever's fueling that could actually have you ignoring, you know, some of the other, you know, valuable information. That may either, you know, boost, otherwise boost the resume up in its, you know, sort of match score or is there other disqualifying, you know, attributes within that resume. I was just looking at Glen Cathy's post from this morning where somebody submitted and got an interview from a completely fake resume and it was like it had a couple impressive sounding, roles and accomplishments, but then it had a bunch of completely irrelevant and inappropriate bullet points and it just got completely overlooked. So I don't know where they got the interview, but of course, as you can imagine, everyone's going crazy like, oh, well, that's just a bad recruiter who didn't, you know, read the resume thoroughly, or that's just, you know, this or that. And it's like, well, people are mass applying with these kinds of resumes, maybe not with the exact bullet points, the inappropriate bullet points that this fake resume had on it. But still, you've— some people, I think, just don't understand how some resumes get through and some don't. You've got this right, perfect fit candidate.

Could also not be, you know, it could be timing. There's a lot of things that go into it. Could be timing. It could be it was on the top of the list. or it was literally the recruiter was watching their screen when it came in.

Yeah.

Um, I mean, there's so many things that, you know, it's not always just a robot doing it is what I'm trying to say.

Yeah. Yeah, I think, um, as, as, I mean, since we're focusing on AI, I mean, it just seems like if you're gonna, um, if you're gonna use it, make sure it's an appropriate use case and make sure you have the appropriate sort of controls and parameters, you know, baked into it. I mean, that's part of having this technology be explainable and transparent, not to mention, you know, private and secure. But if you can't explain how it works or why, you know, this got, you know, got a B-minus—

So explainability, you're talking about explainability being— I guess what you're saying is explainability is a big issue. So, um, you know, bias aside and all the other things, if you can't explain what it's doing You're already at a disadvantage because you can't figure out what's wrong. You can't fix it.

Well, you've got to explain. So if you were a company that got where someone had filed a complaint with the EEOC, for example, in the US, you'd have to go back and explain the decision. And so the first problem is if you let the AI essentially make a decision without human supervision, human supervision, human in the loop, then already you've got some explaining, explaining to do. And then, but even if it was, you know, part of your decision-making process, so in the audit world, we would say that's an automated employment decision tool, or an AEDT. And so if that's involved in helping you sort of down-select and shrink the hiring funnel to get to who's going to really interview with the hiring team, the hiring manager, um, every single one of those sort of stage gates as you move through the funnel, a decision is being made, right?

So this, even if it's a decision by a robot, right?

Anything that says that you're—

it's a gate, anything like pass or fail, right?

Exactly. Or not, that's a decision. And you let AI make that decision. So, um, so yeah, so if technically if a human is not, um, you know, looking at every resume, technically, it's because the algorithm has, has basically, you know, knocked it out somehow, or it fell below some threshold. And you decided, you know, you made a human decision to not—

which was established by somebody. So whoever established that, I guess, is responsible, right?

Yeah, but it's also hard because you've got, like, how the whole point of AI to help us here was to make us more jobs efficient, but it needs to make us more effective as well. So let's not let someone with a perfect, perfect, you know, credentials, a great resume, not, you know, they should— those are the last people that should be ghosted, right? So there's a lot of little nuances to all of these different tools and, and where they're used within the funnel. And how much we rely on them, you know, to make some of these decisions as we try to find the best candidate.

And obviously, not everybody in a position to make recommendations, make purchasing decisions, evaluate these tools, they're not all educated in what you're talking about. And so, I think that's part of your evangelism is to promote the— I love the AI or AIQ as in AI IQ. But what I was going to ask is, how do you, how do you recommend that those of us who may not be as familiar with these complex issues that we're talking about inform ourselves about it? How do we elevate our AIQ? Where do we, where do we do that?

Set up question. Yeah, so I think first of all, the organization has responsibility.

Yeah, it doesn't matter, right, right, right. Even if you bought This from a vendor, you can't just— I mean, the vendor has some responsibility, but you can't just say, hey, we didn't know we bought this software like that. Yeah, the organization has to do it. Yeah.

Yeah. So I mean, obviously through procurement process, you want to do that, that vetting, right? And then assuming the vendor is helping you with implementation, let's talk about what happens when this hits everyone's desk. Let's talk about what happens if I want to have, have your software learn from my data. Right? This is— we've had success in hiring in the past, but now we're growing faster. So, but what can we learn from what we've done before and the decisions we've made about people before and try to incorporate that? Well, so there's a— to some degree, there's a little bit of a— there's data maturity and data literacy that everyone should have. But in talent acquisition and management, you know, there's extra sensitivity above and beyond like normal, you know, AI literacy that you might get through, you know, when you're onboarded and, you know, through, you know, L&D programs and things like that. But there's an added responsibility because you're making decisions about people because there's a big— their careers, their financial risk. There's also, right, yeah, I mean, you're, you're, you're judging people. I mean, everyone in TA knows that to some extent, right? And so you've got to have extra sensitivity about the data and about what this decision actually means. These, these rows in your ATS are not just, you know, cogs in, in the wheel.

That's a good point. And another thing I would say, maybe this is what you're trying to get at, is that there's, there's even like the, um, Fair Credit Reporting Agency, FCRA, They issued a caution about AI because it might be impacting creditworthiness, even if it's just a— not just, but I mean, it's— you feel like it's a candidate and they're applying for a job. What does it have to do with credit? But still, they've issued some warnings about that because of what you're saying.

Yeah. Yeah. No, there's a lot of flaws.

So how do we elevate our IQ?

There's historical bias in— So part of AI literacy is not just learning how to prompt, not just, you know, write me a new, you know, job description for this, this rec that just got approved. Write this outreach email. I've got, I've got a, I just sourced, you know, 50 great candidates. I'm gonna now create, you know, this outreach, you know, sequenced, you know, outreach campaign to these, these candidates, can you help me write this stuff? Well, I mean, that's one use case, but you really wanna understand how, what are all the tools that are at your disposal and how much can I trust the output of each of these tools? Did I give it enough information? What are its sources? What data is it using to give me this information? So part of it is, yes, of course I need to understand how and when to use different generative AI tools, but I also need to understand how predictive AI is working behind the scenes that I don't interact directly with, which is more likely what's used to do like a match score, a stack rank of resumes and things like that. Anytime you come up with a numeric score, that's ripe for audit and it's usually predictive AI because it's giving you a confidence level that this is what you're—

Right. Sort of like the whole, you know, here's 3 different shapes, what's the next shape? What, you know, like remember those tests? I don't know if the ACT or whatever.

Sure.

So, you're talking about that. Like, here's a bunch of these, what would be the next one? And so, that's how it's doing the scoring. And so, what's the takeaway there? How do we— you're saying we need to be aware of the—

Well, Well, the key is it's not that you should be so savvy that you'd be able to say, well, why is this one— why'd this one get an 85 and this one got an 87? It's not like that. Yeah, it's more about judging. Like, let's say you, just based on a recruiter's bandwidth, you're only going to have time to You're managing, you know, 10 recs and you've got for this one rec, you've got like 500 candidates, right? I don't have time to look through 500, so I'm going to rely on this algorithm to basically show me who the top 50 are. Well, if you don't look, if you don't question, well, what about 51, 52? What about 107? Like, if I don't, if I just say Here's my threshold. This is now the universe that I'm looking at. You just discounted 51 to 500 for no other reason than it just arbitrary score, right? Yeah. Okay. Right. So that's not fair, but you also have to go against that constraint of your own bandwidth. We're trying to hire this person by the end of the month, which means I gotta back, if I back into that, I need the time to set up interviews. I need to, you know, I've got to say, well, I've got to move forward and I've got to give the hiring manager the best, you know, crop of candidates. And so that's why I'm a huge proponent of looking past the resume and getting— giving everyone, every reasonable candidate, a shot at going beyond the resume and doing some kind of— even if it's AI conversation, just have some kind of interview where you can get a little bit more color. And enhance people's applications because the resume should be just the starting point to discuss their, their candidacy, right? It's really just a marketing document. So I still think there's too much focus there. But anyway, I think that from a literacy standpoint in talent acquisition, you really need to understand like what is this vendor ultimately doing for me to make my life easier, to make my job, to make me more productive, but also make me more effective in my role? And based on the output that I'm getting from these tools, when do I need to question or go beyond the tools and do some legwork myself and do a little bit more digging? Because there might have been one, you know, attribute that threw the system off. Maybe the match score was low because there was a, you know, an error importing the resume, or maybe there was a typo.

Oh, gotcha.

Okay. Totally. So anything like that.

Totally unexpected, like you said, like it was— they just— the resume didn't parse, period. Just never, never got absorbed by the system.

Yeah, I mean, as a candidate, you don't know. Like, you submit, it says you successfully— you got an acknowledgement that you're— you have no idea. You have no idea if it made it through and it got in and someone's human eyes were—

right. You have no— it may have— you don't even know if it went through, but if it did, you don't even know if somebody looked at it or if it just kind of got fell off the assembly line because of some, you know, sorry, we couldn't scan this resume, we don't know what it is, we're just gonna assume it's no good.

Yeah, I, yeah, you have no idea. Okay, you can test it. There might be some ways to test it, right? Give it to a resume evaluation tool and see if it tells you that something's off, or, you know, maybe might want to use different formatting, or you might want to, you know, do this. I mean, I've had issues where because I worked— I had so many different jobs at IBM, I wasn't even sure how to list. First of all, do I list them all? Because now we're going back to 20th century. Um, but also, um, do I actually list IBM 10 times? Yeah, do I list IBM 10 times under that, or do I have a heading that says IBM and then list the roles? Because all these, these normal change— like, a human reading that would read it without issue, but maybe some of the ATSs don't like the way that that's formatted. I have no idea.

Because it was— it looks like 20 jobs, so you're a job hopper when it's really 20 promotions. Yeah, whatever. Yeah, exactly. Yeah, I'm not saying that that is the case. I'm just saying what if. Yeah, okay, right.

Yeah, so, uh, so overall I just think, you know, so literacy is part of, you know, can I trust these systems to evaluate these candidates properly? Is it making me better at my job and more efficient at my job? But then also, as you use it in your day-to-day, I mean, there's plenty of other processes and workflows and things that people are using either an AI assistant for, maybe, you know, the evolution of a conversational AI. You know, I give you a question or a request, you give something back. So it's what we used to call an IBM Watson, like question answering, like just—

Or what I used to call chatbot.

Yeah, basically. Yeah. And a lot of that was rules-based. It was within domains. Right. Yeah. Or sometimes it would say, you have to pick one of these 5 things. Those are the 5 things I know about. Right. And eventually we move past that and we're starting to see quite a lot of that where it's more general purpose. You know, can access the internet, can find you all kinds of things. But there's, so there's the evolution of that. And then there's these agents, which are actually, they literally have some type of agency or autonomy to go.

Right, so yeah, let's clarify what an agent means is that it could, what an agent means or agentic AI means that it has agency, as in it has the ability to go out act independently or autonomously. That's what distinguishes them versus generative AI, which is answering a question. That's the difference. Just for the folks that are listening that might not be familiar with that.

Yeah. I mean, I think so you've got the predictive AI, which is basically like you said, trying to identify, doing pattern recognition, pattern detection, and then predicting the next thing. So there's a component of that even within generative AI, right? Because We said it's predicting, it's constantly predicting the next word to spit out. But within generative, yeah, you've got the evolution of chatbots and conversational AI, which if you're using ChatGPT or Claude, that's what that is. And then agents are, yeah, you're basically making a request like, you know, you're sending it out to go, you know, go on Kayak, check for the latest flight if there's a flight. If there's a reasonable flight under $500 on Thursday, and then, you know, also look for hotel, like, so it's basically, it's doing actual tasks.

Browsing your website, doing a search, collecting information, measuring it, and then sort of, I don't know, bringing it back somehow, displaying it.

It's like really advanced automation that can also go in and out of different systems. So you give it credentials, either yours or you, create a profile for it as a, as a sort of digital worker. And, and it can basically go out and access those systems, go from system to system. That's when, like you were talking about, agentic workflow, you've got agents actually talking to each other and then, you know, bringing you information. So let's say you had those top 50 candidates and all you had so far were the resumes. Well, now you need to gather a bunch of information. Maybe you're going to go out and look at their socials. What do they talk about on LinkedIn? What kind of conversations do they engage in? What's their, do they seem to be an authority on these particular topics? Because we really need someone who could lead a team. How is their, you know, what can you decipher from the rest of their online, you know, presence? So maybe you have, you know, some agents going out and looking for that information, or you're internal and you're trying to gather information about, you know, the workforce plans of different groups. If that— if you're privy to that information, you could go out and see, um, and have the agents, you know, sort of pull back all this relevant information. And now all of a sudden you've got this, this dashboard that agents have created for you with all this new—

and only what's on the dashboard is visible to you, and you don't know what's not there.

Yeah, I mean, a lot of the things that we've seen over the last decade where people, you know, you may have a team like a, like a data team that's getting requests all the time. Oh, can you build me a new dashboard? Can you build me a dashboard for this? It's, it's, it's very rigid the way that's done. And in the future, actually, probably now, there's agents that can basically do a lot of that for you. And to your point, Shelly, like, completely customize that dashboard. So give me I want this type of information, I want these types of data visualizations, and then tell me the, you know, based on all of that information, give me your top, you know, sort of insights based on that. So we see AI used in that way today, both inside a company and if you're doing market research, that's one sort of output of using AI for market research is it's aggregating all this disparate information and making sense of it and then presenting that to you so that you can make the most informed decision that you can make. And so I think even in talent acquisition, I think that's what we're moving towards as people try to up their game, as people try to become more talent advisors and take some of the, the rote, you know, routine tasks, you know, off, off their plates so they can focus more on relationships with the candidate, relationships with the hiring manager. Understanding what the needs are of the business units that you're supporting. Yeah, I got you.

Yeah, makes sense.

So did you— oh, and this is— go ahead. I was just going to say this, it's evolving rapidly. I mean, I know people might be overwhelmed by just what I just described, but it's important to just continuously learn about some of that.

Was actually my next question. Is—

yeah, I'm not expecting people to know like, oh well, I used, um, you know, ChatGPT 4.5 and it was better than Gemini 2.1, and like, if there's— there's plenty of people out there doing that level of experimentation. And I would just say start with just, you know, figuring out who's using some of the ones that are best for the tasks that you need to get done. Just pick one or two tools and try to stick with those. You don't have to experiment with all of them, just leverage the crowd and, um, and, uh, just kind of gauge which are the ones that might work best for you. I mean, some of them aren't necessarily text-based, right? There's a lot of AI tools in the creative spaces too with, um, images and, and, you know, videos. So it really just depends on what what your focus is. But, um, but you've got to have that learning mindset so that you're curious, um, about how these things work and you can distill, you know, what can I take at face value and what might I want to go get a second source of, um, information about so I can know whether I can trust this, this AI, because it will Like we said at the beginning, like this, it doesn't know necessarily like right from wrong. It's just gathering information and sometimes it leaps to conclusions. Right.

That's the problem exactly is it's a robot and it's gonna do exactly what you tell it to do as a robot. Yeah. The reason I jumped on that when you said that is 'cause I was gonna ask you, is there a place that you go to or recommend that people go to to stay informed on some of these bigger, you know, I know you can't, you can't learn about all of it, but is there a place or a couple places that we can go to, to get a gist of what's going on in this space from a layperson's kind of perspective?

Yeah. I mean, I have gone overboard in terms of the newsletters that I follow and the blogs that I read. Substack newsletters that are giving me information. So I don't have a particular like educational source site, and we can certainly add a couple. I don't know if you can add like some, some links in the show notes or whatever after this, but, but there's a couple newsletters of people that I trust that are— I like the way they, they write and they helped me just sort of separate the wheat from the chaff, as it were. And so there's a couple, this guy Don Van Rossum has a great site, I think it's called Lead with AI. He's got a site, he's got a bootcamp. Allie Miller, if people don't know Allie, Allie K. Miller, she's got an incredible newsletter and she talks all the time about the way her team is using AI constantly.

I've asked her to be on the show. Hopefully she'll accept.

Okay, okay, cool. Yeah, she's great. There's a couple folks I know that do this AI Build Lab, which they have an amazing, you know, boot camp where you can learn, you know, the difference between an assistant and an agent and build your own agent. So even if it's just like a using AI as a thought partner and a second brain. Like, we can't— we forget stuff all the time. We forget, you know, the people we meet, or, you know, the, you know, maybe a summary of a webinar or something like that. And so being able to have that sort of your own personal AI sort of copilot is a pretty powerful solution that people could learn to build themselves. And I'm, you know, I'm still learning a lot myself. I mean, I got to jump back in that boot camp and I try to help automate my own workflows, some of it for like sales and social media posting and outreach, some of it for my podcast production, right? I'm constantly going in and out of right now like 6 different, you know, tools, some that have AI bolted on, some that are AI native, and others that are more sort of just old school tech. But, but how do you string those all together so that I'm not constantly tasked switching and copying and pasting and things like that. So that's just some, some basic, you know, automation stuff. It's not even like full-blown, right?

That's the, that's the Copilot or the, that's the companion app that is there as an assistant to like, hey, sort this, rank this, spell check this, rewrite this, those kinds of things. Yeah, you're talking about, um, you know, actually helping you with automating your workflows, uh, as opposed to just sort of task-driven stuff.

Yeah, like I'll have, like I'll do, if I do a prep call for, with my guests, I have a note taker, an AI note taker that takes notes, gives me summaries, and then I can take that, put it into, I need to send them not a script. My show's unscripted just like yours. It's more, here's a framework for how I think the discussion would logically flow. Let's try to hit on a couple of these points. Let's make sure we mention your latest book or your latest podcast or whatever it is that you want to promote. But then maybe I take that and I could have something going behind the scenes, like what else, what have they been doing over the last month or so? Maybe there's something there that I can that I can bring up, um, how to make my show notes not sound like they were AI-generated completely. That's always a little tricky. Um, so yeah, I mean, everybody has these balance points where like you want AI to help, help you, but you don't want it to do all the things for you. Like, I want to— one of my goals is to blog a lot more. I just launched my Substack, so I'd like to get out and blog more. Well, if I'm going to do that, I'm not going to off— I'm not going to outsource that work. Not completely. I want it to help give me ideas. I want it to help refine those ideas, but— and maybe put them in context, right? Maybe I want to talk about a particular— maybe I want to talk about human-centric AI, which is what I was talking to people at Transform about. What exactly does that mean? How is it different than responsible AI? But I can take— I can say to one of my AI tools. I want to talk about human-centric.

Give me some ideas.

Also, go, go me some ideas, but also go back into my all the transcripts which you have stored from all of my previous podcasts and give me a couple guests that I can mention, or, you know, sound bites that I can mention and allude to in that blog post so that I can sort sort of cross-pollinate, you know, podcast audience with, you know, blog audience or whatever. So there's all these different ways in which you can use it to better, to make yourself more impactful and effective without actually outsourcing the, you know, production of— I still want it to be my words, right?

It's like my voice, right?

Write a book someday. I want it to be my— it should be my thoughts and my words and just use the AI to help me you know, just, just clean it up. How are we using AI to, to be better versions of ourselves is one way to think.

Okay, right, to help us do what we do, help us be us, not, not replace us, or, you know.

Okay, um, yeah, that's, that's the basis from what I— what we have been calling human-centric AI. So I'm working with a bunch of other folks in on that area, but that's, that's what we mean, and that's how I differentiate from responsible AI, which is the much more structured, ethical pieces of it. Yeah, it's definitely a lot of overlap, right?

Right, right. Yeah, it should be— it should all be ethical. It's not like you're like, you have, uh, human-centric AI is unethical. That's not how it goes, right? They're both— they both should be ethical. But yeah, one is more the structure of how do you make it, um, explainable, and you know, who's responsible, and what are the ways to opt out, what are the ways that it's be being audited versus the other one is more like, how do we keep the human involved without causing more work for the human, which the point is to reduce work. So yeah, I get it.

I get it. Yeah.

We're getting close to the end of the time here. So I wanted to give you an opportunity to answer the two questions that I always ask everybody on this show. And the first one of those is, who should be our next guest?

Well, if you can get Ali, okay, um, she would be fantastic. Um, the, the guy that I had, um, on my show who was one of the co-leaders of this AI Build Lab, um, this guy named Tyler Fisk, he's, he's just amazing. So I think he'd be a Tyler guest to have.

Tyler, if I ask— okay, all right, I'm gonna invite him.

Yeah, okay, cool. Yeah, um, he's awesome. And his, his partner in crime, uh, Sarah Davidson, is amazing as well. So I think either of them would be great. Um, one of the things I liked about them is it's not just their style and making sure everyone, you know, understood things as we went stage by stage in building these these agents, it's the community that they created. And so I just think there's a lot they could talk about in that regard as well, because I know I'm a member of a lot of communities focused on recruiting and talent acquisition as well. And I've just, over the last 2 years or so, I've seen the incredible power of having a community of like-minded folks who are on this journey with you. I don't want anyone No matter what your role is, I don't want anyone to think that they're on this journey alone. Everyone is trying to figure out how to get comfortable with it. Where do I use it? Where could I use it? But where should I use it?

So you said Tyler Fisk from AI Build Lab, that one.

Okay. Yeah, Build Lab. Yeah, that's great.

Cool.

Yeah.

All right. And then as we close out, the last question, I usually ask for something that you want our audience to think deeply about over the weekend. So, not a statement from you for them to consider or contemplate, but a question for them to ask themselves. So, what would you ask our audience to ask themselves to think deeply about?

I would think deeply about the kind of work that you really want to do and play to your, play to your strengths and figure out how AI can help you achieve that.

Okay.

Right. Don't look at AI as a, as a threat. Look at it as an opportunity, right?

So ask yourself, what's the kind of work you want to do and how can AI get you there? Got it.

Okay.

Yeah, that makes sense.

Yeah, because it is, it is there. Um, I know there's a lot of, you know, doom and gloom, it's taking jobs away, but there's going to be a lot of other jobs and we all have any talents and interests and, um, transferable skills. And so what, so what could you be, what else could you be doing?

And how do you prepare for that? I like that.

That's, yeah, we've got to be adaptable. So I just don't, I don't want people to think, oh, this is coming, this is going to take away, you know, X percent of what I do. My, your job may not go away, maybe it just gets, you know, redefined. Or maybe, you know, if you speak up and you're clear about what your goals are, then you can be, you know, reskilled into another role. But people are making transitions all the time, and it's normal for career paths to be nonlinear these days. It's almost an expectation. So don't feel like just because you can't continue on— it's not your job title now, it's the end— that you're screwed. Yeah, no, you're not screwed at all. You've got great— you've got talent, you've got skills, and there's a place for you if it's not where you thought it was.

All right, so yeah, I like that. So basically, ask yourself what is the kind of work you want to be doing, and how— and once you figure that out, how can AI help you get to that, um, so you can, you know, switch roles or, or do something different. Thanks a lot, I appreciate that. Thanks for joining me on the show today.

Um, yeah, absolutely. Thanks.

Absolutely love to have you back in the future again if you're, if you're not too scarred from this one. And folks, folks, we're going to be having actually a funny man. Joel Zeff is our guest next week and he's an improv comedian turned consultant in the HR space. And he's going to talk about ta-da moments at work. So, kind of a departure from the typical stuff. But he was a referral from someone else that said that we should have him on the show. And I went to his website and looked him up. You can, you can search him if you want. Joel Zeff, Z-E-F-F. Really looking forward to having a few good laughs and learning some things about how to make everyday moments more meaningful and lighthearted at work. So, that's our topic for next week. Hope to see you back here. Until then, be