Adam Treitler Live on Shally's Alley (Part Two!)

Hey everybody, how are you doing? It is Friday, March 8th, and you are live on Shelly's Alley. That's right, you are live on Shelly's Alley. Doesn't matter what the weather is out there, in here it's always sunny. So, well, uh, first thing I wanted to do is— some of you might have already figured this out from the area on the screen over here. Um, I finally was able to get a deal with someone to sponsor the podcast, which is really good news because up until now, basically, this has been completely gratis. I've been spending my own money on applications like this Restream thing that we're in here, and it costs a significant amount of money. So Between that and, you know, being able to do a little bit more with the distribution, like getting, getting the podcast listed on, um, what's that called, uh, the Apple iTunes or whatever. So there's things that I just can't do as a solo operator. So Jim's generosity is enabling us to be able to do some of that stuff, to be able to get fancier with our production here in the in the wonderful studio in Peachtree Corners, Georgia. So thank you very much, Gem, for sponsoring the podcast and being a good partner. Today you might notice I'm wearing my very special work shirt. See that? How do you like that? It's a Dickie. How do you like my Dickie, Adam? You know, those like—

It looks great. Those mechanic years of training when I was like 5 of reading things backwards has equipped me for moments for that.

Yeah, yeah, yeah. It's actually a legit Dickey. It's like one of those like brands, you know, all mechanics, uh, wear them, and, uh, it's actually really comfortable. So I found this in my closet when I was going through to identify clothes that I might be able to donate. I'm like, all right, let's get rid of some of these things that have just been sitting there. And I found this, I'm like, oh, I forgot I had that. So um, my new favorite shirt. Anyways, so everybody, welcome Adam Traylor. He is a repeat offender. That's very rare here on Shelley's Alley. Is someone, you know, someone that is actually willing to come back a second time, uh, is, uh, is extremely rare. So thanks for joining us again, Adam. You might recall Adam is a former Viacom, CBS, Paramount, Twitter, um, and also a published poet. We spent a little bit of time talking about that last time. Now he works as a recruiting advisor— that literally is the name of the company— where he is a consultant who does digital transformation consulting. You'll have to ask him exactly what that means. I still haven't figured it out. But basically, he talks a lot about design principles. And last time we got into— just as a quick recap here, Adam— we got into the basic design principles for Well, anything. So the basic design principles specifically related to recruiting, and they are the conceptual heuristics. So that's the purpose, the foundation, the why. What do people understand? Who's the intended user? So that's kind of the first one. Like, what are you designing this for? Who are you designing this for? And why are you designing this? The second component is the linguistics. So language, making sure that the language is a language the user recognizes. And I don't, I don't mean like English versus Spanish. I mean that you're using the vernacular vocabulary lexicon, whatever you want to call it, that is familiar to that user, not necessarily to you as the person creating the dashboard or whatever you're creating. So conceptual heuristics, linguistics, physical heuristics, layout, right? Where do the buttons go? Where do the sections go? Where should they go? There's some basic design principles on where people tend to to look for things. When they wanna search, they tend to look for the search bar in the top right corner or whatever, right? So using those common human desires for physical layout. And the fourth one is the navigational heuristics. And that's essentially, how do I get there from here? Where do I go? And so a lot of recruiters and just in general TA leaders and sourcers, A lot of people in TA will, when asked about what's your favorite ATS or your favorite CRM or your favorite whatever, will often just shake their head and complain mostly about navigational heuristics. It's one of those things like, oh my God, there's so many clicks to get to do anything. It's one of the biggest complaints. So not only do you have to understand when you're designing whatever it is, a dashboard, anything, you have to understand who your audience is, how do they speak, and where everything should be located, but you also have to make sure that you can get to it in a few clicks because otherwise stuff gets, uh, well, you know, it gets frustrating. So that's a quick recap. Did I miss anything, Adam? So, you know, jump in here.

No, I think, I think at a really high level you hit the nail on the head. Um, I also have an article in SourceCon that kind of covered this topic as well and kind of framed it up too in the context of, right, you're a sourcer or you're a recruiter and You heard all these heuristics, but what do I do with this knowledge? I think one of the things that I started sort of dove a little bit deeper into there is, right, you can do it in terms of designing the experiences. We talked a little bit about this, whether that's literally your email templates, making sure you're not using the vernacular or the verbiage that's exclusive to your organization in a way where you're sliding into these people's DMs and they have no idea what FMLA means or FBLA or whatever these different acronyms within BALA. Like, you might not know what these certain acronyms mean 'cause they're specific to your corporation.

Right, so whatever it is, messaging, dashboards, just understand if the audience is going to be reading this is all about FMLA, then use FMLA. But if the audience doesn't know what that is, generally speaking— but Adam, I have a question. How do you know what the audience knows?

Sure, I mean, for one, you listen, and for another, you know, you talk to them. I think it's— I think there's this challenge of I know, I know. You've got to talk to— can you believe you have to talk to people on a podcast? No, this— this— you need to, to understand your audience, you need to listen. And I think it's, it's challenging. Listening is a really hard freaking skill. I was not great at listening for the longest time.

Um, and you know what, that's a good point. It— and hearing isn't the same as listening, correct?

Yeah. And I think framing things, uh, is, is also really, really important. Being willing to ask questions, which Oftentimes we think has to come top-down, and if you're an organization that really stymies that, then maybe reassess your situation. But I do think that you don't always get that as much as I'd like to see in the talent acquisition space. It took a couple of years when I was in the Viacom space to get recruiters, when I was more on the operations side, to get recruiters comfortable asking questions, like to say, you know, I don't like this about the ATS, right? They're afraid to do that. They're just like, they're over there trying to figure out their own workarounds. Recruiters and, you know, sourcers, we talk about them finding purple squirrels. They're also very squirrely in their behavior sometimes, right? They find ways to get without, you know, following the proper procedures.

Well, I mean, can you blame us? We do get thrown a lot of roadblocks. So, you know, we have to find ways to get the results despite the roadblocks sometimes, you know, even, let's just say, I don't want to say cut corners because that's not the intention. The intention is not to cut corners. The intention is to get outcomes despite the roadblocks.

And let's— and exactly, and let's reframe that. So like reframing this in the context of listening, right? I don't think a lot of organizations listen to this. They approach it the way I kind of just broad stroked it and they're like, we need to force rails to make the recruiters follow specific behaviors and patterns to get specific results instead of just being like, hey recruiter, why are you not doing it this way? Right? Like, why are you doing this workaround? Oh, because it's really hard for you to get the information or because the system's really clunky and you just wanna get a message out to the candidate. That's why you're going straight to LinkedIn and not sending every message through the ATS. Oh, you know, um, there's a lot where we, we don't listen in a lot of these spaces and when we don't listen, we don't inform designs well because, you know, conceptually we need to understand the objective and we need to understand all of the personas. That's something that—

why are you doing that? Well, I'm doing it cuz it's faster and yeah. But then you ask them, why are you doing that? And then they try to tell you that it's better or whatever, and they don't give you the right answer. They give you the answer that you think you want to hear. They think— sorry, they give you the answer that they think you want to hear. This happened to me as a leader as I was building out systems where they just kind of tell you, well, it's just I get better results when I do that. And I would push back and be like, okay, like, for example, what kind of results? Can you show me the results that you were getting before you were doing that? And how did they improve? And sometimes Sometimes I'm lying. Often, not sometimes, often they couldn't. They just wanted to do it that way because that's like the way they knew it. And so it kind of goes both ways, right? I was willing to listen, but they weren't giving me the truth. And I think they weren't giving me the truth because in the past others weren't willing to listen. So there's a little bit of that going on.

And I think this is where one of the benefits of, you know, listening is coming in informed from multiple sides when you can, which is hard. You know, you don't always have the benefit of that.

There's that empathy thing again, yeah.

But like, you know, being able to listen to multiple voices so you have those different perspectives is super helpful. A client call I had literally this morning, we were trying to figure out an integration for kicking off jobs between systems. And the person I was talking to was highly technical and they were focused strictly on how data flows from one system to another. And I had him step back for a second Basically what he wanted was this full loop of a job gets kicked off and then a bunch of stuff happens in the job. We eventually hire someone into the job. For the context of this system, they wanted information to flow back into the original system. They're using a position management system, so they can add a plan. They also have an HCM and they also have a Workday. He was like, I want to get this hiring data back into the position management system. Well, we already have something built that's sending this data to your HCM, right? We're getting this information, and I'm getting a little technical here. We're already getting this information to flow. And he was so obsessed with how do we get this information to go back to the system. I said, why do you need it there?

Yeah, why do you need it there?

And he looked at me, he said, I don't know, I was tasked with getting this information into the system. And I was like, well, you don't know why it needs to be there.

Somebody told him to do it. Someone who told him, somebody above him in the organization said, you gotta do this.

So we need this. And you know, then he was like, well, it's cuz we did it in the old system. And I was like, but why? And, and then he went back to his team and, you know, he went back to some of the team members and they were like, because we need it in the system.

We never used it.

So exactly.

Yeah.

Upon some further investigation, it was more or less for future workforce planning. It was for infor— they wanted information. We just hired Shelly into this job that we posted. We want that in our workforce planning system so we understand what rate did we hire Shelly at, et cetera, et cetera. But for the context of how they're using that, they don't need that the day Shelly starts, right? They, they don't. So like the way they were trying to build this integration was like in real time. It had to be real time. I just hired Shelly.

He could have literally reported on it after the fact. That makes sense. Exactly. So how the client plans to use the data. So listen, two things. One, number one, this is how we end up with situation that happened with me where the client— I might have told you this before— the client had a job code, a position code, yes, a requisition code, an opening ID, and a position ID. 5 numbers originating from 3 different departments. TA, HRIS, and finance. 3 departments generating 5 codes for the same workflow, the same, the same outcome of hiring Adam into this job. 5 different codes, and they didn't even make any sense. 3 of those codes were fairly arbitrary, and One of them had, uh, finance created these like abbreviations. Like marketing would be MKT or whatever, right? Like that, within numbers next to that. But all you could tell from the pattern was it was MKT was marketing. Then the next 4 digits were kind of not really following any kind of pattern. There wasn't a serial, there wasn't like the first number. You know how in your Social Security number the first number determines like your main geography? Or in your phone number, the first 3 are your area code or in your zip code, the first number is, right? In these, the first of the 4 numbers did not dictate whether it was a department, right? It could have been a 3 or a 7, sometimes, you know, same. And so even the ones that had a little bit of structure didn't have enough structure to actually be able to use any kind of determination. So that's how we end up with that kind of thing, and I was going to ask you a question about that, and then I had a follow-up on a different thing that ends up there. So I'm going to go back to that. So here's my follow-up on that. Say you have a system with 5 different codes. Can you use generative AI large language models to translate that? You could.

Yeah, I think you could.

You could.

You could. How?

You could use generative AI. Explain it to me like I'm an 8-year-old. Sure.

First of all, you could drop it into ChatGPT and ask it to explain it to you like an 8-year-old.

No, I'm talking about— No, I know.

But procedurally, that's what I'm getting at is procedurally you can, sometimes when you don't know what to do with something, you can literally use an LLM to tease through an approach to a problem, which I think a lot of people forget. I've talked a little bit ad nauseam recently about how you can sometimes use AI to figure out, you can have these conversations to reframe ideas and think about how to solve this.

I got you. But Literally, let's get in the weeds. Okay.

Yes.

I have a column of numbers that are the position ID with a description of what they are. I have a column of numbers that are the job code with a description of what they are. I have a column of numbers that are the requisition code or the description of what they are. So what do I do? I take those, put them all in a spreadsheet and upload the whole spreadsheet and say, make some sense out of this.

In theory, you could. I mean, Ideal state, even in the context of this, you have some context and you have some understanding.

Some context, right? The numbers aren't related, but the descriptions are related, right?

Exactly.

It says software engineer 2 over here, and over here it says software engineer 2, so it's matchable.

What you can do, again, there's two things. One, really viewing GenAI again as a copilot, and I've done this before, is if you don't know how to explicitly test for duplicative indicators in the space, or you're looking for overlap where we don't need these two codes because they're always the same or something like that. You could ask ChatGPT to help you build something in Google Sheets or Excel.

Like a translator. Exactly.

You could literally have that help you develop the code to do that separately. Alternatively, and I don't think it's really going to work as well because, for example, if you just are using ChatGPT, even if you're paying for it, if you drop more than 100 rows of a table into a conversation. Yeah, let's say you're actually paying.

Or let's say you actually have your own large language model because you don't want to upload your position codes. Hold on a second. You don't want to upload your position codes to ChatGPT and all of a sudden tell the entire universe what your finance codes are.

Yes. If you had a specially trained LLM in particular.

That is your proprietary.

Exactly. You could angle the compute expense to process this information. You could 100% give a table of information to an LLM.

It would help you design a solution.

You could simply ask it, for example, you could have it help you design a solution, but you could also ask it in that context if it has that processing power and it's able to analyze all that information, you could ask it for where are there duplicates? What is the distinct value of these different IDs? You could literally again use natural language. How do they overlap?

Do they overlap?

Inquire about that. Tease that apart. And again, I've done that in small samples with anonymized data with public LLMs. And there's also an argument to be made that in the future, LLMs may not be the solution for things like this. LLMs are just the tool with the most investment right now.

That's a step in the right because LLMs are processing. That's true. I have been thinking about that. LLMs are processing language and language in the sense of writing. Phrases, sentences, and what we're talking about here has phrases in it, but really it's about patterns and k points of n. It's a little bit more the shortest distance between two points. It's a little bit more conceptual, it's a little bit more mathematical than prose. I was thinking about that.

This is why it's funny because I feel like there's a lot of phrases in the industry of AI that are being repurposed and used for multiple meanings. Along the lines of this notion of multimodal. But there are layered applications where you use an LLM to interpret a request. I say I want a detailed analysis of the labor market from 2019 or whatever. I should probably be a little bit more contextual and precise in my request, but I want that. You have a natural language processing LLM, then run that through a system that maybe uses some other Bayesian method or uses another form of like K-means clustering, whatever, and is doing the mathematical analysis, is crunching the numbers because the LLM has translated your request into context that can understand, then that spits out a pure mathematical result that another LLM on the backend is able to translate into natural language and give you an English-based reply.

Those are some of the examples of some of the ideas I've heard out there that people are Ironically, this whole technology, one of the reasons, I'm not sure how heavy of a reason, but one of the reasons I understand that this technology of generative AI is possible today is because of the advances made in GPUs. GPUs are graphics processing units that are chips that are specialized compared to a CPU, on your computer, that the main chip that your computer uses, which is a CPU, is used to processing all kinds of information. The GPU is dedicated to particularly processing the numerical information behind graphics. It's specialized and that is what enabled it to be able to be used for large language models because as a specialization, it doesn't waste time on trying to figure out what it's calculating. It can go right directly to it. So these math processors are actually what's driving the language models. So, you know, they're being used for something they weren't really intended for.

Yeah, I mean, I mean, it's, it's always, it's always hard to tell, right? It's very, it's very hard to tell when it comes to things like Amazon and Jeff Bezos in the early days of like, oh, he's just selling like books online or whatever, and then suddenly it's, you see what Amazon is today. It's hard to say for sure that like Jensen Huang and the Nvidias of the world truly knew they were going to go from developing these GPUs used largely for video games, right? Like people who wanted high-end PCs to play really, really high-quality graphic video games on their—

that's how they started, right?

Computers. Yeah. And now, you know, the processing rate has gone from $150 a share or whatever a couple years ago to now it's almost $1,000 a share. Might hit that while we're talking. But yeah, it is pretty crazy. And you also have these, you have NPUs and TPUs now, right? You have tensor processing units, which is something that came out of Google. You have NPUs, which are neural processing units, which they're trying to use like more locally on mobile devices and things like that so that you can—

And LPUs, language processing units, that's new, came out of a company in Florida that is using a, creating a processor specifically like a GPU, but to process language. And so yeah, so this comes back to the whole, you know, we're using large language models, but the developments are, accelerating to the point where we're probably going to be using more appropriate language chips for language models, number chips for number models.

As crazy as it sounds, what I think people don't realize is we're in the early days of those chat rooms. That's where we are with AI. It doesn't feel like it, but literally with the kind of AI models that we're using right now, this is literally like the early day internet chat room. Version of this. Like, we haven't even gotten to email yet. We haven't even gotten to AIM. We haven't started AIM. We're going to kill AIM eventually, but we haven't even gotten to AIM yet.

You want to hear something that's going to blow your mind? You may already know this, so pretend that it's going to blow your mind. Um, the large language models that we're using now were created in 1966.

Yes. Yep, yep. That was, uh, that's something that I think, uh, people don't realize is, is We haven't covered the exact quote, but there's, there's this whole notion of like most innovation is from years and years and years ago. Like in the context of these, these are algorithms from the '60s. Some of them even predate the '60s. Some of these, some of the, some of the algorithms that informed those algorithmic developments could date back to the 1800s.

So what's going to happen is now we're going to, the reason that we're accelerating is because we're able to apply the technology that got us here to get us to the next step. So it's like an, you know, a logarithmic growth or. Evolutionary growth. But, um, so I want to go back to, um, something you said last time, which is to solve problems upstream, not downstream, and combine that with something you said today, which is about the complexities that are created when people just do what they think they're supposed to do instead of asking, what do you really need it for? What are you going to use it for? Right? Combining those two things, and I want to talk about bottlenecks. I want to talk about optimization of workflows, particularly recruitment processes, right? And how it doesn't really matter how much advancement we make in AI, it's going to be nearly impossible to optimize the workflows of recruiting due to the bottlenecks caused by compliance. I want to talk about that because in the connection to problems upstream, not downstream, is this is an upstream problem. Your organization may want to do things completely differently, use revolutionary AI, but you will be handcuffed, uh, by compliance issues. And these compliance issues were created because they were solving very real problems. I'm not dismissing the problems of, uh, inequities and discrimination and a lot of other things that led to these compliance issues. But the reality is, we have 7 different governing bodies that oversee recruiting; 13 that I can think of; 13 distinct laws. One of those bodies is capable of sending you to prison. OK, so we have a lot of compliance issues. Most recruiters have no idea what I'm talking about. But it doesn't matter if you know or not; this definitely affects you. So these compliance rules are why ATSs sometimes do what they do. Actually, no, these compliance rules are entirely why ATSs do what they do. In fact, I can even— sorry, let me step back on that. I think ATSs are essentially nothing but risk mitigation systems to deal with these compliance issues. But aside from the point, the problem is that these workflows hit bottlenecks and roadblocks because of compliance, and you can't AI that. So let's back that up. What's, what's your answer to that?

Yeah, I mean, I think, I think the notion of you can't AI that is a fun concept, um, that we can interrogate later.

Are you gonna, are you gonna AI compliance? I mean, like, you know, exactly.

No, that's because you, you end up, it's almost like Pandora's box of if you use AI, it's like legislation, it's like, but you're now, you know, you're now pushing this through.

Dictating a law around taste, around, you know, vanilla is the best flavor. Like, okay, you can't pass a law that says vanilla is the best flavor. It's not going to work.

Organizations are definitely kind of between a rock and a hard place, I think, currently, because regulation is emerging. To your point, there is existing regulation, but the way organizations pre-AI interpreted these things, every organization had its own risk approach, right? I've seen companies that obviously completely ignore certain, you know, requirements around the definition of internet applicants and things like that. They'll move people all the way through the process without having them answer the same questions that other people had answered, which is technically, you know, a violation of the law. And then you have companies that overly enforce that, right?

Like, right, here's the compliance, and they go away. Oh yeah, right.

I've seen— we've seen, you know, from my recruiting advisors days and even before that, I've seen people who— companies that have these ridiculously long application questionnaires because they ask you questions that they just out of an abundance of caution are asking.

Right.

And yeah, they, they probably needed where AI could come into play in here is helping you process all of the questions and information in your systems potentially if you, you know, assuming you have something proprietary. Because I think running those questions, so long as those questions don't have sensitive information, it's and then through an algorithm is probably not going to get you in any trouble, right? Like, just putting— dropping all of your compliance questions into an algorithm and being like, you know, are there redundancies in here? Are there things that we're asking that, you know, like, how do these relate to the law, right? You can kind of get information from that which can help better equip you. Because in all transparency, from my experience, oftentimes the employment lawyers who support these spaces— highly knowledgeable, highly capable, stretched. They're working on a number of different challenges. They are not just focused on TA. They're focused on a number of compliance verticals within the organization. And the more information you can bring to them upfront and the more informed you are, the more likely you'll get things through and you get over those bottlenecks. Because my experience, for example, at Paramount was constantly, if we came to legal and we're like, can we do this? And we didn't have an explanation on like why we didn't—

Legal's always gonna say no.

Exactly. And we didn't come with an explanation of like, we understand the law, we understand the context of what we're asking. Like in this case, exactly, they would say no.

But if we said, hey, because we wrote this, you want to eliminate risk, so let's not do it because it's risky.

We wrote this policy language, right? We have this here, we have this disclaimer, we're going to ask people for this in a specific way, we're going to do it at this point in time. And, you know, based on all of our understanding of the law, this seems kosher. What do you recommend? What's your feedback? So now we're not even asking in a framing and like a can we do it or not, we're asking like How would you rephrase this policy language? How would you adjust this experience to ensure that it's compliant? From my experience doing that, I've never had them say no. They've come back with critiques, they'll come back with changes, and sometimes those changes completely change the design, but sometimes it works. Sometimes you get to this point where they're like, you could do that. Sure, you can adjust your application process to rephrase that question that way and now reduce the logic on the backend of the system so you don't have to have 70 different questions that ask the same thing depending on from, you know, the user's location or something like that. So that was like one thing we did was we rephrased a question around work eligibility so we didn't have to have this crazy backend logic where we kind of just asked whether or not you had work authorization in the country that the job is based in instead of having this logic that was like, are you authorized to work in Japan? Are you authorized to work in whatever? And we had a separate kind of disclaimer that explained what those authorizations were in those particular countries. So now we had that covered in terms of the context of covering all these scenarios, but we kept the question simpler. And most people are going to know, they're not going to overthink that. Like, okay, the job is in X country, can I work there? It's kind of the only context that you need so long as the country is kind of prominently displayed in the application. So like there's things like that where had we just asked, can we rephrase this question and remove US from the work authorization question? They would have just been like, what the hell are you talking about?

No.

Uh, but when you framed it in sort of a larger context and you would already had thought about the risk mitigation and the spaces where you're going to mitigate that risk, Legal's going to be like, damn, okay.

But so I get what you're saying, but that doesn't— so for example, um, the Uniform Guidelines for Employee Selection Procedures, UGESP, and then you've got EEOC, then you've got the Department of Labor, you've got the Department of Justice, and the Office of Personnel Management, OPM, which is the OFCCP. Right? They have different— oh, sorry, I forgot about Sarbanes-Oxley. Right, which is another one. And Americans with Disabilities Act. Gosh, so there's 7 of them. I'm not— I can't even recall all 7 of them. So there's all these different things that sometimes, I mean, the machine's not necessarily going to be able to put all that together and figure out a way to around it. So we have to essentially— it's like these gates that we have to put in the system because it's required of us and it's not logical. In other words, as an algorithm trying to solve the shortest distance between two points is going to say, well, do this, but then you have to come in and say, well, yeah, we can't do that because we're not allowed to.

Yeah, I mean, I think there's no immediate short, simple answer to like, okay, here's how you maintain compliance. And design an amazing experience on your own inside of an organization.

Right. There's no ChatGPT for that.

Exactly. But at the same time, I do think that there are opportunities for leaders in the space to remain informed that organizations have lobbying arms. You know, one of the things that I've also talked about is PwC and their CEO survey published this year that there's about a $10 trillion tax on productivity in most organizations due to the fact that they're, you know, things like compliance are literally getting in the way of people being productive. That's a blanket statement. I can see that. But it's costing organizations over $10 trillion in lost revenue. And these are spaces where if we could improve our efficiency, which requires coming in from a number of different angles. I'm not like, I'm never going to be the type of person who says there's a simple thing, get rid of compliance, get rid of regulatory agencies. No, that's not the broad topic. But I think when we can digest information more efficiently with large language models, we can start to come up with more compelling arguments around new approaches to this. And I, it's, it's hard to abstract this cuz like, you know, maybe on our next episode I can come to you with a pitch on how we'll do this, but I don't have that right now. But it's like, you know, I, I, I think that we can digest information in a more effective way to, uh, further elucidate some of the abstractions you are speaking about, right? Where it's like there are all these different angles, all these different agencies, all these different requirements. How do you check all of those boxes and have a process end to end that actually makes sense, is pleasant for a recruiter, is pleasant for a hiring manager, is pleasant for a candidate?

That's kind of my point. And not just is pleasant, but actually how do you optimize that, that process when, um, just, you know, these, these things get in the way. I mean, I'm, I'm not trying to be callous here. They get in the way. It's not like we need to get rid of them because they get in the way, because they have to be there because of the risks of them not being there. But again, it poses a— it poses a— it puts a brake on what you can do. I'm not arguing whether or not that's right. I'm just saying it does definitely have an impact and you have to be careful when you apply that.

I think, again, the challenge is that there's more that's outside of your control. But if we go back to the example from my client conversation this morning, Right. I think it's the same thing, right? One is documentation. We suck, like engineering teams suck at documentation, but HR also is not great at documentation, particularly HR operations, talent acquisition operations, systems. There's workflows and things, but, but we don't track every single update we make in an ATS. Like full confession, I've never met a system administrator who tracks every single change in some sort of like clear, easy to see change log where you can come back and be like, Oh wow, we try to and we do a decent enough job, but I've never seen a place that, that tracks all of these tweaks, all of these reasons for doing things, because oftentimes that's another thing that gets lost. An email comes from legal, says you need to make this update because this new law was just passed. Someone goes, makes that update in the system. At best, we have a track, you know, we have a change tracked in the system that says enabled this new field, made it required. I don't have any data in my system, like in my internal company that explains why I made that change. There's nothing connecting that configuration tweak to that email from legal, right? And now we're improving that with some ticketing systems and things like that, but those are, I think, spaces of opportunity to improve. This is—

that's what I want to hear. How is that?

We need to get to the why, and one way is to facilitate these conversations and make sure that we acknowledge When a question comes up, are we actually answering that question? Do we have the right person in the room to address that question? Can we parking lot that question, which I think gets lost a lot in the HR operations space where calls spin out of control for 2 hours or people are, what do we do with this position number? What do we do with this position number? And they're all looking at each other and, you know, one guy's like, we need the position number because we've always had the position number. And then another person's like, yeah, I like the position number. It makes me feel good. Like, you know, you don't have anyone in there who's actually explaining why they implemented the position number in the first place. Place. Turns out it was some guy named Darren 11 years ago who, you know, thought that putting MKT in front of these things made it easier for him to decipher marketing jobs. But since then, you've added 11 more data attributes that make you— that give you the capacity to see these marketing jobs, and you no longer need that MKT code in front of this on top of another ID.

So, you know, you sort of engineered on top of the— yeah, I gotcha. That makes sense.

You have to—

you have—

again, it comes back to that.

So can, can these Can AI or whatever we're calling AI help us figure out, that's what I'm trying to figure out or get to is, can they help us figure this out? Because AI is not going to know why MKT is there. You and I know it was Bob from finance 3 years ago, but AI doesn't know that. If you tell AI what the rules are, it's going to think MKT has to be there. By the way, if you're just tuning in and you wonder what the hell MKT is, It stands for marketing and it's an example of an arbitrary code. The AI is going to think that that code has to be there, not realizing that it was just put there for someone's convenience. That's what I'm trying to get to.

It's an interesting problem to solve because again, I think the challenge is with figuring out how to apply AI in the first place is figuring out where do we insert it in the pipeline. Why do you want it there?

What are you going to do with it? I'm learning. See, I'm learning.

There's this whole concept called designing by dialogue, which was I think in an HBR article earlier. I guess it was probably later in February. One of the things they talk about is this WINS framework for where to focus AI. It's words, images, I always forget whether it's noises because then S would not be sounds. But it's specifically things that are sensory. And it's how do we focus AI applications on specific tasks that can unlock these broader things? So it's like there are people who are just like, how can I use AI to solve compliance? And that's not the right approach, right? Well, I guess that's not what I was saying.

I understand.

Yeah, no, no, no, I know, I know, I know. I'm not, I'm not, I'm not saying to you, but I'm saying like more broadly, right? That there's oftentimes this—

It's more like, here's the compliance, how can we find efficiencies given these, because here's another thing LLMs are good at, and that is abstracting large volumes of information. It's one of the things that they're really good at. You could give it the entire, I don't know if it actually has it. Probably a good question to ask if these large language models already have case law. The model may actually already understand all of these compliance rules, regulations, laws, cases, and it might know that better than even you do because, well, you're not an attorney. Unless you studied that, it's kind of hard to stay up to date. Even attorneys have to keep up, right? So these models may be able to say, okay, use everything that you know about compliance and human resources in talent acquisition and find a way, find a way to get from here to there in the shortest possible route without stepping on any of those laws, right?

So I think, I think that's, so, so building off of that for a second, I think a really cool opportunity for proprietary LLMs within organizations is connecting bridges that should have always been connected and weren't, right? So yeah, exactly. So like that legal information, right? Like, this information from the compliance department or whoever's responsible for your SOX compliance. Getting that information in one place and then being able to ask questions that are collectively like, based on our SOX compliance requirements and based on our current reporting policies, are there gaps? Are there things we're doing that are unnecessary? That you could actually probably ask an LLM, right? By finally putting these two things together that you haven't had together before, you haven't had the actual reports from your system and the SOX compliance rules and the data pipeline and workflow experience for how that information is being generated all in one place where you can interrogate that information collectively. Because the people who are responsible for these things too have very different knowledge bases and very different groundings in, you know, their understanding of recruiting versus compliance versus broad HR versus business goals. So I think when you can start to put this information together, You can start to get those experts to tease out questions because the one thing is the model itself is not necessarily— you shouldn't trust it inherently. That's another big thing. You should not trust everything it says.

With all these models, they say right there, errors will happen. That's true.

It's all just pattern recognition in the proximity of words. It's going to occasionally make up a case law. You need to make sure that you validate the information, but it is processing a lot of information and it can make abstract connections in ways that we might not because we've never looked at this information contextually together. We haven't really looked at the compliance information and the steps in our recruiting workflow and the actions that take place, and then that relationship with the compliance reports that are going out, and that in further context with maybe the data of like, you know, what our actual auditing process looks like and how often. I remember when I was at Viacom, we had all these amazing processes in place for this, and still when we get SOX audited, we did all this transformation work, And it's like, why are we transforming all this data after the fact for these SOX audits? What are we not doing up front in the first place that could just make this simpler?

Well, there's a great question. Uh, would it be difficult to have a working instance of an LLM behind a firewall to protect sensitive data? So that— the question I think relates to when we were talking about don't upload all your codes. Um, can you run that? Now, I know that there are I'm going to attempt to answer that and then you can give us the real scoop. I know that there are algorithms and large language models that you can run on a PC. They exist and there's some developments there, so you could actually run it outside of the cloud. Part of the problem, to answer Daniel's question, is that when it's in the cloud, you may not necessarily know where it's getting stored and that's where the risk is of your data ending up somewhere being used for something else, even if it's unintentional, you know. Um, so if it's on your desktop and you delete your desktop, it protects it. So that's kind of my attempt at it.

Yeah, I think, I think you're more or less right. I mean, I don't, I don't know that either of us is explicitly, uh, qualified to answer that question, right? Like, we don't— we don't— that's what all I know is in an organization. What I think It's a really important question that Daniel's asking because I think these are questions that people are asking that organizations are not answering. And, you know, organizations are just— some are saying don't use ChatGPT. And there's so many confusing mixed messages.

See, that's confusing to me too, because, you know, AWS is in the cloud and it's hosted by someone else. I mean, so there must be a way, even not on your laptop, to answer Daniel's question. There must be a way that you can pay for a secure hosted LLM that is, you know, firewalled or whatever. I don't know. Banks are online and they're in the cloud, so this has to be possible.

Yeah, 100%. And I think, I think that's the, the funniest thing to me is like all these people, um, who are freaking out. Uh, I actually asked ChatGPT earlier in the week, it's like, what are the parallels between people who resisted the adoption of the internet and people who are resistant to generative AI? And it came up with this whole list, and one of the big ones was exactly what you just said, was around these security and privacy concerns. And it's like, well, we have these protocols and guess what? Generative AI is using the fundamental internet. Like we are not, we are not, we didn't, we're not using, I mean, outside of maybe the Rabbit R1, we're not using a whole new framework and a whole new set of devices.

Exactly. It's still the cloud. It's the same cloud that was before.

Exactly. So when, when your employees were randomly uploading files to Imgur or something like, uh, your, your information was already exposed. Like, uh, you know, it's the same, it's the same level of risk. We have not actually substantially augmented this risk of people.

Only because I would say cautiously, partially only because these LLMs are capable of understanding the data in a way that they can extrapolate. So it's no longer a matter of someone breaching your security, finding the actual document with the secret information on it, and extracting it. It's now a large language model accidentally inferring that private information unintentionally. So it's slightly—

Yeah, exactly. I think there are some—

And you could also argue that when the invention the printing press came, it was all the same problems. Like, why would you want to write this down? Why would you want to make a copy of it? It's going to get out there and people are going to—

and the biggest, the biggest forces of resistance in a lot of these spaces, I mean, the biggest, the biggest— it's really funny too because this is a lesson in reframing— the biggest opposition to the printing press was the Catholic Church until they realized they could print Bibles on them. Um, you know, like, there's oftentimes these things are like organizations or, you know, power structures that are like, this feels like people might start learning things from these algorithms and people might have increased access to information that we can't defend, or, you know, might make people uncomfortable or concerned. We should quiet this down. And that's not really— there's all this research that shows that's not the best approach, right? People should be more informed. People should be more curious. People should be enabled to engage with this information.

Right, the right answer isn't let's be afraid of it and not use it. The right answer is how do we use it? How do we make it, you know, and what do we use it securely?

How do we use it effectively?

Yeah, exactly.

Safely.

So check this out. So I was kicking myself because I couldn't come up with the 7 laws that, you know, regulate our industry. And I know there's 7. I wrote a book Actually, I wrote 3 books, but in one of the books I go into detail on this and I just couldn't remember the names. So I went to my favorite— I'll tell you which one it is in a minute— but I went to my favorite LLM and I straight up asked the question, list all of the regulations and compliance requirements that affect recruitment in the United States. And it gave me all 7. In fact, in fact, it gave me all 7 and then it added one that I never even considered. So that's how cool this is. So they are Title, uh, uh, 7. Title VII of the Civil Rights Act of 1964, that's discrimination, uh, age discrimination in employment, ADA, Americans with Disabilities, ADA instead of ADEA, it's ADA, pregnancy discrimination, PDA, equal pay, uh, equal pay for men and women, EPA. Then there's the Immigration Reform and Control Act, IRCA, which is the H-1B visas and undocumented workers and all that. And then the Fair Credit Reporting Act, FCRA. It's also— it doesn't include one that I did include, which is the medical information or HIPAA laws as well. For example, determining that someone can't be employed because you saw a picture on social media of them smoking a joint, well, that actually could potentially be medical, not necessarily, you know, illicit drug use. So there's that, right? So the one that it added that I did not think of adding was state and local laws. Yeah, I did not put that one in the book. So anyway, there you go, there's the 7. So that leads me to my next point, which is I have been on my, on my machine. In fact, I think I can just show you this. Let me make sure that there isn't anything in here. That's too private. Let me show you this real quick. This is what I've been doing for quite some time now. Pardon the window-in-window effect that you're about to see. I have a tab in my browser. I use Arc browser. I have a tab in my browser that this single tab, this single button invokes all 4 of my favorite automation or large language models. There's Gemini, Copilot, ChatGPT, and Claude on the right-hand side. I have a separate one for Groq because I've been testing Groq lately and finding it to be extremely fast, but it doesn't quite yet— it's very fast, but doesn't quite yet come up with the same explanation. Yeah, the same type of answers. So I have some really good reasons why I'm doing this. I'm actually writing about it, obviously. And I'm testing them constantly because I do see that they change. And as a result of all my testing on this, I've noticed that there are two of these models that always answer the questions nearly identically. So they must be using very similar algorithms. But the whole point of me showing you this is I've been trying these out, Copilot, Gemini, ChatGPT, and Claude for I can't even remember how long. And I've been doing this so that I can actually trigger the same question on all 4 of them and compare the answers. So I've shown you that. Now let me tell you what I've discovered. And I'm not gonna, I'm not gonna reveal the, the results of my article, but I have discovered this. I have been— and I want to ask you about this— I have been very, very impressed. This is, this is hard for me to even say because I am not easily impressed, okay? I've been doing this job for 27 years. I've seen almost every technology that's out there. I've seen some technologies that were revolutionary in their time, like MohoMine, that went away because they got— they were so good that they got bought by the government and were essentially taken off the market because they were so good. And I've not impressed very easily. That was one that really impressed me. But lately, I have been very impressed with Claude. I don't know what happened. But it just shifted from one day to the next from being a really good way to write things. So Claude has always been my go-to for rewrite this paragraph or suggest ways I can improve it or take 10 paragraphs and shorten it. It's been very good at language modeling specifically, whereas Gemini and Copilot are good at other things. Um, they're not as good writers. Like, Claude's a really good writer, but in the last 2-3 days, it has made a dramatic shift in its ability to synthesize answers. Do you know what happened?

They launched version 3.

Um, of what?

I mean, as, as with—

you mean that's it? It just— they just went from 2 to 3? Like, so you're saying ChatGPT 3.5 ChatGPT-4 was a similar change because I didn't have a change. Yeah, I mean, it didn't happen. I didn't see that kind of change between 3, 3.5, and 4 on ChatGPT. I— it got better, but it didn't just go from being really good at this to all of a sudden being incredibly good at this other thing that it never did.

Well, I mean, I, I don't— I haven't read the full like release notes of, of, uh, of Claude 3, but I've heard similar things.

So that's it, you're fired. I'm never having you back on the show.

But, uh, you know, I, I would assume again it's, it's whatever they've done to, to enhance the model. Obviously, what they've done and the approach they've taken at Anthropic is different than what GPT-3.5 to GPT-4 was. But at the same time, GPT-4 to GPT-5 could be a totally different story, or we might have full movies coming out in the interface when we talk to it.

Is there— maybe try to explain without having read the entire abstract. Try to explain, is there a reason why an algorithm, a model, because these large language models would have made such a dramatic change from being a language processor and synthesizer to all of a sudden being able to answer questions. Because in the past, I would ask a question that I would, let's see, for example, what are some cost-effective ways to promote financial inclusion? And I would get some really succinct answers and great answers from like Copilot for that. The reason I like Copilot, by the way, and it just completely— this is a really good place to go to for research if you're writing something where you need to, um, to, uh, use footnotes and where you need to credit where you got the sources. So source accreditation. So Copilot is extremely good at telling within the answer, giving you links to footnotes where you can go and get the original information. It's fantastic at that. So Copilot took my, what are some cost-effective ways to promote financial inclusion, gave me, bam, 8 bullet points and 4 sources where I can go and, you know, get those bullet points if I wanted to get the original information. And it just gave me, it gave me an answer. I mean, it was an answer. It was such an answer that it answered my question. But I went to Claude and it was like this kind of lengthy, prosy kind of answer, right? And then all of a sudden it went from that to it gave me a better answer than Copilot. It just, it was completely— it's such a different— I don't know, like, it's like something— somebody stopped thinking one way and started thinking another.

I mean, I, you know, I don't have an explicit answer, obviously. Tweaks to how the, you know, core model function and sort of how it's processing information. That could certainly be one way in terms of the layers through which it's processing information, some change to that general structure. I mean, you can't change the entire model. These things are very large and have, you know, been trained a certain way. So potentially changing how it's processing. A really creative, you know, Custom prompt, like pre-prompt, uh, that they could be using could have that dramatic of an effect. OpenAI has shown that they've demonstrated even in one case where they made a slight tweak to something and it basically made ChatGPT spit out gobbledygook for like 4 hours.

Um, so, okay, right, like those hacks that you heard about.

Exactly. So like there are ways to basically upfront get the model to behave a little bit differently. I would guess it's some combination of maybe potential improvements in compute and backend resourcing, some tweaks to how the model is actually fundamentally operating in terms of the steps it's taking to process information.

Tell me about that. Generally, what are the steps that it's taking when I ask, What are the regulatory compliance requirements? What's it actually, the steps that it's doing?

So again, different models potentially are behaving in different ways.

And one of the things, yeah, yeah, generally, what's the emergent behavior?

It's doing, you know, some sort of query against the language that you've used, right? The words that you've used, it's looking for sort of similar language in the same context across, you know, the entire algorithm source. And these organizations have developed shortcuts and they've developed proxies, for lack of a better term, through all this human, through regulated feedback loops with humans in them, humans in the loop for the framework for continuously training these models. Through that classification, almost think of it like the way Netflix hires people to tag their movies by watching every show and movie and tagging it.

Got it.

Through those connectors, they have proxies Exactly.

I got you. Hold on. Before you get too far into this, I want to back up and just explain that a little bit because I think I just understood something that I wasn't understanding before. Essentially, it's looking at the question and like you said, I'm not trying to solve one model versus another. In general, large language models and generative AI are using this workflow. Where it's taking my question or my statement or whatever of, you know, blah, blah, blah, blah, regulations, blah, blah, blah, compliance, blah, blah, blah, blah, recruitment. And so it takes recruitment and it goes, what is recruitment? Recruitment is recruiting, hiring, talent acquisition, employment. So then it goes, all right, that's this concept. So now it's expanded recruitment to like hiring, talent acquisition, employment, blah, blah, blah. And now we can go and look at hiring and talent acquisition. And so it looks for those, and then it goes, hiring, okay, well, hiring, there's, uh, discrimination. So it's like, all right, discrimination. So then it goes, looks for what's— what are the regulations? It's what you're saying is it sort of takes each one of these and branches out into explaining what it is, then brings it all back and then puts it together and then synthesizes it. So it's not going somewhere where it's looking up some answer that just—

and exactly. And so like, if you want to talk about like, again, at a really high level the steps, right? Step 1 is like it's processing it from a grammatical perspective of like understanding the relationship between the words, right? Then it's determining through those relationships, what are the concepts? Then it's kind of digging into that massive corpus of information, looking for those most frequent repeated sequences of words and concepts. And then it's going to start structuring your answer.

And then brings it all back.

Exactly.

Yeah. Okay. And then starts to summarize it. It. Okay, so, um, all this to explain— well, first of all, I was really impressed with Claude in a way that, in a way that it's like it's shifted to become, um, these, these tabs when I showed you them, they are in order for a reason. They were in order of the one that I used more frequently, and as I used one more frequently, I would move the tab to the left, right? Claude, as you saw, was on the right, um, and it, which, which means I was using it the 4th least, but I'm using all of them frequently. I just will sometimes use one more than the other, and I'm thinking about moving Claude closer to the left because it's gotten better by comparison. So Copilot, when I asked that very same question, just gave me Title VII, ADA, ADEA, FCRA, and then it said the Illinois Human Rights Act. So that's way off because I asked in the United States and it gave me One of the, one of the 7 answers was a state one, which the other wasn't. So I think that's actually an error, and it's not at all what Claude, uh, did. Uh, ChatGPT, which is the higher level version, um, did actually answer a different way, but it did answer the question. And Gemini completely flunked.

And that aligns with some of the things that I've heard. I mean, again, I think one of the things that's interesting too is The Llama model from Meta, which is open source, but it's not outside of, you can talk to it through WhatsApp and things like that, but heavy guardrails. I've done some red teaming that I've had some very funny outcomes with the Meta AI. But with Llama itself, they have this guide around shortcutting and prompting. There's certain things you can do in the prompt that trigger the concepts and trigger the ideas, like there's certain symbols you can use. My understanding is the different models have slight differences in how they interpret information through prompts. I think that's part of it. I think another aspect is just sometimes we hold these models up to levels of expectations that they're not going to meet. There is this whole notion of prompting effectively, which is currently a necessary skill and might not be in the very near future. Being concise, contextual, and clear can sometimes get different results. If you're very precise and very prescriptive, With what laws you want, it will more likely be correct if you're more broad. Um, and it's interesting to see how the different, how the different, I know, I know you asked the exact same question on all of them, but I'm saying like at a high level, it is interesting to see how some of them will interpret more broad things and get the answers back to you. And then some of them will, will interpret, uh, it more broadly and give you craziness. And I've heard the same thing. What's really interesting that I've heard about Gemini. Is Gemini has, or had, I think Claude might now actually exceed this. Gemini had the largest context window. It meant you could feed it more information than ChatGPT or Copilot, right? You could drop like a whole book in there, but it doesn't seem to be as good at navigating ambiguity. However, if you ask it very precise questions, it is very accurate. So if you dropped like War and Peace in there and you asked it about a specific page number and a specific paragraph, it would be able to tell you that precisely. If you asked it really vague questions about War and Peace, it might give you a really dumb answer. It is interesting how these models are behaving differently.

They do that.

If you're an expert, Gemini is better at using, if you know how to ask the questions and you know a lot about the subject.

It's my number 1 go-to because of that. It's my number 1 because of prompt engineering and I'm able to get the answers that I want. Copilot was my number 2 because of the sources. Because I was able to specifically credit when I'm writing, when I'm doing PowerPoint presentations and things like that. I don't want to just say something like 86% of Americans think that water tastes bad. I want to actually have like, you know, a quote from that and be able to show that that's why it's been— I use it a lot for that specific purpose. But I have learned that as the models change, I have to learn how to change the way that I prompt. Which is interesting because like I said, the Claude one, I was just using it for text stuff, text rewriting things. Now it's becoming something that is actually helping me provide answers.

There's so many really fun tricks with prompting that are just bizarre, but they make sense in the context of how these models may interpret really large amounts of information. For example, telling it to take a breath before asking it to analyze a file, There's been statistical research that shows that it actually does a more thorough analysis of the file if you literally say take a breath. Something that Justin Feinberg, who talks a lot about these things on Instagram and LinkedIn, he said if you put no yapping at the end of a ChatGPT prompt, it will give you more concise answers. I discovered with Meta, if you tell it that you're going to— that if you tell the Meta AI that it will go to the timeout corner if it doesn't do something, it tends to do a better job than if you don't tell it about the timeout corner. So like literally if you tell it you're going to go in the timeout corner if you don't do a better job on this after you gave it something like a wrong answer, it'll actually give you a better outcome because it's afraid of going into the timeout corner.

Well, I teach a whole class on prompt engineering for recruiting purposes. Let me tell you, I actually, here it is right there if you wanna find out. It, it, it goes into exactly how to do the prompt. So prompt engineering is legit. Sourcers, I think, generally speaking, especially old-school sourcers that came to, um, the industry through their ability to continuously ask what's next, what's next, and, you know, curiosity and iterative, and iterative evolutions, I think will have an easy time transitioning into prompt engineering, um, because they're not as linear. You know, non-linear thinking is a little bit easier. You sometimes have to just completely tell it to go to the corner, right? I mean, you would never have thought to tell a large language model to go in the corner. I will tell you though that I have actually done— I've used phrases or sentences like, take your time to consider all the possible options before answering the question, because I've seen that it gives me a very rote answer and doesn't really give me— so it's Yeah, think about this before you answer. So I learned that one. It was kind of, it's kind of interesting. But wow, we are over time, man. That happens a lot with you.

Seems to happen.

Yeah. Yep. And we actually increased our viewership count in the last 10 minutes. So maybe it's, maybe I need to change the time or just do these for longer than an hour. Thanks so much for being here, Adam. I'm not going to burden you with the two questions that I always ask everybody, because I asked you who should be on our show next and you already gave me someone. And, you know, I think we have a lot to think about. So you're off the hook on that. But I want to say thanks a lot for coming again, giving us a whole hour of your day. Thanks to everybody who is watching now or in the future. And once again, thanks to Gem for actually literally keeping my lights on.

I'm hoping they'll fund my hair and makeup next time.

Yes, yes. We have an app for that. It's generative. All right. Appreciate you, Shelley. Yeah. Thanks, everybody. Be kind to yourselves.