How AI-led interviews unlock hidden talent

Hi everyone, and thank you for joining us today for the Better Together event. We're here to dive deeper into the findings from our study, Better Together: Quantifying the Benefits of AI-Assisted Recruitment, conducted by the MicroOne research team in collaboration with Stanford. In this randomized trial with over 37,000 applicants, we found that AI-led interviews surfaced hidden talent while making hiring faster, fairer, and more accurate. And we're fortunate to have the authors of the study with us today. So I'll introduce them before letting them start with the event. We have Ada Aka, PhD and professor at Wharton University and Stanford, Emil Palikot, PhD, professor at Stanford and Northeastern University, Ali Ansari, the founder and CEO of MicroOne, Nima Yastani, Lead AI Researcher at MicroOne, Ava Fitoussi, Lead AI Researcher at MicroOne, and Mark Esposito, PhD and Chief Economist Officer at MicroOne and Professor at Harvard. He will guide today's discussion and walk us through the economic analysis and implications of this research. And with that, I'll pass it over to Mark to get us started.

Great, Tofia. Thank you very much. Very nice to see you all. So Ada, Emil, Nima, Ali, and Ab and I, we kind of prepared this together as part of what we're trying to do in the economic analysis team at MicroOne. There's been a lot of expectation about this specific webinar. First of all, the paper has been almost like seminal for many things that we have done within the context of how the company has been pioneering the commission of recruitment and AI. So we'll be using this paper several times as benchmark for conversation and an anchor for many more conversations we want to have. Besides this, and I guess I'll start from a question that I don't have the answer for because I think no matter how much we read the paper, we don't know how the paper originated and started. So in our prep conversation, Ada, you gave me a wonderful excuse to start with you. You know, you are an academic. You were at the time, I think, doing your postdoc at Stanford, same as Emil. And so you were not part of MicroOne. So how did you get into this collaboration? So how did this start? I'd love to have your perspective and I'll try to then bounce it back over to, to Ali from a perspective of, of an entrepreneur is always looking for possible problem to be addressed with the business solution. And we'll try to get the story right.

Ada.

Yeah.

Thank you, Mark. Hello everyone.

I'm Ada.

I am actually a cognitive psychologist and now a professor at Stanford's Graduate School of Business in the marketing area. So I study how people make decisions, and with all of the technological advances that are happening, I became increasingly excited about studying how these things shift the way people think about technology, they use it, as well as some of these outcomes. So for an academic, it's basically an amazing world where you start studying in the lab and then move these bigger questions to the field setups, the types of situations that you all deal with every day. And it's been such a pleasure to be able to work together and collaborate on one big experiment that we ran.

Yeah, I'm going to use that. Ali, yeah, go ahead. I want to have your side of the story.

Yeah, sorry to call you out, Mark. I actually— it's funny, I just— I DM'd Otto on LinkedIn. This was like, I think maybe a year ago at this point, year and a half ago. And I just said, hey, I think this would be a nice collaboration. We have some really incredible data on AI interviews, and I think we can run an experiment to see, you know, how AI interviews affect the overall hiring process. And she got back to me right away and we kind of started the project and it took a while. I thought it was gonna be quick, but we took a pretty long time running the experiments and, you know, doing peer reviews and so forth and came out pretty nicely.

[FOREIGN LANGUAGE] Look, I'm gonna stay with you now 'cause I think there's a part of what you said that intrigues all of us, right? I think as an academic, I can understand the starting point that Adam mentioned before, You know, it's kind of where you want to test an idea, right, in the real world rather than just in theoretical constructs. But on your side, you know, you are an entrepreneur. You're trying to solve problems that are real in the real world. What were you noticing in the recruitment space that suddenly made you think, first of all, this paper, this project is important for me. Second, I'm going to put it into, into context of maybe a product or service. I'm going to run a micro one. So What was from your perspective the problem that recruitment and AI were facing that eventually brought you to Zara, right? Which is an important, an eventual development for MicroOne.

Yeah, that's a great question. So I think I'll give the quick origin story which kind of helps answer this and then, you know, in retrospect we understood the problems we're solving a bit more. But the way that it started is basically I, I was running an agency when I was at Berkeley, a very simple software development agency. And what we would do is we, we would build web, web applications and, and iOS apps for other companies. And essentially one of the main kind of limiting factors of that, that agency that I was building was every time a project would come in, I would have to hire a bunch of new engineers for that given project, and I would have to vet those engineers. And essentially a lot of my time was being spent on interviews as I was trying to build that company. So long story short, I decided to build this internal AI tool using one of the early GPTs that had come out. This was like probably about 4 years ago at this point to screen these engineers and basically conversate with them before, before I do and gimme, you know, gimme a report on how they did. So this, this, this initial internal tool at the agency basically became MicroOne. And of course, I ended up kind of focusing on that and it went from a simple AI screener to this like end-to-end AI recruitment engine, which allows companies to source and vet different experts. And I think that the problem initially was just me trying to solve the, you know, me being on interviews as the main thing I do. Most of the days I was trying to not, not do that anymore. That was kind of the initial problem I was trying to solve. And then later on, you know, in retrospect, we realized that it's obviously a much, much broader problem, which is kind of comes in two categories. The first is companies are bombarded with applications for almost like any role that any company posts. You, you will get applicants and, and sometimes you get hundreds and sometimes even thousands. And you, the companies don't really know what to do with those applicants. And so a lot of resumes end up being kind of wasted and not really looked at. And so what we're doing is, of course, helping companies save time by finding the right candidates through an AI interview process. But we're also allowing candidates to actually present themselves as much more than their resume that often actually doesn't even get you know, get an eye on it. Um, so those are, those are sort of the two fundamental, uh, problems that we're after.

Thanks for this. I, uh, enjoy a lot the story. I also enjoy how much LinkedIn kind of, uh, sometimes helps. I think it's the very same way I got into, into MicroOne, right? There was a conversation with LinkedIn. So I, I'm happy that, that we have a story to tell about, um, social media, uh, tools sometimes helping to connect the right people. If that's okay, guys, I'd like to go a bit deeper into kind of the structure of the paper. I'm going to go to Emil again. Like Ada, you are an academic. You are— you did your postdoc at Stanford, and I think now you are a professor at Northeastern. So look, you run the results that this, this paper is bringing forward are quite interesting. It was a randomized trial, if I understand correctly. At one point in time as a researcher, you must find yourself in front of this and say, oh, I wasn't expecting this. What was that for you? What was the unexpected part of the paper? And like Ada said before, tell us just a little bit about your background. So we kind of connect in what kind of academic discipline you were bringing to the conversation. Emil.

Awesome. Thanks. Great question. I actually met Ali on the— also in social media, but that was X rather than LinkedIn. And I have a more computer science background, which I guess that's why you can find me more often on X rather than on LinkedIn. But yeah, basically we started chatting on like what's interesting in recommendation systems for the labor market and got together and decided to collaborate on this project. Let me start actually why we wanted to do a randomized experiment, right? Because this is, I think, what really gives credibility to the study. And the reason we took so long with planning and pilots and all of this is that there is so much, you know, hype and at the same time fear around AI in hiring that we were thinking that it's going to be really easy for people to project their opinion rather than really look at the results and take them at face value. So if we would only look at, for example, historical data, I think people could say, well, maybe the candidates who don't like AI opted out, or maybe these are different types of people. So This randomized experiment really gives us rock-solid evidence. And we know that this is due to use of AI rather than selection of candidates or something like that. So what surprised me? I have to admit, and I think I never mentioned this to Ali, I was quite skeptical when we went into this project. At that time, you know, there was no high-quality evidence that those systems are effective for hiring. I was maybe thinking it's going to be time-saving, that the recruiters will get the, will go through candidates faster, but I wasn't sure whether we're going to see improvement in the vetting quality. So what really surprised me is that the results are so, that the magnitude is so large, right? That we have over half of candidates who were selected by AI passed later on the human interview as compared to a third of the resume screened candidates. Candidates. So this is a very big effect. And that means it's not really about saving time so much, although that is there as well. It really feels like AI was surfacing better candidates. And, you know, even add to this, like, we have collected this external data from LinkedIn profiles of candidates months later, and we saw that the candidates that AI selected not only were more successful in human interviews, but they were also more likely to find jobs in other companies. So there is really something pretty remarkable about the way the system works. And then on the top of that, around this, the same period, a couple of months after we released our draft, there are other papers that started appearing showing that those AI systems in hiring can work for call centers, can work can work for tutors. So I think there is a body of evidence that is being built kind of following our initial work that is actually shifting my initial belief. So I think that's one.

Yeah, that's great. I mean, look, I'm going to ask Alba to ask a question that she was kind of wondering about. Alba just graduated from Berkeley in econometrics. And so I think it's great that she was looking at this paper with respect to like, what's the structure research there? Alba, you were kind of playing with this question in your mind. What was that about that you wanted to ask Amel?

Yeah, so my last class at Berkeley was actually at the same time that I was reading the paper. And I actually found out this is one of the first large-scale RCTs, so randomized controlled trial, in AI-assisted recruitment. So I was wondering on like an economist, like, point of view, or like, how did it come about? Like, why did you choose to do an RCT to run this specific experiment, and what did you hope it would show?

Yeah, thanks. Uh, so RCT is a gold standard in terms of the quality of evidence it can, it can produce. And I think the main— I think Well, two main motivations. Ada and I are independent researchers. We care about the quality of work. We want to show what we believe is true rather than what will promote the company. And it's great that those two things came together. But we really wanted to have evidence that our peers and other people in the field will say that this is a carefully executed study. And second is because it's an important issue that impacts so many people, right? Everybody's looking for a job at some point. So if we participate in this discussion, I want to be able to say something with a high degree of confidence. So I think that's basically the two reasons.

Nice. Thank you.

Thanks so much, Albert, for this. Thanks for this, Emil. Really appreciate it. And fascinating to see kind of the story. We kind of heard about three of the four authors about this. The last one that I'd like to pull in of the conversation is Nima. Nima, you are a lead researcher in the AI field at MicroOne. You and I had some conversations, some of them also very philosophical last time that we were on a call together, right? And I remember when Ali came as a guest speaker the first time to my class at Harvard, you also came and you were talking about basically the research. So I always found fascinating that there is a significant part of your time at MicroOne, not just running the company, but validating this from a point of research. Now, one of the things that I guess is quite fascinating about this paper is the fact that you found out that a lot of the resumes, right, that are really coming through are not necessarily what they are. So you can say there are misrepresentations, there's inflation. So how do you make sense of all of this? Is it just that we never had a way to vet them? We were based on good faith, or it was mostly a process that was organically growing into something that now became out of proportion. How do you make sense of that part of the paper that kind of reveals a very different kind of numerical evidence about the resume.

Yeah, I think what this paper shows, and it's really like the nail in the coffin, is that the resume is only like a small piece of what is needed, you know, which is much more holistic view of a person. And it's so hard to tell, you know, the character and the work ethic and the experience of somebody just through, you know, a one-pager. And this paper and the results from this paper really show that AI interviews, they really are able to level the playing field in some sense, give people who don't have— maybe they didn't go to the top universities, or maybe they didn't have experience in the flagship companies, but they are extremely talented. They have great skills. They've worked on themselves. Many independent projects, and this really highlights those people, those like, those gems that are just not usually found, and it's hard to find through just a simple resume. So the resume is now just, I think it's very clear from the paper, it's just a small part of the whole, you know, candidate's profile. And I, you know, from my experience and from my friends and colleagues, I feel like everybody always feels like, oh, if I could just get the interview, like if I could just get in front, of them and explain myself and talk about my work and show my passion, then I could make it to the next rounds. But a lot of times, like Ali pointed out, the resume just stops right there. Most people don't even— a lot of people feel like they get ghosted. They don't even get their resume seen. And this really gives everyone a chance to get in that first, to really show off and flex what they've learned and put so much time and effort into, which is all of the work experience and in education.

That's great. So just a small comment, guys. Over 100 people are currently live with us right now. For those of you watching this, please, there is a Q&A part if you want to put your question. We'll try to find some space towards the end of the conversation for your question to kind of shape our dialogue. Just a note. I have a question that I want to start with Ali with and then kind of extend that in a two-tier question to both Ada and Emil. So Ali, from the time you were part of this paper and project to the time that you ran basically a company process, right, in the real world, how do you find that we were able to really address, right, some of the findings the paper was really putting forward? Did you find that somehow we do have answer to some of this challenge that we had in recruitment and kind of technology is trying to close the gap? Do you find that we're just at the very beginning of this? And I'll try to finish on this context, Ali. I think for the time that you start to work on this, we didn't have as much buzz around employment, AI, and young people as much as we see right now. I think sometimes these days we talk about an entire generation that's been impacted, right, by the fact junior skills are kind of mimicked in some of the AI models. So where do you see the industry going with this?

Yeah, great. Question. I think there's two things I'll say here. And I think, you know, the first one is we really were trying to answer one really important question, which is what is the ultimate hiring outcome if you go through a resume process versus an AI interview process? And more specifically, what is the human interview pass rate for those that for those that don't complete an AI interview and for those that do. And I think we were able to answer that really nicely. Of course, we started the paper with historical data, which Emil didn't like, on answering this, this question of if you, if we look historically on those that did well in the AI interview, which we defined with some combination of skills that some ratings on the skills that they had in the interview, for those that did well, what was the subsequent human interview pass rates? And we saw a very large correlation between for those that did well, getting a very high pass rate, and for those that did, you know, for those that failed, quote unquote, and didn't do well, getting actually a very low subsequent human entry pass rate. So of course, the historical data showed that, but then we ran the control experiment to really solidify that answer, which I think we did a good job with. I think with this, one of the main implications of this is, and I think it's very sort of conveys, we convey it in the title, which is Better Together, which is we want AI agents and humans to work together. I think recruiters are going to become way more powerful using a software like this. And that's really our goal. I think if you think of like, you know, sort of a typical recruiter, oftentimes they don't have domain expertise in a lot of the interviews that they take. So they can use the software to conduct very technical interviews for any given domain and then review those reports and kind of give their final stamp of approval. So we're trying to orchestrate as best as we can this human and AI, you know, collaboration specifically in the recruitment space. And I think, you know, maybe ironically, we help labs actually hire experts to train their models as well. So there's also a human and AI collaboration happening on the downstream jobs that a lot of the candidates that we help hire happen as well. So So there's basically human plus AI in kind of both aspects of our funnel.

Great. I'm going to build from where you say— I'm going to go to Ada. Ada, fascinated by the fact that you've been doing a lot of decision-making. You are a scholar who studied this, right? And then I want to bring the conversation over to Emil on a slightly different angle. So it seems good news if I'm just hearing the conversation. Light is happening right now. But kind of the role of, of a researcher is also to investigate on kind of the ripple effects of things. So is there any risk that you see in what seems to be a good story so far, right? Things that you find that maybe it's a good story because it offset a problem that we had, but sometimes, as you know, we might fix a problem that backfires later on. So what's your take on this? How do you see the risk taxonomy shifting in light of this work.

Yeah, thank you for these additional perspectives. I think the efficacy results, they were very promising. They are very promising, but that basically opens up a full set of new questions that we're very excited about. We have done a little bit in terms of the mechanisms we study in the paper, yet there's so much to be done. So one question we tackled, just scratched on the surface, is about who likes these AI interviews. Can we link some of our results to the demographic profiles of our candidates and see if particular people drop out or are more enthusiastic about being a part of this, as well as who benefits from such AI interviews? And these are all open empirical questions such that we had not necessarily thought thoughts about before running our experiments. And in terms of the dropout rates, or those who are less likely to be a part of the AI recruiting process because of their own preferences, were those that were more experienced and older in our candidate sample. So, so that really opens up a potential new question of can we better study these candidates and can we study different populations to be able to point what are some of the biggest thoughts in these candidates' minds and then have rigorous evidence to be able to back up any of the future claims we're showing or hypotheses that we're testing to be able to see if, for example, there's a general algorithm aversion for certain people and basically provide the data to show what we have. And it's basically one question leading to many. But that's a direction that we should go.

I'd love to have one more follow-up with you. And you said something that, although I wanted to go over to Emma, but I can't help it but ask you more. Look, a lot of, I think, the benefit of this is that we're kind of fixing a level of agency that was not too efficient, right, by improving the access. On the other hand, if we are removing that agency, is it somehow downsizing, you know, some of the, the, the more human-centric processes that are at the same time so much echoed as important. There's a lot of emphasis trying to bring people back in the loop no matter how much we automate in. So is there any concern that you have sometimes when you see that we're disintermediating basically entire professions by improving them with technology?

Yeah, I think there's a lot of self-selection going on who chooses to be a part of this versus how things were done in the past. But we just My answer would be we just need more and more data to be able to study these questions. And I'm optimistic about this process.

And sorry, Mark, one point I want to add. Yes, I think, you know, it's a really good question. I, I think there's two things I would say. One is we, in my view, we are actually making this process even more human. And there's two main reasons. One is Candidates are— they get interviews that are very relevant to their skills that they often don't get in initial screening calls. Of course, recruiters are amazing, but they're not domain experts in every domain. I mean, that is an obvious statement, but I think it's an important one. The interviews that happen with, you know, the medical doctors that go through our platform, lawyers that go through our platform, and, you know, engineers from top schools, they get the first screening call to be extremely relevant to the skills they know. And they're often very pleased by the level of conversation and questions that the system is able to conduct. And so that actually, I think, really improves the candidate experience relative to an initial recruitment screening call that I think is important to point out. I think that the second thing is we always make sure that humans, human recruiters on the other end are making the final calls on candidates. They're reviewing reports and they're reviewing not only the interview recordings, but they're looking at their LinkedIn resumes, etc. And making the final decisions, which often could include subsequent human interview rounds that are more tailored again to the, to the, to the candidate. So yeah, that's one point.

That's great. No, I think it's great because I think you are really talking about the fact this is not about automation in favor of automation only, is automation by augmentation, right, of the recruiters, right? If I understand this correctly. So this is, this isn't great and I'm happy you brought this up. Emil, I want to bring you back in the conversation. Look, there's something you said before when you were talking to Ali about the skepticism that you had at the beginning of this. And it's great, right? The power of research is really that it puts in front of evidence that sometimes, you know, definitely changes the way we see things, which I think is humbling. Now, I want to use that skepticism moving forward. You know, a lot of this study seems to show a disproportionately good impact for the younger talent. But do you think that we're kind of shifting the problem to a different part of the labor equity, that now we're making it less fair for other people. So do you have another area, or let's say future research, where you think there is— it's worth for us to be skeptical because we need to be vigilant about the question we want to ask?

That's an awesome question. So I— well, so what we find in the paper is that younger candidates, less experienced candidates, candidates with fewer high-level credentials are benefiting from being in AI. That's the result that we have. And there are several mechanisms for it. I think the, the, the, what holds maybe more experienced, more older candidates might be the fact that they are less familiar with AI systems. Maybe, you know, the opportunity cost of their time is higher. But I think these are transient. I think as we get more familiar with these technologies, those problems, I think, will likely go away. I'm speculating, but that would— that's my feeling. And in the same way as, you know, a couple of years ago, maybe applying on a job through LinkedIn felt awkward. And right now it seems like the most natural thing to do. But adding to this, the— I'm actually seeing this as a great thing, right? The labor market When we think about labor economics, we often think about the Matthew effect, which is this rich get richer type of dynamic where, you know, people with early credentials, they went to a famous university, they get internship in a famous company, they are, they keep compounding those advantages. And the resume screening seems to be a mechanism that feeds that cycle, right? Because if we have a screening process that will interview everybody who went to Stanford or Harvard or Berkeley and will not interview people who didn't go elsewhere, then we have rich gets richer situation. And it looks like the AI interview seems to blunt that a little bit. It's probably not a silver bullet for equity, but I feel like we are moving things into a better direction.

That's great. Thanks, Elisa. I really appreciate your analysis on this. And it's really where I think there will be even more space piece of research. All I can think of right now is, uh, how many more projects and data can we really work on this, right? Because it's really expanding our own understanding of the same question that, that we're having. Um, I'm gonna bring it back to—

yeah, of course, like, you know, the new direct— new, uh, research direction, right? This is a disruptive technology, right, that will change a lot of, uh, behaviors in the labor market, a lot of practices, a lot of institutions. One big thing that we document in the paper is that the time investment shifts, right? That we now have candidates taking long interviews rather than recruiters reviewing a lot of, a lot of, a lot of resumes. And this will have downstream consequences, right? Maybe people will be applying for fewer jobs and we might end up in a place where you have a better screening technology, but a smaller pool of candidates to to screen for. So these are, you know, dynamics in the labor market that we need to watch very carefully. And these are the dynamics in the labor market that actually highlight the importance of intermediaries like MicroOne, right? That the platforms that can, you know, play a little bit of a coordinating role where we can aggregate interviews, we can aggregate candidates. So the market is kind of going into the equilibrium that we like that is more efficient rather than equilibrium, which is an anti-candidate outcome in which candidates are, you know, spending 1 hour talking to a chatbot to every job that they apply to.

Yeah. I mean, look, I, first of all, thanks for saying this. When I started to get closer to our friends from MicroOne, what I really found interesting, right, is that their sense of rigor on the data was not necessarily different from what I would expect it for me to be an integral process. But then it wasn't determined by basically supporting just the academic process discourse. It was really trying to determine a problem in the ground. So I'm really happy that you said that. I think there's a generational company that will have both an applied research angle that will nonetheless keep the level of rigor necessary for the data, as Ada said before, to be the only thing they really want to consume to understand more and more. So this is absolutely critical. And I'm happy you brought this up. There's one element that Alva was actually sharing with me about temporal morality in terms of short and long term that I think, Abba, you wanted to ask to Nima, right, on that account.

Yeah, actually, I think this is a good question for both Nima and then Ali to answer. Nima, because you do a lot of research, and Ali, because you see the effects of Zara or AI Recruiter on an everyday basis. And so I found that, so the study shows, you know, short-term performance, like interview pass rates, and medium-term labor outcomes, so job position, but I'm wondering, do you think that now we have any insights on, you know, performance data post-hire, like the longer-term performance in terms of like retention of those candidates or productivity or, you know, anything that's more longer-term?

Yeah, I think Ada said this well, you know, this answering this one question opens up many other questions. How can we look, how can we identify long-term signals that can be traced back to going through an AI interview rather than, you know, traditional resume screening. I think future research does need to focus on this, and this is what we have our eyes set on, is identifying those signals like retention, maybe looking at negative signals as well as positive signals. So, you know, staying at your job for a while, getting promoted, those are all great signals to show that the AI interview was actually successful in identifying, you know, a great candidate of good character, of commitment, of hard work and, you know, learning at your job, getting better at your job, getting promoted at your job. So those are all, you know, long-term signals that future researchers, like, you know, needs to take into account. But as Edda said as well, you know, this answering this one question opens up many other questions that we're very keen to answer.

Yeah, and just one thing to add there, I think it's, it's a really important part of our overall loop, which is kind of two, two things on the job ground truth performance data as well as on the job happiness data. It's kind of like two, two kind of separate categories of like how happy is the employer, therefore how are you performing and how happy are you on, on the job? How are you kind of enjoying the role and We, for this control experience, we, we don't have that data. Uh, we have some downstream, you know, LinkedIn data to see the same candidates on how they, how they did on future jobs and so forth. But there's, we're, we're grateful to be working with a lot of projects where we do have that on-the-job ground truth performance data, and we have it in a really cool way where it's in a very wide variety of domains. Um, you know, and often highly skilled domains as well. In large volumes. And so we use that data to, to kind of analyze and further improve the system because at the end of the day, what the, what our model is trying to do is predict the performance of any candidate on any given job and predict how happy they will be. The latter, we have, you know, a lot of work to do on still. It's a much harder problem to kind of predict the enjoyment levels of candidates on any given job. The former, of course, we've made good progress on, but it's still, we have a lot of work to do. And I think that this kind of ties us well into like the fundamental research question we have, which is where should humanity spend its time is what we're trying to answer. And of course, humans spend most of their time on their jobs. So we're answering this question in the context of jobs, which is, which kind of, you know, infers that we want to find this optimal job matching problem and really fundamentally solve this job matching problem. Which I think, you know, the AI interviews is a good step towards.

That's great. Ali, look, I think you're giving us kind of the perfect moment for us to switch over to the Q&A. Again, to those who are writing this question, guys, thanks so much being part of this. We don't know if we'll be able to answer all of them, but we'll do our best to cover some. There is one question, and I'm not going in the order in which the question came, just in terms of what makes sense. I want to bring Ada into conversation. There's a question from somebody called Sarah who basically asked whether these AI interviews evaluate emotional intelligence. And if I want to expand the question to say, are these interviews only looking at analytical intelligence, or would you consider it as a space for us to expand on the multiple intelligence that defines a job somehow? Considering what you studied, somehow behavior is an important part of your work. So how do you find the limitation, but also the expansion of this conversation around the typology of intelligence that we're really testing with interviews?

Yeah, thank you for the question. In fact, we get a similar question often when I present this work in academic spheres as well. And the way interviews work, I think, so far from the paper's results, what we see is that we have much more consistent interviews from person to person if an AI recruiter is in place relative to when you think of human decision-making, the types of cues humans concentrate on or the types of information they'd be— they become stuck with. So I think there's much more consistency in the AI interviews. That's what we find in the analyses that we run. And even though our position was a technical position, I think there's some more information from the language in which people are interacting with the AI recruiter, and we can quantify that type of situation. So The current evidence that we have seems to be that even though we're evaluating 3, 4 skills that are technical, this conversational nature of the interview still adds to the conversation. That being said, when you're comparing situations, we have decades of work looking at human interviews, only human interviews. So I think there's, there's a long space that we can, we can cover when you get to AI interviews. So far, it seems like the conversational nature of it, the flow, as well as the technical quality of the AI interviewer, um, is good for the technical positions. And I think, uh, we're excited to see for the soft skill positions or other domains as well.

If I could also just, you know, add one more thing.

Yeah, please. Yeah.

So I think that emotional intelligence evaluation is a really, really great example of this point that we want to drive is that these AI interviews are assisting humans. It's AI plus humans. And though the AI interview does evaluate soft skills based on how you get your point across and how the interview goes absent of your technical language, it's always— this is something that humans are still very good at relative to AI, and that's, you know, having a gut feeling or an inclination onto the emotional intelligence of the person. So we never want to take humans completely out of the loop here. And I think this is a great highlight of, like, you know, evaluating emotional intelligence is a great highlight of humans and AI, you know, working together to get an absolutely amazing level of evaluation.

Yeah.

And so if I can kind of build on what Ada and you said now, Nima, right, there's a question actually when I was reading Emil, you came to my mind, and I want to bring it over to you. It's actually Anas who's asking this question. So he's asking about fairness in the process. And the reason why I like this question to you, Emil, is that you and I were talking before about kind of labor equity, right? And you kind of brought the idea that we should be careful that we're not necessarily creating a process that is creating this archetype of success to the successful. And so this question that Anas is asking is about whether or not we are able to take into account elements that would generate a lack of fairness, like, for example, race, gender, last name, right? It was mentioned in the example about women's names selected at equal rate or racial biases or any kind of minority. So what's your take when you're looking at this research? How fair do you think we are able to see this data as representative? And how much do you think is kind of still camouflaged by but the very same kind of archetype of success you were mentioning before?

Right. That's a very deep question. And I realize that, yeah, there is, there's much to it, right? So there is the— so the first thing is the mafia effect that I talked in the past, right? And the mafia effect is definitely somehow reproducing and amplifying the disparities between social demographic groups, right? Like there are differences in, you know, who gets to the best college and then those differences get amplified. So if we can target, counter a little bit of that, that's great. But having said that, the, like, I don't know the nitty-gritty details of how the Zara works, but this is an LLM-based system. These models are black boxes. We don't exactly know, like, how they perform. This has to be audited. It's very important that there are independent researchers involved in auditing outcomes. It's, we will, you know, be publishing papers about this. And, you know, whenever we see something that is alarming, definitely, we'll definitely write a paper about this. But one, like one last thought here is that this, I believe is also the onus here is on the employer, right? At the end of the day, the employer gets a report. Report might be biased, might be not biased, but at the end of the day, it's a job of a recruiter to, of the employer to take it to make a decision. And the computer science research in the field of recommendation systems has been generally arguing that what we need is an accurate understanding of whether that candidate is a good fit for the job. And then ex post, we should be introducing fairness criteria on the selection. And John Kleinberg has done a lot of research in that direction, for example. Great.

Well, I mean, look, I appreciate you took a stab at a question. I agree with you. It was deep. It's one of those questions that if they happen in the classroom, they can take a good chunk of it, right? And it's great that nonetheless you gave your insight on this. Ava, I want to— oh, yeah, sorry, sorry. I think— oh, sorry, Ada. Yeah, go ahead, go ahead. And then I think Ava wanted to ask a question to, I think, Ali or Nima. But yeah, go ahead, Ada.

Yeah, from the research perspective, I think one of the additional questions that I personally am very excited about is to see if this type of a technology can go beyond the black box decision-making type of approach, but also tell us tell the candidates, tell the employer additional insights, additional information that would be not at all possible by simply looking at a resume. So I think that's a direction I would personally be very excited to pursue and look forward to. And hopefully, maybe that's going to be the case for Micro-1 too. And we can, we can tell all of you about it in the future.

Yeah. And just one point before, before you guys jump in, just one thing I want to add to You know, like Emil said, really deep question that we obviously think about every day. One key point is that the AI interview system, when it is conducting the interview, does not have the full demographic data in any way to ask the questions and then more importantly, to assess the candidate. And of course, this does not mean that you're removing the bias that may come from demographic data because you can infer that and the model may try to guess at that, you know, innately and so forth. But I think you can make a fair argument that this does at least minimize it relative to a human that has that full demographic data as they're conducting an interview. So I think that's one key thing to think about. We're able to remove this data. The AI interviewer simply does not know the demographics or resume of the candidate as it conducts the interview. And then, like I said, more importantly, as it comes to its final assessment of that candidate, which is it rates the skills, it does so merely on the merit of the content of their answers, which I think is a really good step towards the mitigation of bias. And then lastly, of course, the black box issue is a big one. There's, you know, the way to truly understand this is to— you have to get all the weights and see how they add up on every call. Obviously, that's impossible. And so you can get to a proxy understanding of how these models work, and you can only get to better proxies over time. I don't think we'll be able to fully understand the exact decision-making process on these models. So I think the way to try to mitigate bias over time is to look at outcomes and compare those outcomes to only human-led outcomes. And I think we've done that partly in this, which is we look at minority groups and younger folks and so forth that actually benefit from this. That outcome is a promising step towards less bias.

Thanks for this, Ali. Ava, over to you. I think, and I know, guys, we are, we're not too far from bringing this to a close. Just so if you stay with me for 5 more minutes, then we'll try to, to bring it over to a, to a place where we can at least set up the path forward, right, from where we are. Ava, over to you.

Nice. So there's actually 2 great questions that I think can be answered by both Nima and Ali again. But one of them was towards Ali. So Ali, you mentioned earlier that, you know, those interviews also cater to domain-specific and skill-specific, you know, interviews. And they were wondering, like, what are the benchmarks used by Zara to compare the candidates' answers? How do you decide, you know, what's the threshold to pass an interview? And You know, just generally, how do you choose those key indicators?

Yeah, it's a great question. So the, the AI system does not pass or fail candidates. The way the structure is, the interview has some set of skills that are defined. Let's say, you know, in a, in a math context, you can define specific math topics that you want to interview a candidate on. Algebra, linear algebra, geometry, whatever it is. And the candidate will be interviewed on those set of skills. And, and then our model, you know, does the assessment based on the answer, based on the conversation that the candidate has for each of those skills. And the way that we, the way that we further train the model is we look at ultimately how the candidate does on this, you know, maybe it's a math tutoring job, let's say we look at signals on the job, and we improve the model's assessment through that.

Nice.

Thank you.

You want to go ahead?

Yeah, exactly.

Oh, you're muted, Nima.

Sorry about that. So, yeah, so it is really— what's great about these interviews is that they're tailored to every single job. you know, that you're, that you're, you're applying for. So, um, like Ali pointed out, you know, a math, the, the questions in a math interview is gonna be wildly different than the questions in, um, engineering or for a physician. And so based off the job description, based off the role that is being interviewed for, the, the AI interview caters its questions and follow-up questions to your responses to the actual job profile. And the signals are, you know, how you respond and how you respond to the follow-up questions. So these are extremely dynamic interviews that are unique. So the signals are unique to each individual interview, though there's, you know, a broad structure to all of them. They're really catered to every single role and every single person that is taking the interview.

Great. Thanks so much for this, Nima. I think there was a question that was addressed directly to you, but you kind of gave us an answer in the direction. Question too, Ray. I appreciate that. So guys, we've been trying to keep this a little bit less than an hour, right? And so I'd love to maybe get our final round with all of you sharing just a short reflection on the way forward. We won't be able to cover all the other questions, but I'd like to ask each one of you. So Ada, Emil, Ali, and Nima. So moving forward, where do you think we should actually prioritize our focus and our intents? And that can be both in terms of research that we want bring forward in terms of where do we go from here. And in terms of, I think, MicroOne, where do you see this taking the company forward? So I'd love to have a sense of progression towards the future that was kind of triggered by this paper, because I think it's great that we're legitimizing papers. Research is a great form of thought process that we can really generate no matter whether we are academics or entrepreneurs. So yeah, I'm going to start from you. Where would you like this research to take your own research next?

Yeah, in this first paper we concentrated on the efficacy dependent variable and now I would love to shift more towards the user perspective and get to an idea of what can a candidate beyond the job related, specific job related benefits, what can they benefit from this type of a system? So can they learn something about themselves? Can they learn how to better interview? All of these potential motivational, behavioral, and job-related additional benefits. I would like to study them and quantify them.

That's great. And so, Emil, you come from a slightly different background, not in psychology and behavior, but in computer science. But equally, this kind of question we're discussing today, there's something that must be tickling your curiosity on what do you want you could study further, what would that be?

Um, so let me start by echoing what Ada said, that the candidate experience, I think it's, uh, is a very interesting, uh, research direction, very obviously very important for candidates, and very, very little research done on the topic. Um, but just to add to this, I think, you know, when we take the step back, I think it's Is it, I think there is this best case scenario and then there is this worst case scenario of how the AI can disrupt the hiring process. There is the, you know, the best case scenario in which we really broaden the opportunity, we surface the most capable candidates and everybody gets a fair chance. But there's also the worst case scenario in which we shift costs to applicants, we're diminishing the pool. And we create new inequities. So really, I think that the fundamental question here is how we should design and govern this market. And I think that's why the intermediaries are so important. And that's why I'm so excited that MicroOne is actually interested in doing research and figuring out with independent researchers on how we can design this market to end up in this good equilibrium because this will, you know, influence every single job seeker everywhere.

So far, Ada and Emily, I agree 100% with what are kind of the wishes for us to take forward this research on your individual interest points. And I agree on which direction to take and why the partnership with companies operating in the field is important. Now let's bring it over to, to, to Nima and Ali. You guys are you're running successful companies and you're seeing this directly from a perspective of clients, markets, opportunities, kind of the convergence of the entire technology landscape. Where do we go from here? Where did the paper take you in your thought process that is now inspiring the next set of actions? Either of you folks.

Yeah, happy to chime in here. Nemo, feel free to elaborate on this. I think I 100% agree with, with all on Emil. I think The biggest limiting factor that we need to find equilibrium on is the total time spent by candidates. I think we are very clearly, you know, I think the paper is very convincing, even though we're not, it's not a sales paper, we're not trying to convince anyone. But I think as a result, we have very much convinced how effective it is for companies. I think it's not really a question, it's going to continue to get more effective over time. We just have to ensure that it is very effective for candidates as well. And of course, we have good reason to believe that it is, but I think it's still the sort of factor that we need to heavily focus on. I always tell Nima and everyone on the team, we have two customers that are really, really important. They're our clients as well as candidates. Candidates are also customers of ours. And we think about the candidate experience really deeply every single day. And I think one of the steps which Emil, you know, really kind of summarized well towards solving this is using this technology to build a marketplace that gives this sort of global profile for candidates, which results in one interview can result in many jobs in the future, not just one interview mapping to one job, which then results in, you know, of course many candidates go through interviews for any given one job. If we make that mapping actually more than one job and they have this sort of global profile that's built for them that represents their skills really well, that they can share with other employers as well, whether it's on the Micron platform or not, I think this is a really good step towards solving this, this equilibrium. So I would say that is an area that I'm also very interested in researching.

Yeah, I think there's a really exciting future that's increasingly becoming a reality and I think already is is that the MicroOne talent pool, the platform is now becoming where you can create your own profile that's very representative of who you are, your experience, your skillset, your talents, and that is widely applicable. So I think as the talent pool continues to grow very rapidly and people take even one, two, many interviews, they're expanding their profile. You really understand who each person is And the recruitment process gets further and further away from, you know, the traditional old-fashioned, just simple interviews that we now know are just ineffective. So now we're transitioning to this much better marketplace of individuals that benefits both parties very much.

Great.

Guys, look, I want to say, first of all, thanks for having been amazing on this virtual series that we're running. There are many unanswered questions, and that is not not because we didn't want to answer them. It's because it's like really is a scarcity problem right now. Not enough time for the number of questions that we get in. But it inspires me to imagine that maybe we can have another session where we can address some of those questions in a more dedicated way because I think we're just getting started on the conversation about AI recruitment and everything that really comes with that, no matter the angle. Ava and I, we have just actually published a short piece for the World Economic Forum where your paper was cited. We do talk about the importance of your paper, across other articles that are supporting this kind of trend. And so again, you inspire our personal work on this. Always been great in supporting what we're doing in the, in the economic analysis unit at MicroOne, and truly appreciate that. And look, I always regret the fact that we don't do this in person. Sometime it would be nicer over, over coffee and in a place where we can just simply be together. But I think it's, it's is going to be basically a rain check, I think, with you, Emil, for sure. In Northeastern, we're going to see each other relatively soon with Ali and Nada and Nima, I think, and Alva as well. I think it's a matter of me going over to San Francisco and finding some time to meet you in person. But guys, thanks so much for being wonderful. We had over 100 people or more at any point in time that were basically watching us. It'll be available later on. But it really speaks highly, first of all, about the interest on the topic.

Topic.

But more importantly for me, on how much good research and great research it is what we need these days, where we're having daunting questions where it's not easy to have an answer, but at least we have a good way to formulate a question, which is, I think, what research really does. So thank you again for having been part of this, and I wish you all a great day ahead. I think I'm the only one on the other side of the planet which is night, unless you are also in Europe, Emil, I don't know. But otherwise, all the best. And from Mark, uh, have a great day forward. Thank you, guys.

Thanks, guys. Bye, everybody.

Thanks.

Thanks. Bye-bye. Bye-bye.