Identifying Genuine Talent in Finance Using Simulation-Based Assessments

Welcome to the Market Maker Podcast. Today we are talking about one of the biggest challenges happening in early careers hiring across finance, something that affects both the firms doing the hiring and the students trying to break in. Traditional CVs, interviews, and online tests are not predicting real performance, and it's wasting time, money, and talent. So I'm joined by Lara, our VP of Product, who's leading AmplifyMe's new simulation-based assessment solution to unpack how banks, trading firms can hire smarter and fairer, and how this helps the next generation of candidates prove themselves through real-world ability and not their background. Okay, so maybe let's start with setting the scene. We've obviously had this massive shift with how firms are hiring. Volumes are insane. We talk to clients all day long. Who are saying, you know, the quality hasn't followed the increase in volume. So maybe to start this conversation, a couple of stats for you. I had to write these down.

Yeah.

So you have to forgive me. But there's a couple I think that really make sense of this conversation. So this year, JPMorgan, so one of the big bulge bracket banks, had 630,000 applicants for about 4,000 jobs. So acceptance rate, is 0.7%, right? And 2 years ago, that number was closer to 3%, right? So just the volume has gone up, so that acceptance rate has gone down. So that's stat number 1. Yeah, stat number 2, uh, research shows that— so research shows that AI-generated CVs can inflate applicant pools by up to 40%.

Yes.

So many recruiters reporting difficulty distinguishing— this is a big point of topic, I know, distinguishing genuine intent and talent. And then another stat, traditional CVs and aptitude tests have less than 25% validity for on-the-job performance.

Yes.

And then a final one from Bright Network who did a study found that 80% of students, and I know our internal community studies have found this even higher than that, more like north of 90%, that students use AI tools in their job applications.

Yes.

I was on campus last night at Warwick University and I was talking to students about this, which is what I do as a job all day long. And it was almost felt like this prisoner's dilemma is the best way I can think of it.

Right.

Where most students don't want to cheat, but they know that everyone else is cheating and therefore they feel compelled to cheat even though they know they shouldn't.

Yes.

It was such a weird abstract kind of situation that we're in, or paradigm that we're in now. Yeah. Kind of like the 100-meter race.

Right.

When people are thinking about the Olympics and everyone's, you know, margins are so fine for those who make it or those who win or lose. Some of them are cheating, then they all feel compelled to cheat. And if you're not cheating, well then you're just at a loss.

It's true. Or maybe it's a win, right? Because that's the only way to differentiate your CV and your cover letter. Because all the other ones are gonna be quite similar to each other because that's what happens, right?

Yeah.

You put your CV in an LLM and say, well, optimize my CV for this role in this company, in this setup, and they all come out very, very similar. So possibly not cheating is actually— Yeah, the best way to cheat.

So maybe we could talk about then the current system. Yeah. So how it works. So the I guess let me just do the landscape of a normal, I guess, sell-side investment bank. And large financial institutions would be the same. So the first step is for a student to then submit a CV and a cover letter. So that's, that's point one. And then normally they, if they pass that situation, they get a numerical situational judgment test, something of that nature, online testing.

Yeah.

And then that leads to a video hire view situation.

Yes.

And then that leads to, um, in-person interviews, assessment centres, things like that.

Yeah.

Now I know, having spoken to all these students, every single one of them now is cheating. I won't name names.

Yeah.

A large proportion are gaming these traditional early parts of that process.

Yes.

So things like CV, cover letter, AI tools, the online testing, AI tools.

Yeah.

So yeah, maybe you could just talk a little bit more about about that area in particular and how we can address those sorts of things?

Yeah, definitely. I think it's a, it's a massive problem, right? Because a lot of those tests, I mean, all those tests and all those practices were built way before AI, and AI just came overnight almost. I mean, it's been a while now, but it's still overnight for, for when you're thinking about progress in psychometrics. You need to do a lot of research, you need to test and validate, and you need to administer it to many people collect performance data and all that. So it takes years. And so of course they're cheatable and it's a massive problem. That means that we need to start thinking about how do we— well, a lot of people think about how do we fight AI, right? How do we counterbalance it? How do we proctor in a different way? I think that is part of the problem. For me, the best solution is to embrace AI in the same way that we embrace calculators, in the same way that I remember when I was a psychology student, I didn't study that long ago to do stats by hand, but long enough ago to hear my professor saying, all the students are using software to do statistics. They're going to be very, very dumb. Yet statistics was still the subject in which everybody failed because nobody knew how to use that interface, and let alone doing things by hand. So there's a similar situation now with AI that we need to understand how people will use AI in the workplace and how people will use AI to present themselves for a job. So really embedding AI in the way in which we assess people, assuming that they will use AI, we just, we just put that in the assessment, which is kind of where we're trying to go at Amplify for certain parts of our assessment. So, so really trying to understand If I was, for example, a quant trader and/or a banker or any, any person in an early, early role in finance, would I be able to use AI or not? The answer normally is yes, I would be able, if not encouraged, to use AI. So how do we test how people use AI, how people prompt it, how people do quality assurance, how people challenge what AI is actually generating rather than just give people an assessment and then hoping that they're not going to use AI and using some sort of assessment to see if they have used AI and reject them from the process. That's not going to give us really good hires. So I think that is one part of the challenge. The other part of the challenge is numbers. So there's this huge amount of numbers. These CVs are very, very similar. And as you said, 90%, I would say probably even more than 90% of people would cheat. And I've heard teenagers calling not using AI the vintage way. So I assume that a lot of, you know, our target group are going to be using AI. If more than, uh, than 90%. As I said, best way to differentiate yourself right here, right now is to not use AI because, um, you know, there's too many, too many similar CVs and too, too many similar cover letters. But from a, you know, from a perspective of, of an employer, if I have to think about all the TA, the talent acquisition teams having, you know, waking up one morning and seeing like half a million, uh, almost application for a couple of thousand roles. I wouldn't want to be them, right? Because there's no way of, of telling which one is good, which one is bad, and which one is completely unsuitable. Because anybody can, can, can apply for anything. There's no gatekeeping there. So having a really good filter at that point is really important. But what do you filter on? That is a problem.

So on that point then, so big part of your time is spent developing these specific assessments. We're a company where our entire purpose is simulation, so experiential kind of learning and experiences. How does assessment and simulations come together then?

Yeah.

In this context of trying to ascertain talent.

Yeah. So, simulation, when we're talking about assessment, is described very often as a work sample. So, you're given an example of work, people to do, they'll do it. That is normally a really, really good predictor of performance. So it's one of the highest predictive validity in the field across different industries. It's a completely different way of assessing people because you're not asking them questions, you're not allowing them to cheat in terms of like— it's not like a right or wrong question most of the time. So if you're thinking about what we're doing with the market simulation, You know, you have, uh, you have all these, uh, assets that you have to trade, and you have to make decisions, you have to interpret news. You need to have a very good understanding of the fields. You have to be really, really quick in making your decisions. And there's no ChatGPT or Claude or Gemini that I can tell you in very few seconds, these are your assets, this is the price that— this is how the price has moved in the last, I don't know, 2 hours of simulated activity. And, and this is how the news is relating to this asset, but not to the other. And giving you an answer that will make you— they will give you an advantage within the time frame that we're looking at, how you can make a decision. And, and also, and I don't want to spill too many beans on our IP, that's not all we're looking at, right? So we're looking at how people react to certain mistakes that they make, how people's performance shifts between periods of high intensity, of many conflicting news, for example, compared to when there's absolutely nothing going on. So there's a lot of behavioural trace data that we use in the simulation, and that is not something that you can—

Yeah.

You can ChatGPT yourself out of, right? Yeah. And this is quite similar to the work that I've done previously. And at that time I didn't know about, about AI. I mean, big LLM didn't even exist. It was just a, like a something that, uh, very few people used in very few user cases. It wasn't like widespread, so it wasn't a threat at all. The reason why I started researching a new type of assessment was because I was adamant to find a way to prevent people from cheating assessments. So I had that pet peeve during my studies, and then I ended up defining this scientific rationale for Startup, which is still going on after like 12, 13 years, where we use game technology and neuroscience to look at how people behave and make decisions at a very early stage of cognition. Not like in a declarative way. So we didn't ask people, is this right or wrong? Or how many friends do you have? Or do you consider yourself assertive or agreeable? We just put people in, in this exercise, in these, in these tasks where we manipulated little things in their environment to observe their behavior. And that is more like a game-based assessment or behavior-based assessment. And I'm bringing a little bit of that into Amplify. So really looking at what people do, what kind of behavioural trace they leave behind when they are trying to maximise the performance on a given task.

I think that's really cool because actually I see lots of articles where pretty quickly what students do is they'll circulate the questions that are being asked for an interview, for example.

Yes.

But what you're saying is actually that every experience interaction is unique that they're having. And so therefore you're just measuring these underlying core behaviours.

Yeah.

That are much I'm assuming stronger signals of future performance or intent of how they might be.

Absolutely. And those are things that you can't really mimic. There's a whole theory of behaviours being a measure of performance. So for example, when you're looking at introverted and extroverted people, we're thinking about this is just personality traits, etc. But there's also a theory that says you are able to be an extrovert, you're able to produce those behaviors. You're able to put up with people and still be happy. And if you think about the kind of things that we're looking at in our simulations that are above and beyond your P&L and your drawdowns and all those like very objective and measurable metrics, it's your ability to sustain performance when the market is going a little bit crazy or your ability to make good decisions after you lost everything. Or your ability to just step out of your simulation for a minute and really think about what you need to do. So those are things that you can't just decide to do. Those are innate predispositions that will show, that will emerge in measurable behaviors when you have them in you. Yeah.

And some of the really encouraging early signs that we've had from some of our partners is that I know that not only then when they're getting the job, but when they're on the internship, the conversion rate of them then getting from intern to grad and then from grad to a high-performing individual within the organisation, there's a pretty strong correlation with us picking out those signals. I think, you know, one of the major US banks that we work with, the conversion rate was in the high 80%, whereas normally it would be in the low 30%. That's amazing. In terms of conversion. Um, so it's amazing finding those core key behavioural elements is really a real distinct marker. One thing I was reading, I was trying to think of what's a really simple analogy. Yeah, because I saw this article saying that getting into Oxford or Cambridge, I think the application rate was like 17.5% of applicants to get in. Harvard is like 5% or something. Uh, the SAS or the Navy SEALs would be like 1.5%.

Yes.

But then Goldman Sachs is 0.7% or JPMorgan. And it was like, oh, hang about. So you're saying it's harder to get into Goldman's than it is to get in the SAS? I was like, I find that hard to believe. Yes. But I actually thought that the SAS selection process, 'cause I think most people are aware of that, it's like this crazy process they go through, 4 weeks of continuous training.

Yes.

Where they're put in scenarios where they're having to think deal with challenges, solve problems, communicate, work as part of a team, all because they're looking for high performers when it actually comes to them in the field. It's so interesting that actually, you know, it's exciting that that's almost what we're trying to mimic is a similar type of process.

Definitely.

To try and find the future finance professionals that we're working with.

Absolutely. It's so close to the same thing that they do in SAS. They, they they really mimic what people will do in the workplace. So they have a really good understanding of who they have to bet on. And that's the same thing that we're doing with a simulation. So we're really replicating what people will do in their job, and we have a snapshot of their future, right? There's, at this point, because we're focusing on early career mostly, there's so much room for training people, but we can really see the potential, right? And in the same way in the SAS, they will then train people that will become stronger and stronger and stronger. So what they want to see is that, do they have the predisposition? Can they do the thing that we need them to do if we give them all the training that we have planned to give them? And this is how we can help employers, right? We can help them to see, to, you know, to have like a little crystal ball and see like this person, this is how this person may operate. When you put them on the task. So is that good enough performance for you to think that you can train this person to become excellent? Can you give them all the job rotation? Can you give them the coaching, the mentoring that they need versus, you know, this is a— this is an aptitude test. They got 9 out of 10, right? And there's another 300,000 people who got 9 out of 10. But this person has done it like 10% faster. Doesn't matter. Absolutely not. Because it's still within the margin of error of not differentiating people from one another. But when you're looking at the amount of data that you can collect from a 40-minute simulation that really replicates what you would do in real life, or even more, because our quant simulations are actually a bit fairly longer than that, you can see a lot. You can have a lot more data. And a lot more meaningful data, contextualized data that you can really stand by.

And so, so what's the practicality of this? So as a, as a product that we're developing, assessment tool, where would it fit then within this traditional model that these hiring managers are currently using?

At the moment, it's quite late in the process because that's when, um, normally, um, the way in which recruitment processes work is that you keep on filtering out people who are not suitable for the role. The first big filter is at the very beginning, and then as you progress through the selection process, you get richer and richer data. Because if you think about the receiver of that data, you know, if you have to skim from half a million to 100,000, all you want to see is a yes or no, right? You don't really want to look at all the data, how good they they are at the P&Ls rather than how good of an M&A they are suggesting, how good a model they're making for an M&A. So there's a tendency to put these more richer assessment types at the end where, you know, you can actually leverage the data to inform, say, for example, a conversation in an assessment center or even an interview guide. For example, now we're working with a, we're partnering with a specific client where they say, well, there's so much data coming out from the simulation that it would be a shame to lose it. Let's try and work out a way in which we can pick little bits and bobs from the simulation data and ask very specific interview questions to people about their decision, about the behavior, about how they felt when that thing happened.

Oh, that sounds so cool from a student's perspective. Yes, because now it feels like you're actually involved in a genuine interaction. Because sometimes I feel so sorry for the student. It's like, here are these really abstract, just slightly disconnected questions that are like the necessary ones that you just have to answer. Yeah, it doesn't feel very personal. That entire current process feels very unpersonal.

Yeah.

And so then you end up going through, as many students will, will always do, apply to multiple different firms, and they're all very similar. There's no uniqueness, there's no personality. I don't feel much affinity for that brand. When now the competition is even more fierce, yeah, as an employer I need to differentiate. But if I can then talk to that individual about their experience that they've had through that interaction within our pipeline, yeah, it'll be such more of a richer engagement. I think you get way better interaction with the candidate.

Yeah, and also from a candidate perspective, there's this, um, you know, there's a massive perk of having the opportunity to perform. It's a, it's a really good predictor of acceptance of, um, of an offer, for example, having the opportunity to perform in, uh, in the selection process. And if you think about it, when you have like really standardized— I mean, standardized is good, right? Let's not say the standardized is not good. So one of the best tools that you can use, generally speaking, in selection is really standardized interview where you ask really specific questions, you have a rubric, etc. So, but sometimes what happens is that you are not asked the questions that bring your, your special skills and your true self to light. So you ask, you ask really interesting question, but you don't have the opportunity to really showcase who you are. When you still have a standardized framework of harvesting data points to inform specific questions about, for example, you did something really good in that, in that simulation. Tell me why and tell me how. Just tell me, tell me how you formed that decision. How do you feel about that? Did you realize how good that was? Or Or on the flip side, like, what happened there? You made a mistake, but can we talk about it? Rather than saying, oh, you got that wrong, minus 1 point. There's a lot more opportunity for the candidate to really talk themselves.

I think it's authenticity. The candidate, you get a true insight into them and how they feel and how they behave and things like that. And that's because, again, these students are so well trained now to answer these. I mean, there's loads of documents you can find on the web. Here are the top 1,000 questions you'll be asked in an IBD interview. And literally the student will learn all 1,000 and now use the AI tool. What's the perfect response? And it's a memorization process. Yeah, it's just, I'm going to regurgitate the perfect technical answer.

Yeah.

So there's no authenticity at all. So for an employer— so, so the student's not getting much, the employer's getting zero actual genuine feedback on this person's potential. Yeah, it's just a memorisation test. And so yeah, I totally agree. It's like if you can then bring to life and have real tangible experiences to talk about, then you see the character of someone. Yeah. And then you get the— that, that's the purpose of an interview, right? They kind of, they met the requirements to be in the room. It's more about a personality fit, a cultural fit, and you need those genuine signals to make those decisions, right?

Definitely. And I think it's very important to, to understand assessment performance into context because there could be 1,001 reasons why somebody has done something really, really badly or really, really well.

Yeah.

And sometimes it's a strike of luck, especially like in finance, you can say, oh, Yeah, let me make that bet, and that bet works, right? But you make it completely blind, and it's not repeatable. Whereas if you can show that you've made a very informed decision, even when you made a mistake, then you really showcase all your knowledge, all your thinking, all your planning, and all those really good things that employers are really interested in.

So one of the things that we have is we have the luxury of interacting with lots of different firms.

Yes.

These could be quantitative hedge funds, These could be boutique, more investment banking type firms. These could be asset managers.

Yeah.

So how do we take this underlying philosophy of what we do from the simulation assessments and apply it into these other unique kind of pockets of finance?

Yeah, so we have quite a good breadth of different type of assessments and simulations. So we cover all the major work tasks, and we keep on growing with more and more subtasks that we simulate so that we can, we can offer quite a good portfolio to the clients. Say, those are the things that are important to me, um, I'm hiring for these specific roles. For example, in trading, it could be commodity, it could be something, something different. We can have many different aspects that we change. And, and what we're building now in the product team is, is a series of features that will enable the client to really think about So for example, you take 10 different firms doing ostensibly the same thing, they will have different points of interest. They will have different styles in which they operate and a preference for certain skill sets rather than others or certain niche knowledge. And what we are building is a way to really understand what they need to measure. So in old psychometric term, like doing a job analysis, exists essentially, but we, we want to make it not, um, we don't want to make it a heavy process. We want to make it something quite light touch and innovative where the clients can talk to us, or in the future have their own self-serve facilities, uh, sorry, features, um, and they can really define what's important to them and what good looks like in their specific, um, in that specific segment and the team, and really focus on those things that matter for them. So for example, say that there's a firm that work with a lot of volatility in there around them. We can build simulation that recreates those challenges because let's face it, some traders, for example, will work in desks that have a lot more volatility than others. Is not just a quantitative difference in skills, it's a qualitative difference in skills. Who thrives and who completely fails in, in, in, in one environment rather than the other. And so what we're doing, we are, we are building really interesting partnerships with, with our clients to understand what are the nuances and the idiosyncrasies of each and any of them so that we can keep on refining our models iteratively to get sharper and sharper and sharper prediction of who the superstars are going to be. The way in which we're doing it is that we collaborate together into building our algorithms. So we do a lot of— we front-load a lot of work with the clients. We understand what they need, and we tweak our simulation as much as we can. And when I say tweaking, that doesn't mean that we do bespoke work for every, for every client. That couldn't be further away from the truth. We have a very good evidence base of what works, and we, and we really stand by that. What we're tweaking is the situational constraints and the, the changes in context that make each workplace unique. but all our core metrics will stay the same. It's just that, so if you think about it, we have a very sturdy and robust and validated 70%, maybe 80%, and then the 20% that is unique to a company that can be modified. And we keep on learning year after year, cohort after cohort, and making our prediction sharper and sharper. And this enables us also to continuously customize what we're looking for. So say, for example, something changes in the company. We know that we have been hiring and selecting for a very specific set of skills. Something changed. We know what we need to change. So it's highly configurable throughout. And, and also in terms of what metrics the, the clients are interested in, there's, there's many different ways in which you can present the metrics. And, and we're working with the clients to, to select which one of the best.

I was just thinking about what you said right at the beginning of the conversation. You were saying that a lot of these standardized tests that are used currently haven't changed for decades and decades and decades.

Yes.

But actually what you've just described there is the ability to be quite fluid, working with the client, adapting to conditions in the marketplace.

Yes.

Candidate changes of behavior, new technology changes that we're seeing with AI and other things that I'm sure will come in time.

Yes.

And actually we can evolve and learn. And as you said, it's a continuous iterative process to refine.

Yes, definitely. And I think it's important to stress the fact that there's an element of iteration and flexibility, but also an element of something that is really well validated, is really reliable, and we can really stand by. And that is not going to change, right? A good P&L is always going to be a good P&L. A good model is always going to be a good model. Good code is always going to be a good code. It's just that we can add and remove other things around it in a really, really fast and flexible way.

Okay, cool. So, you know, I found it so fascinating so far. I'm, you know, luckily I get to work with you every single day. But let's talk a little bit more about your, your product vision. Like, where are we heading in the 1, 2, 3 year kind of roadmap?

Yes.

Working on—

without giving too many spoilers, I'm working on, um, on defining all the architecture and all the features for the Agnostic ecosystem that incorporates, uh, all the journey, all the cycle of early career, from attraction, early training, selection, and then training and performance tracking on the job. So the reason why, why we're doing this is because we want to build a data pool of, of candidates and trainees and jobs and, and companies where we know a lot about all the different parties, and we can, we can suggest matches, we can suggest training solutions, we can, we can identify really early on who are going to be the superstars of the future and enabling our employers to spot them very early, making bets on them and, and help them train and thrive in this field. This is not an easy feat, right? Because it requires a whole platform that really cater to a number of users. So we have the students, we have the candidates, we have the employers, and employers come in many shapes and forms. There's a talent acquisition, there's a learning and development, there's a hiring manager, there's the coaches. And so there's a lot that this platform needs to do, and there's a lot of data that this platform needs to learn how to make those predictions and how to make those matches. We're really lucky. We have the right people to do this. We have people with a great understanding of finance, of course, and people who work every day with all the clients and every day with all the students. We have psychometricians, data scientists, and we just need to keep on building the platform and also collecting all the data in the right way and making models that keep on being more and more refined as, as we go. So essentially, the amplifier of the future is this, um, is this diagnostic ecosystem where universities, financial institutions, students, candidates can just come together and maximize, you know, all the hopes and dreams, whether it's to, uh, to learn more about finance to land their first job, to upskill and completely reskill in a new job, or to get, you know, get in front of the future superstars and interview them, train them, investing on existing employees. We want to be that one-stop shop that does all of that for finance. Yeah.

No, I mean, that's awesome when I hear you saying that because having been through the journey of Amplify and seen the company evolve, I don't think there's ever been a more exciting time to have you here working on this project. The challenges that our clients are facing, you know, the, the AI situation that's occurring right now.

Yes.

And how we can solve that problem, not just now, but going forward and how we can evolve and really change this, this approach that traditionally has been, you know, unchanged for a very long time. So yeah. Super excited. Thank you.

Absolutely.

Thank you so much for your time. I know how busy you and the team are.

Thank you.

But yeah, very much appreciate it. Thank you.

Absolutely my pleasure. And looking forward to more chats on this seat.

Yeah. Thank you.

Super.