How @Indeed is Building Agents without Opening a Browser

Joining us today is Oliver Boden, Senior Technical Product Manager at Indeed. Oliver, welcome.

Thank you for having me. Happy to be here.

Oh, I'm excited to hear about a little bit about Indeed and what you're doing over at Indeed with Agentforce.

Yes. So whenever we talk about Indeed, we talk about our mission.

Okay.

So our mission is very simple. We help people get jobs.

All right.

There's a lot of responsibility with that, with that mission. And the way we help is we have the leading marketplace connecting job seekers with employers, right? And when it comes to numbers, we have millions of job seekers applying and we're connecting those job seekers with employers together. And we— one of the key metrics that we track is hires per minute.

Mm.

And we just announced a big metric where we're doing 31 hires every minute globally.

Oh, wow.

It's a pretty big metric.

That is a pretty big metric.

So, whenever we think about building agents, we think about how can we build agents that help us improve that metric even further. Okay. Right? And to give you a sense of how we think about that in implementation, this diagram is how we built our ecosystem of agents. You can see we have Salesforce as a platform at the top. And then we think there's two key personas that we're working off to whenever we build these agents. One of this is internal-facing employees, which we call them employee agents, where we have two agents that we've been using for IT purposes and also for GTM teams. And we also have external-facing agents as well. We have an SDR agent, and we have a service agent, which is the one I'm going to be talking a little bit about today. We then also, we need flexibility when it comes to data, right? So we are able to leverage Data Cloud for certain data points. And whenever somebody posts a job and they're reaching out 5 seconds later to ask for a status, we have flexibility to use MuleSoft to go straight to the source. So we have those options that are very important for us. And then all the way to the left is the big shift that we've done when it comes to building agents, which we're leveraging coding agents. Without realizing it, we started using coding agents. And then we saw ourselves copy-pasting a lot of information to bring that context into the agent, so we— into the coding agent. So we work very closely with the product team to leverage these skills. You know, we were early adopter of the skills.

Right, right.

To then use that to make our— give our coding agents more autonomy when it comes to that build.

I would love to see how you're working with coding agents.

Yes, I can, I can show you. And let me give you a quick use case.

Okay.

So you can get a sense of, of, of a specific problem that we started fixing with our coding agents. So the service employer agent, um, has a couple of different sub-agents that, that we provide service for. There's this one use case where you think about employers, you know, they wanna, they wanna post a job quickly. They have to hire somebody.

Right.

We need to make sure that the company that's posting is a high-quality company.

Right.

So they have to go through a verification process.

Makes sense.

And then whenever the company is, is vetted, then we have to make sure the job has the right quality standards. So we also have to evaluate evaluate that job. When we launch our agent, we had a situation where we were trying to— somebody asked, hey, why is my job flagged? Which is a very common question that we got in our service channel. We want our agent to investigate the account, make sure the account looks good, and if the account looks good, then investigate the job.

Okay.

What we started seeing is that it was, whenever it found the account is active, instead of it going to the job, 75% of the time it was sticking with the account status and telling the user the account is active.

So it would say, hey, your account is whatever active, but it wouldn't even go to the next level and look at the job.

Correct. The job was flagged, so there was a lot of frustration as you can imagine, right? So for that use case, that's when an agent for script comes in.

Okay.

Because we can add that determinism. So I'm going to show you through our coding agent in Cursor how we can use these skills to then do that implementation. All right. Okay. So this is Cursor, which is the IDE that we use to run our coding agents. I'm going to call a skill that we built that I'll talk through in a little bit. I'm going to explain the issue as far as me running the skill. So, I can say, you can explain it the way you want it. So, you can say, when an employer reaches out about their job being flagged, the agent is saying their account is active, active. Instead of investigating the job. Can you fix this? Very simple.

Right.

Right. So at this point it's running its process using the skills from Salesforce, from Agentforce. And then we have this orchestration skills that are then controlling when do we call this? When do we make sure we understand, make sure the agent, the coding agent has context.

Right.

Because that's where a lot of the times the coding agents fail is that they don't, they start drifting. They don't understand the context and then they, start building quickly before understanding, which is very common with, with AI. So we add in— this is some tricks that people that are starting to work with coding agents— write your play— write playbooks and write them as files that then a skill can orchestrate, get the agent to read the file so you don't have to keep briefing the agent over and over.

Say that whole piece again, 'cause I think that's an amazing best practice.

And it's something—

write those playbooks.

Yeah. And it's, it's— we saw that issue a lot on our end. We saw ourselves briefing the agent over and over again. Then we realized, hey, instead of having to explain it all over again, why don't we create a playbook?

Yes.

Then we literally, it's just a Markdown MD file.

Right.

Through all the conversations we had in the past, we use AI to help us create the playbook.

I love that.

Then now we see the coding agent, when it's drifting, it goes to the playbook and we see it self-correct.

You have to make sure all the developers are using that playbook and not creating their own universal playbook.

Correct. Because that's happened a lot. You have the developers using, so that buying from an architectural standpoint is critical because you had to treat it like a new agent that you're maintaining. Right. So in this case, it takes a long time. It has to do some discovery work, pull metadata, do some baseline testing to make sure that the fix is going to work. It takes time. I ran that already here on a separate tab.

Almost like a cooking show. It's like, I already have the precooked one here, right here.

Exactly. Just make sure it's ready, and then you can see the history of it. And you can actually pull a report. And that's something we also did for our stakeholders. It exports a report that you can just share your stakeholders with all the go/no-go recommendations, right? Right. So this already ran, and I'm going to actually show you how that looks like in Agent 4 script. So this is the agent. What's new is these two variables here that you see at the bottom, employer status active and employer status inactive. So those were added as part of determinism to make sure that we're not accidentally going to the staying in the account status when we should be investigating the job. I'm going to give you a quick preview so you can see how that looks. I want to see that. So here's somebody, very typical question, why? Let me just make sure because we want to make sure this answer is personalized. Yes. So you can see it's set up to Oliver. This is because it's a builder environment. But obviously in production, we use OAuth tokens to identify users. So just, you see the Oliver Bodin set up as the variable, so it's going to be personalized. So let's go ahead and run a question. Like I said, very common question, why is my job flagged? Right? So at this point, AgentScript is going to be running through its process and you'll see it's much faster now because we're using determinism, where before it would take longer. And now instead of just staying with the account being active, it's now passing to the job and then providing recommended next steps, which is something that we started adding also. Not just about what is your current status, but what you can actually do next.

Next, I like that.

And then the agent is able to understand.

So not just answering questions, but taking actions.

Taking actions after, right? Because that's what employers want to do. Okay, understand, what do I do next, right? And then you can see here on the right, the different steps it took when it actually ran the deterministic approach. If you click on one of these, you'll be able to see the full trace and then do your troubleshooting there if you wanted to.

I love that. Now, that was great testing on one use case, one record. But how to ensure that this agent really is performing at par at the right level? How do you test at bulk and do it enterprise level?

That's a very good question because obviously this is unit testing. Right. A human can only do so much when you're human and you're testing manually. We have thousands of conversations happening every day. So how do you evaluate different flavors of the conversation? That's where you can go back to your coding skill and do some testing. So I'm going to show you how that looks. You can either start a new chat or use the same one. We actually baked in testing into our orchestration skill just to make it a little bit easier, but you can run it specifically and say something like, let me go back to the—

I think your coding agent is using MCP protocol to—

Yes. We use MCP to pull data from Jira tickets. OK. And then the skills use CLI to start pulling data from Salesforce.

Love it.

And it's happening all autonomously. And another tip is to do human-in-the-loop logging.

Yes.

So that we also have that being logged so that way we can go back and check, OK, where did the agent fail that the human had to correct it? And we use that to improve the agent afterwards.

And that's like a whole self-healing process.

A self-healing process. That's what we want to get to, to the point where the agent starts self-healing.

Yes.

We remove human-in-the-loop. Actions there. So now I'm going to show you real quick how to actually run a bulk test. Again, you can either part— make it part of your orchestration skill, or you can call it specifically as a testing agent for skill, which runs Testing Center behind the scenes. And you can say something like, bulk test the job status agent, um, and make sure you cover the scenarios of account being active and being inactive.

That's all you have to give it and it will come up with all the different tests.

Yes.

I remember when you would have to, as a developer, come up with every single test case and it would take weeks if not days, and now just like that.

Exactly right. Then we started, it comes up with its utterances and then you want to fine-tune them.

Right.

So we also have our library of utterances within there. Just like we have a playbook.

Yes.

We have baselines.

Another great best practice.

Yeah, because then it starts going and the And then the next time it'll be closer and closer to what you actually wanna test.

That's great. That really covers a lot of it. And the key thing here is like, okay, so we're doing our testing and you're testing at bulk. This is great. The next piece about that after you deploy, right?

Yes.

You then have to observe how are these agents performing? How are you observing their performance and tweaking it? And then you go right back again into the lifecycle. So how are you guys doing over there at Indeed?

Yeah, that's a very good question 'cause you can feel as prepared as you want when you do this.

It's all theory.

It's all theory. When you go to production, you're going to go, okay, it didn't go quite the way I wanted to, right? So our starting point is there are some manual evaluations you have to do at first, right? And we leverage observability.

Okay.

Agentforce observability. Latency is big for us. So we look at latency, we look at quality just to make sure that we don't have any big regression, right, in production. So there's some key metrics and we can see them by sub-agent. Or you can see it by intent, right? And then very here, if you scroll down here at the bottom, we also have SMEs per different sub-agent. So whether it's the account job status, which we call moderation, or the invoice, there's different SMEs. So when we have our weekly sessions with them, we can easily find chats about that topic, review it with them, and then they can help us write the tickets to improve the agent after.

Ah, okay. I love that. So once again, bringing the human in the loop there. That was a lot of goodness.

Yes.

I could sit here and talk to you all day, because I just love getting into the details, my inner developer coming out. So, thank you, Oliver. Don't go anywhere. We're going to come right back to you.