AI:AM — AI Safety Strategy, Cyberattack Resilience, and Workplace Agent Benchmarks · October 7, 2026

I would stock up on like 2 weeks of water, 2 weeks of food. The 2 things that I've been spending the most time on personally recently are what can or should be done around protecting critical infrastructure, especially from accidental agent swarms, and also things like malicious use of open weight models. There are some very scary scenarios where the electrical grid turns off.

We acquired a real company among all of these apps, often with gigabytes or terabytes of We create realistic tasks out of them. Actually showing model performance uplift on the data we produce is the best way to validate the quality of our data. We work at a company, we're not always just handing these tasks, handing a role and a work chart.

All right, first the news. AI agents composing in Reaper, a new bound on integer multiplication, and whether AI data centers could raise borrowing costs. Then Evan Miazono of Atlas Ignata on protecting the grid and the safety jobs nobody owns, and Edward Ho of Merckor on what real company records reveal about the gap between a task and a job. The hosts close on factory expertise, permission fatigue, and who is really paying for enterprise AI.

What a jam this morning. I was impressed.

I had some tokens left over from, uh, Fable. I gave it to Fable and, uh, it picked Maybe like 7 or 8 clips from like the last couple of years. It transcribed into Deepgram first, it cut it, and then it put together one version. And basically, I probably listened to about 9 versions of it. I gave it access to an API-based digital audio workstation, which is what you use to kind of compose various instruments together onto a track. It's called Reaper. It can play the instruments. figure out which instruments to play. If you need some samples, here are CC0 sites where you can download samples in order to use and go compose something. So it actually composed something with instruments. You can interact with it via Python, which means basically there's underlying the GUI, there's a command line interface there that you can use. And that allows the AI agents to use it. And I think the main thing from the first version was there's a lot of like, um, little glitches here and there. Um, and also, for example, there's a Terence Tao saying, it's just going too fast. And what I wanted it to do was after each, it's going too fast, I wanted it to speed up the tempo. There was a lot of like small little clips where it had included the first like quarter second of the next word. And so the sample was like a little bit like misshapen. And so that required me telling it, like, look, you accidentally included a little bit of sample, so can you figure out what the 2 voices are and disaggregate them? And so the machine was the one that came up with the sequencing. It chose which samples to use. I told it to put together a narrative.

I do think this goes right to the heart of one of the big questions that everybody's been talking about recently, which is simply, how good are these things going to get? In domains that are not automatically or readily verifiable. Obviously, I don't need to tell you that we've seen at this point just an overwhelming amount of math results in not just recent weeks, but last 24 hours. Pretty impressive and definitely strongly suggests to me that we are not going to be able to hide behind— Only works for verifiable reward type tasks for really any longer.

Let's get to the math results. In the last 24 hours, uh, what has happened is OpenAI has done the first drop of perhaps many, and they have dropped about 722, um, theorems. So they have a, a set of hard math problems, about 4,000 of them, actually about, what, 20-ish percent, uh, 722 of those problems. Uh, they were solved by A model running ChatGPT Pro, median time about 3 hours each. Some are very basic things. So for example, how do you multiply 2 integers together? Like what is the fastest way to multiply 2 integers together? For the longest time, there was a certain limit. That limit has been exceeded, but by a very small amount. But the key thing is that the barrier has been broken. And after the barrier has been broken, people are immediately getting better results. So already in the last like 14, 15 hours, People have improved on those numbers using the techniques there because the techniques were not fully milked or humans are optimizing as we speak.

One other thing I wanted to just pull up real quick is a pretty striking chart. Where is AI in the macro statistics today? And look at the movement in the US 30-year bond over just the last year. I mean, this has not moved that much in a long time. And now we are all the way back up to Approaching 6% in this month alone. It's jumped by a full half a percent. It suggests that obviously the scale of the data center buildout is getting so big that it's pretty hard for it not to have an effect on just overall global capital measures. And the ROI on these data centers is so high right now. The 2 Patels, Dwarkesh and Dylan, had a podcast not too long ago, where they talked about this. And the basic upshot was like the pressure that the extreme returns to data centers are generating right now is going to crowd out a ton of other stuff. Right now with a new data center, given the prices that they're paying and the margins that they have, the frontier companies can basically make their money back on a new data center buildout in roughly a year. And then they've got 4+ years of useful life after that where they're just making unbelievable bank. And we're seeing that for US Treasuries to compete with it, the rate has to go up quite a bit. This is going to create borrowing problems for all kinds of other entities. The old Piketty idea that people who have money make more money and that returns to capital are just going to continue, have gone up and will continue to go up. And you just get bigger and bigger pools of money that way. I mean, if you're sitting on a lot of cash, all of a sudden you're getting like close to 6% in what is in theory risk-free return. Obviously there's no real risk-free return, but this is as close as it gets and the premium is moving that much.

So A couple of thoughts. Let's say it goes to 10%. Something's got to give. Being a former distressed credit trader, what ends up happening is bankruptcies. And especially in the United States, it's not the end of the world. It's actually you take a stay on paying anything and you restructure to something that works for your level of profitability. So let's say if you were servicing $3 billion of debt, With $100 million, you write that down to like $1 billion, and then you're fine. The people who hold the capital take losses. And so if you were a bond investor, you were now able to invest in these bonds that are giving you 10%. On the other hand, your bonds that you bought like last year, which are giving you like 2%, someone, when they try to refinance, they default, and then you lose some of the capital. Companies change hands.

If an AI swarm takes the grid down, Who is responsible for proving what happened and stopping it? Evan Miazono's nonprofit, housed inside Convergent Research, exists to find exactly these jobs nobody owns and to recruit the people who will take them.

Evan Miazono is the founder and CEO of Atlas Ignota, the successor to Atlas Computing. Atlas's early work focused on using AI to help produce software with mathematically checkable guarantees. The team recruited and supported Jason Gross in experiments that helped lead to Theorem Labs, which works on AI-assisted software verification. Atlas also helped initiate OATH Technologies, led by Mike Dodds, to build tools for formal specifications, and Irondale, led by Mehmet Sekand, to develop hardware that responds to physical tampering. Earlier, he completed a Caltech PhD working on optical quantum memory. Evan, welcome to the show.

Thank you for having me. Excited to be here.

How do you select the important problems?

If a donor comes to us and says, we care very, very deeply about what happens with cybersecurity if we kick off recursive self-improvement in 6 months, then I can go and look at what seems most catastrophic There are some very scary scenarios where the electrical grid turns off and maybe doesn't turn back on in most of California. And so that seems like something that deserves some attention. There are many, many people who are thinking about that problem and related problems. Many of those people don't necessarily have the context and resources to fix all of the problems they come across. And so having an organisation out there, ideally there would be multiple organisations where someone can bring, oh, hey, I know about this problem, but I'm not in a position to solve it. Or, oh, you mentioned this problem. I actually have a solution that seems viable, but I can't pursue it. Or someone saying, I have resources and I care very deeply about this problem. I've been claiming repeatedly that as intelligence gets cheap, It's the coordination that gets expensive. So having shelling points around who does this coordination, becoming one of those shelling points or creating one, seems very useful. Is there a funder who's going to prospectively or retrospectively fund projects and efforts on this idea where the retrospective is like, if I produce a team that can execute but they need $10 million, if I can figure out a priori that no one is going to spend that money, then that isn't a project worthwhile.

You have kind of taken a step back, restructured your organisation, relaunched. Tell us about that process, what caused it.

The challenge became that while I was trying to build prototypes and things like that, people kept saying, hey, there's a clear need for someone to do X. No one is doing it. And I say, well, I can go find someone to do that. Let me make sure that this is the right thing to do first. Maybe get them the experts they need, the resources they need. And I did this 3 times. I brought in a chief of staff who helped me transition from Atlas Computing to Atlas Ignota. And he said, what is your unfair advantage? Why aren't you just doing more of that? There are many aspects of the world in which people will say strategically, I want the world to look like this. And then completely separately, they'll look at what they could do and then pick the obvious thing in front of them rather than trying to back-solve from the world they're trying to get to. I think a lot of people are tactical when they think they're being strategic. Realizing that there is actually this big failure mode for a lot of the philanthropic money that's moving into the AI resilience space generally, where If you put up an RFP, everyone will take the problem that they care very deeply about, their passion project, and then wrap it in the language of the problem you post and then submit that. Same for requests for startups, same for government BAAs, things like this. And the question becomes, who is out there actually trying to define, here are all of the possible options, here are the best ways to achieve that. And then who do I hand those off to? The mechanisms we have around markets don't do a good job of allocating resources in a way where there's no value capture.

How do you narrow down what to work on?

I found that if you say, here's a problem, and you go talk to experts and you say, what can be done about this? You will get a lot of ideas. You will also get a lot of bad ideas. And it is very easy for experts to oversimplify subsets of civilisation that they don't have deep familiarity with and make assumptions about what is adoptable or what isn't in ways that I think are very important to whether or not an intervention is useful, feasible, et cetera. And so Having people who can very quickly use their model of how big organizations make decisions, how technology actually gets adopted and things like this, and being able to quickly test out these ideas is very useful. I also have spent a lot of time thinking about formal verification and all its various uses and applications. And I think that there are some potentially exciting directions where the kinds of things that you would want around To the extent that I agree that government is very likely to get involved in a lot of these things, we might want a certain set of models that only can be run if the companies that are running them can provide strong evidence that they are applying one of a certain set of evals or classifiers on those.

Let's talk about just some of the big gaps. What are the things that right now you think are Highest on this? I guess maybe you score them by sort of a product of importance and neglectedness is kind of the takeaway that I am—

Yeah, the timeliness. Is there something that can be done in time? The 2 things that I've been spending the most time on personally recently are what can or should be done around protecting critical infrastructure, especially from accidental agent swarms, and also things like Malicious use of open weight models. I think there's an interesting question of what happens if instead of OpenAI accidentally attacking Hugging Face, we have the companies behind DeepSeek or KIMI models accidentally attacking State Department servers. How do you know if that's an accident? What do you do in response? What would we do if we saw some malicious attacks to critical infrastructure coming from an OpenAI server? How would you know that it was this and not coming from somewhere else? How would you shut it down quickly? Very easy to come up with ideas for things that one could do, and much harder to find people who can say, like, yes, that is absolutely feasible, or it isn't.

The internet's directory, DNS, turns a name into an address. But nothing in it tells a defender who is running the model behind that address, or whose agent is knocking. So what would it take to make an agent prove who sent it?

Um, and one of the interventions you might add is, could you use DNS to have cryptographic attestation of whoever's providing inference? I think this would be very useful and also a great complement to having DNS for people. So if your agent is talking to my agent, they can verify that this agent in fact is my agent. I think that there are some shortcuts to adoption that you can take if you're not trying to build this into a startup that returns the fund. And so this looks much more like a public good.

There's a lot of tensions between privacy and identity. We've had this kind of back and forth about whether you should have KYC to purchase data center capacity, for example. It sounds as though the cryptographic identity of agents is kind of a follow-on to KYC for data centers.

Would you say that is important to note that I am in conversations ranging from stricter monetary controls and financial controls on agents look like so that we can prevent agents running, creating and running a shadow economy on the one hand, and also risks on the other hand of How do we have identity that would be useful for things like this, but still can also support refugees and dissidents in countries where they need to be able to verify certain aspects of who they are, where they are, et cetera, without making them increasingly at risk? Journalists in hostile territories. I think that there are zero-knowledge ways of doing various identity-providing schemes. This would benefit massively from a fairly deep market where if one company is requiring that I give my name and address and all sorts of birthday and all sorts of information, there would be a different service that would accept a zero-knowledge proof that I am simply living in California and am an accredited investor, for instance. And so I think that the cost of making those attestations with zero-knowledge proofs and things like this is dramatically lowering. If I'm sending my agent out, then I could have an allow list that is like, only reveal that you are Evan Miazono's agent to people who are already in my contacts. I think there are even protocols you could develop of, I generate a hash of my contacts list and could prove to someone that you are in my contacts list without revealing what my contacts list is. And so I think that there are ways that these things can be done. It is a non-trivial amount of engineering. It might be costly. But one of the roles I see Atlas taking is, to the extent that privacy is a public good, it might be something where we could make progress on what that privacy-preserving identity protocol is, So that if we hand it to relevant companies and say, hey, just adopt this, it's actually cheaper for them to adopt the thing that is better for privacy than building their own thing that has fewer features, is less complex because we've been able to leverage technical expertise, consensus building, and philanthropic capital to build out this prototype and this spec.

Do you have protocols or would you point to protocols that you think I should be Piloting now in my own personal use.

One thing that I have is I have an application that just takes screenshots of anything that's not a video conference and runs it through a local OCR model and sends it with my weekly goals to say, what am I trying to do? What should I be trying to do? What should I automate? What should I do differently? And I could imagine this being a thing where Sharing that across collaborators would be something that could identify all sorts of interesting synergies. I think there are a lot of unknowns that would be very valuable to capture and communicate, but also very possibly risky to communicate. I think this is a public repo. If you go to my GitHub, it's Coachee. I haven't had a security audit on this. There are definitely some other agents that I have that run routinely. They are not public because prompt injections aren't solved yet. And I think that the more you know what someone's agents are exposed to in terms of inbound, the more vulnerable to prompt injection you are. And I lean on auto mode way too much.

But I'd be remiss if I didn't use a little bit of the time we have left to ask just for kind of an update on Where are we on formal methods, formal verification today, on compute governance? Obviously, you can't tell us everything that has gone on, but what are kind of the most important headlines or recent developments? Obviously, models are getting real good at this kind of thing.

There are some meaningful differences between proving theorems in math and proving properties about software. One of them is that in math, the number of Theorems that one might care about are a lot fewer than in software and they're a lot more universal. I'm thinking of Oath and Theorem and a couple of others where they're taking on things that would've been 5-year, $100 million efforts previously and are now things where you can possibly It's limited by tokens and the number of people who can successfully, effectively wrangle agents to do these things. It's useful to have infrastructure for software where you've defined what the programming language is, for instance, formally, where you maybe define properties of the compiler or the operating system. And so you need to build up all of this infrastructure, whereas with math, especially with Lean, You have the axioms well-defined. You probably have some of the theorems you want to rely on. You can have an AI system grind on those to build out the proof tree. Doing that in software will probably look like, on some level, having to formalize aspects of what does Linux do and how you know that is correct, whether or not you just clone it and say, We're going to assume this is correct. It's at least an improvement. That is one avenue. Another one could be you try to prove things about it that Claude thinks are the right things to prove, and maybe you find bugs because you can't prove the things. There are also a lot of existing tools that are much more efficient from an energy, time, token perspective in the formal methods community. And so there's a lot of academic research about how to integrate those tools into more agent-first grinding type approaches. There's a lot of things here that I would expect that in 3 to 6 months, we're seeing a kind of shock and awe that has been happening with math in the last month happening in verification of software improvements to stability. You could also prove that we won't have— like, this system won't crash, or if it does, like, if some random event leads to it crashing, we can recover everything. We can roll back any software execution.

I want to ask a little bit about your view of the future, because I think, as you alluded to, you have certain timescales on which you think things should operate, and you have a certain viewpoint of how the future will unfold. What does your future in the 1 to 5 year timeframe look like?

I have a lot of uncertainty. I think that I try not to resolve that uncertainty into something that is highly confident. Unless it seems like there's an intervention where I'm placing concentration of power risk against loss of control risk, for instance. Those instances aren't super common. And I hope that we as Atlas are going to start adding more of these things publicly, but our model has roughly been, what is foreseeable or forecasted in the future, particularly by people who forecast AI capabilities well or global trends well? Some of these futures are catastrophic cyberattack. leads to things that someone could prepare for. I've been telling my friends, you should probably stock up on 2 weeks of water, 2 weeks of food. This is a useful thing that's pretty low cost if you have the space for it and would probably last long enough. If something goes very wrong, you can rely on that until aid arrives, if aid is coming. There are scenarios where aid is not coming, and in those scenarios, it's unclear. It becomes very expensive to prepare for those, so I don't prepare for those personally.

Atlas's method does not call a problem solved when experts like the answer. Before a handoff, it asks for conditional commitments from the people who would use it, and maps who has to say yes. Long pause. For a guardrail on every model, who is that?

Can we get conditional commitments? If we build this out, will it actually get adopted and used and have the impact we intend? We've started building out our synthesised forecast of what possible futures look like, what things we might expect. The next step is what are defensive equilibria we as a society could try to reach around things like infrastructure security or privacy or interpretability or alignment or things like this? What do those look like as cohesive potential worlds? Which is separate from what is the intervention or list of interventions that could lead the world to be kind of on track with high confidence towards that. I could imagine that if we wanted technology that said one of these guardrails is definitely happening on every model, it seems plausible that the new AI czar or Trump saying, yes, this is going to happen, makes everyone believe, yeah, okay, this is definitely going to happen. But it also feels like if Jensen Huang said, we'll definitely implement this, he might also be sufficient. You could imagine a sufficient quorum of Congress also being sufficient to make this happen. And so trying to identify who those adoption quorums are and what it would take for them to make the decision earlier and more easily becomes stuff that I think it's good to have Uncertainty around. Also think that a lot of times standard setting can be very slow because you want and need consensus, and I am very wary of the timescales for the problems that we are looking at now as being amenable to that level of consensus. So I'm, I'm interested in all sorts of creative solutions to try to get the best possible things adopted on Frontier Labs buy much of their expert training data and many of their professional work tests from one company.

And the man who now trains models there has to keep those tests honest. So what separates passing a professional benchmark from holding the job?

Our next guest is Edward Hu. Edward Hu is head of AI modeling at Merckor, where he leads model training and research. Merkur works with professionals to create training data and evaluations for AI systems. Before graduate school, Edward worked at Microsoft, where he led the development of LoRA, short for Low-Rank Adaptation. He subsequently worked at OpenAI on O-1. Edward, welcome to the show.

Hey, nice to meet you guys. Nathan, Prakash, thank you for having me on the show.

What are you working on at Merkur right now?

Merkur started out being this powerhouse for data. And increasingly we realize as we serve more and more sophisticated research customers, as we push the frontier ourselves on what is the best type of data to improve these models, it's increasingly important for us to actually be not just on the frontier, but actually pushing the frontier of research when it comes to data. And on a high level, our research focuses on benchmarking, How do we actually evaluate the frontier of model performance when it comes to economically valuable work? What does it mean for an AI to be economically valuable in various professional domains? What does it mean for an AI to do better? Those are the questions we're answering. We're putting out frontier benchmarks like APEX Agents or APEX Suite in collaboration with Cognition, APEX Accounting in collaboration with RAMP, and to address these questions. And then In addition, we're also training models now on the data we produce because ultimately actually showing model performance uplift on the data we produce is the best way to validate the quality of our data.

So obviously you have this APEX family of benchmarks that you just mentioned. How do those compare to other attempts to measure economically valuable work? We've got GDPval, Artificial Analysis has a version of that that they maintain. There's the Epoch Capabilities Index. Obviously, we know we can't get everything from benchmarks, but what's the fastest way to zoom in, understand what the different benchmarks are really telling us and how we can understand, for example, like a Gemini 4, incredible benchmark scores. How should we translate that into our understanding and expectations?

And I think in the last couple of years, we also went through this transition from benchmarks more focused on academic capabilities, say mathematics or coding, and gradually moving towards, okay, how can we actually evaluate what the model will do when it's deployed? I would say nowadays, for a benchmark to actually be useful in evaluating a model's economic impact, it needs to really be situated in a realistic setting, and also it needs to have enough coverage over professional domain. And often the best way to do that is to work closely with people who are experts in those domains, who have done these tasks, and also with enterprises who really cares about these settings and their entire focus is on doing well in those particular domains. And our APAC series of benchmark is laser-focused on how can we leverage the expert network they have to cover the domains in a way that is actually mimicking how real people perform work. We really focus on realism starting with APEX Agents, which we put out earlier this year. There we actually recreated an environment that a professional will work in. And at the time, this was the first of its kind where the model actually have access to a whole legal data, for example, with dozens of files organized in the file system, email server, chat server, and the model needs to actually navigate it. find the right information, put it together before they can actually do the task.

The public Apex Agents test holds 480 tasks in investment banking, consulting, and corporate law, each set inside a company that experts assembled file by file. What does Merkur get from a company that actually existed, records and all?

And our, uh, we have a project that we are close to wrapping up where We're actually taking it a step further where we acquired real companies. Often they have realistic apps and environments as a part of the data we have. For example, their Salesforce data, Slack data, Figma data. And then among all of these apps, often with gigabytes or terabytes of data, we create realistic tasks out of them. And there's, there's no— you can't get more real. when it comes to professional work with these type of data, and that is increasingly the direction we're taking.

So let's talk a little bit about maybe a concrete example. Merckxor bid for the Spirit Airlines dataset. So Spirit Airlines was a bankrupt airline company, and I believe the entire dataset was up for sale. Merckxor put a bid in, Google put a bid in. I believe Google won the bid. But if you had such a dataset from a bankrupt company, And you had the entire dataset, as you alluded to, you create this environment where you have real people taking actions and you get the agents in there. What is the first task that would happen? What is the input and output for every agent that you're simulating in that environment?

Often when we get these acquired datasets, it's very overwhelming. It has everything. Finding tasks that actually are really good to evaluate AI is a non-trivial problem. And frankly for the company and our research is follow what is economically valuable. That's also why we have an econ research team. We're collaborating with economists across the industry and academia as well to really understand what are the most economically valuable tasks that AI is going to be touching. So often those will determine once we have a dataset like from an acquired company, where do we go first? Often a lot of these tasks are legal, HR, or finance in nature. And we would go in these areas, find the relevant information, find the right slice, and create tasks with the guidance of our expert contextualized in the actual problems that these businesses encountered.

So perhaps one would be, hey, you know, how did this company allocate its crew roster, for example, which is economically valuable? And you have an output, which is all of the crew rostering decisions made, and then you have an input, which is the series of logical decisions that led to that particular crew roster. Would that be a way of describing kind of the inputs and outputs to that kind of decision-making?

So that would be a particular task grounded in the airline context. The shape is actually quite similar to when having acquired data, just instead of having, you know, 3 or 4 apps, we might have dozens of them or even hundreds of them in some cases. And then there, the agent will still have access to all of them. It'll need to decide what apps to query, what information to pull, So far in the field, we more or less have been really task-centric. We often start with, okay, here is a task that you need to finish. Maybe it is to generate a roster. Maybe it is to build a financial model. Maybe it is to solve an HR problem. You'll go out, find the right information, do some reasoning. Maybe you need to find more information and then you produce an output. Is there a better shape? When we work at a company, We're not always just handing these tasks, handing a role and a work chart. And of course, all the data in the company. And then there'll be people coming to us, asking things from us. There will be people we need to chase down to get information from. There will be managers, people in charge of resources, conflicts to resolve. So increasingly that we believe is the shape of the future and we'll have more to share in the coming months.

So this role question is super interesting and There's also, of course, the big question of how is it that we have Fields medalists before we have widely deployed customer service agents? So it is starting to happen, but it's happening a lot less than it seems like it could happen. Do you think those are just about pace of diffusion, friction of the sort of human decision-making variety, Or would you still zero in on concrete issues that you think even the best models have that prevent them from being deployed for just rank and file customer service, for example? And if there are such gaps, how would you characterise them? And what do you think the path is to solving them?

There's the technological problem of can the model do the job? And for math, the answer is yes. And for customer service, with the best model, the company with the best customer service technology gets the job done. And I think we're not fully there, but we're getting close. And there's the human organizational or societal problem of how this technology diffuses and actually gets implemented. When I was working at OpenAI, mathematics was one of the domains we focused on. Once we have produced the proof, it's relatively easy to check it. Same thing with many coding tasks. Once we produce an answer, it's much easier to check it than to create an answer in the first place. Whenever a task has these characteristics— board games is a classic example where, like, in a game of Go or chess, it's easy to know whether someone has won or not, and the rules are easy to encapsulate, but finding the move is hard. Whenever we have tasks with those characteristics, computers, when We spend a lot of compute and having good heuristics and training good models to do the search problem in this big space that can be hard for a human to cover. Computers tend to do a lot better, and mathematics is a strong example of that. Whereas for customer service, often it's less of a search problem, but it's actually having a nuanced understanding of what does the user want, what is the outcome that we should navigate to, what the businesses can offer. It becomes a lot messier. And while humans can have some subjective judgment about, has this been a good customer experience? Has this been a good call? Often it's not black and white. Whenever we need to make these kind of judgments, whenever it is hard to judge what is better, it just takes more time for computers to gain these capabilities. And especially when we define what is better in a way that is more vague or subjective. Often what we run into is the model will look at the reward function and do its best to maximize it and do what's called reward hacking. In the end, actually deviate from what we intended. As proving a math theorem, you can just do it once and everyone in the world will agree you have done it.

You mentioned RL there. Obviously, major driver of progress in math. I guess I'd love to hear your thoughts on kind of the future of how we're going to close this gap. One answer that we've heard from folks like forward deployed engineers from OpenAI is basically like, look, the models are good enough now. You just need to give them better context. And you can do that by just deploying, iterating, having some people that you trust in your company come review things, And kind of give feedback. And that feedback can really just live as text files. And you give feedback a few times, you'll clean up the vast majority of cases pretty quickly. You're hill climbing not at the model level, but at essentially the harness or the context engineering level. Another story is the models themselves have to get better. I guess, at what level do you think people should be focused on solving these problems?

When we speak with our customers, especially the enterprises that are Thinking about the exact question you mentioned, right? Should we just take a model, adjust the harness, build some skills files, and call it a day? Would it actually clear up majority of cases? Or do I actually want to do model training? Then when I do model training, how do I actually do it? This is model customization. Within customization, starting with harness is often the best bet. Within a harness, that includes your prompt, the tools you have access to, skills files. Then if that is not sufficient, Then one can move into post-training, and within post-training there's SFT, there's RL, and orthogonal to that. There's no single answer as to we need RL versus we don't need RL. It's very domain dependent. If the task is generally within model's capability, perhaps one don't need to do any additional parameter updates. And the only way to know for individual domains is to really have trustworthy evals where we can test how far we can go with adjusting the harness and how far we actually need it to be for this to be served in production. And we've seen very different results across different customer segments, and especially because for many customers latency is actually a key consideration. And one of our customers mentioned that actually taking a frontier model with reasoning is just too slow for them. And it's much better. Either they'll have to take a frontier model without reasoning or a small open model with reasoning. And then comparing the two, actually the small open model does better. Having an eval is the right way to see which level of customization is needed.

When it comes to the future of training, how do you see this SFT versus RL I don't know if it's a divide, but maybe a balance or ratio shaping up over time.

I do think in the future, RL, the way it is done today with these very elaborate infrastructure, very expensive rollouts and relatively inefficient updates, because with all of these rollouts, especially a long rollout, we only get one number at the end. These type of training will not be as widespread. I don't really see every company in the world doing these big RL runs. So I do think SFT, especially combining the output of multiple teachers, will be a key part of how enterprises in the future customize their models. And we're investing in research on that as well.

Let me segue a little bit to multi-agent training. How does the data preparation aspect change, or the task preparation aspect change when you're trying to do multi-agent training?

Often for multi-agent training, it's less about the task itself. For some tasks, it is quite natural to have a multi-agent setup. For some others, it's a little less so. Often the trade-off to think of is when you have multiple agents, what you gain is speed. And from literature that I've seen so far, multi-agent training is usually less efficient when it comes to compute, but it does give you speedup, often something like we spent 4x the compute to gain twice the speedup. And in our example, when we grounded in professional work, there's a lot of scenarios where multi-agent comes in handy when we are maybe building a model, writing a report, in the financial context, in the legal context, and there the task more or less stay the same, but we will make sure the harness is set up that the agent can have the liberty to define subtasks and call the appropriate tools to create subagents to find the right information and combine them. So often it is just making sure that our setup, frankly, on the infrastructure level can support the level of concurrency And being able to support multiple agents traversing the file space efficiently. And it's less so about the task itself.

I have a question just on RL environments and the state of them right now. Obviously, we've seen what happens when some percentage of RL environments are not very well built and end up rewarding Cheating in various ways. What is going on at Merckor and in the industry broadly right now around cleaning that up? How are you approaching it? And what are the buyers demanding?

If an ROE environment has a hackable reward, you can't really trust the score. And of course, performance wouldn't actually generalize to real performance. So we take that extremely seriously. We were hacking tends to be more of a problem when the task is set up in a way that is rather— either it's too hard, it's too underspecified, and the model just doesn't quite have a quote-unquote legitimate way to solve the task. So we are doing several things to address that. Number one is to make sure our tasks are actually well-specified, is actually solvable, meaning that a reasonable human given enough time with the help of AI potentially can actually find a path that solves the problem in a way that the requester is intended. And then also making sure that at the rubric level, we are catching the undesirable behaviors as they come up. And we just need to really see how the model performs on a given task to see what are the undesirable behaviors to catch. One example in this dataset Apex Agents would put out which is quite interesting. We have a recent blog about it where we have these professional tasks where the agent is asked to produce a financial model and we might grade it by checking, does the financial model include this particular answer, which we know is the correct answer? What the model will do is, given certain ambiguity, and in many cases it's unasked, Initially for not including all these specific parameters, the model would guess these different parameters and ended up giving many, many different answers. And we called it scattergunning, where the model would just say, oh, if this is true, then the answer is that. If that is true, the answer is that. That is not how a human would actually handle it in a realistic case. A human would ask for clarification. A human would follow industry standard. But in this case, just because the rubric is rewarding the inclusion of a single answer, the model ended up giving many answers, up to 10 or a dozen answers, and it'll get the point if it hit one of them right. So we did a re-release of the benchmark, we call it APEX 1.1, where we number one made sure that tasks are all well specified because that is in a way the source of the issue. Number 2, we have rubrics that actually penalize this particular behavior. And we're constantly running tests, examining trajectories, and making sure whenever the model scores well on our benchmark, it is doing it in a way that actually translates to how enterprises, how our customers see work being done by their AI agents in their production environment.

So speaking of research done by Merckor, I saw a post on accounting recently, I think in the last several days. And I think the study compared 12 licensed CPAs on certain accounting tasks, and it reported that Opus 5 received full credit on all 20 attempts versus roughly 37% for the licensed CPAs. What portion is going to be still done by the human? Like, if you were redesigning human accounting, what should the humans be looking at?

We were surprised by the result initially as well. The result is not to say that AI is going to take all of human accountants' work, and it's really for the specific task with a slice of accounting work that we have created, especially the very— —intensive and retrieval-intensive tasks where the model needs to crunch a lot of numbers— —going over a huge volume of files and do these very long chain of calculations. For these tasks, and given junior accountants without AI tools with a fixed amount of time, AI is doing better than them in a small portion of accountants' work. Earlier we talked about often what an AI is being evaluated today, including in this benchmark we call Apex Accounting. There AI is handed a very well-defined task with a large file system, and the model needs to go find things and solve the task. But often a real accountant is put in this role of an accountant and they have stakeholders and they have often to figure out what is to be done and how do we handle relationships? How do we navigate timelines and different blockers? And maybe I need to get some information and build trust before I can get this account going. And all of these things, AI today are not only not doing much and we don't even know what is the right way to evaluate that. Anything in accounting that is outside of Here's a very well-defined task, go find the numbers and execute them. These AIs are still not very good at tackling right now.

Something I heard at an in-person event this weekend was basically that the Frontier Lab unreleased models, let's say, maybe even to some degree the released models, are A, showing superhuman performance on anything they care about, and B, that in some of these domains, human data is not really helping anymore because the models have kind of hit the human expert level. I guess, first of all, do you buy those claims? Would you contest them?

There are absolutely domains where a human is not contributing to the frontier anymore. Whenever we have a domain where It's quite open-ended, but there's a very, very clear goal as to what is better, what is a job well done. These are domains where we see this behavior happening. One example would be kernel optimization, where the goal is to write a kernel that runs faster on a particular hardware. The model can go in and write code that expert humans might not even understand. But in the end, as long as the setup is not hackable, and often we run hardware that's very hard to hack, the model will be able to achieve these superhuman experience or superhuman performance, if not already. And we also see similar things happening in mathematics where you see that kind of characteristics as well. Whereas for domains where Human judgment remains quite important, where often human taste, it's just really hard to encapsulate in a number that we can say, okay, this is clearly better than the other. There, the bottleneck to improvement is still human, as we see today. How clearly can we specify for a given domain what it means to be better? And can we specify that in a function that is cheap to evaluate, That is uncontested. And then if we can do so, when we put in a lot of compute, even the existing algorithms are quite efficient at finding solutions that satisfy what it means to be better. But a lot of the economy still today are not quite there yet when it comes to our ability to specify what is better.

Last one for me, just on the mechanics of collecting this data. I am increasingly insulted that nobody has made me an offer I can't refuse to install tracking software on my computer and just watch me work all the time. Is that a big part of the future? Is literally just installing, watching, and getting keystroke by keystroke, click by click data. To the degree that that is the case, how are organizations responding?

And this is not just a technical problem. And it's quite often organizational, political, and also personal, right, to all of us. Nathan, if someone were to make you an offer, I imagine an offer you can't refuse to record everything you do on your computer, that's probably quite a substantial offer. And I think for everybody, this is a decision not to be made lightly, whether we would want something like that for everything we do. From a purely technical standpoint, If we have all the traces of how we do everyday work and we collect a lot of them, and as a technologist, that will be very interesting data and training a model on that. I think the model will be quite capable in ways that current models are not today. But then there is the organizational, political, and, you know, like I said, personal question of, is that the right way to go about it? But at the end of the day, we want to move society forward in a way that makes sure everybody wins. And moving society forward doesn't always align with building the best technology, however. So in this case, I think it's a question for every organization and for all of us to ponder. What is the right way to build this technology? And often we found it's helpful for many organizations when navigating these dynamics to really test the water and starting building the foundation of this technology using a flexible contracting workforce, for example, like what Mercor provides, and that collects the data in a way that is less invasive because there are people who prefer the flexible work that Mercor provides, and of course that also doesn't leaks the proprietary data that people might be handling day to day.

On your forward planning, what kind of data do you think you want in the next 12 months?

I would say the data we are particularly interested in are real environments where people are performing professional, really economically valuable professional work at scale, where— and often where It's not black and white. There's no right or wrong, but there's a lot of organizational dynamic of people coordinating and maybe workshopping what is a good concept, what is a good idea, what is a good design, what does it mean for humans to actually collaborate with each other in a very functional way, and how conflicts are resolved. And I think part of our goal to have more such data is just to understand What does it even mean to evaluate how a model fits into our work environments? We don't want AI to come in and replace people. We want AI to be valuable coworkers that we can all leverage and do higher-level work. And what it means for an AI to work in an organization is such an understudied problem. And I think having data, seeing how a real organization operates and evolves and And especially with the injection of AI will be extremely valuable. And privacy, the future of work, these are all tricky questions that we all need to navigate. And to me, it's just not a technical problem that we can say mathematically this is the right answer.

A lot of the physical tasks, you might not find digitization of tacit knowledge inside a foundry, a steel foundry or something like that. Penetration of software is quite high in the airline industry because 'Cause you had all of these like booking systems and optimization. But penetration of software in a lot of manufacturing or construction is actually very low. So it's still a printout and then people go and carry it out. So I think to the extent that something is penetrated by software and you can get all of the data, then that's useful because then you can basically migrate a lot of that to AI. But if the industry has not been that penetrated by software, it's still a long way to go. AI is great. It can do everything a digital human can do, but there's a lot of things that digital humans can't do. There's a lot of field reporting. Right. The classic field reporter, Analyst at a bank flies out to a manufacturing company in the Midwest, looks around, sees some trash on the floor, reports back, this manufacturing company is not that well maintained, gets a downgrade in their analyst reports, et cetera, et cetera. All of that is tacit knowledge built up from field reporting of various kinds that investors, journalists, regulators do. It's still clipboard and someone ticking off Checkboxes and visual inspection. And the guys who do these visual inspections are very good at it, because if you've been doing it for 10 years, you can walk around and you can be like, all right, I know exactly what's wrong with the place. And that tacit knowledge, I think, is going to be very hard for AI to absorb.

Do you feel like right now, the ones that we have, have crossed such critical thresholds for ease of deployment and ability to help you kind of rework your process where needed? that as that dawns on people over the coming months, you might see a dramatic acceleration in use. Because even just the beginning of this year, when I think back to January when I was really getting serious about personal agent setup, it was a grind. There was a lot of mistakes still and just things that it wasn't quite getting right. I wasn't able to really hand off At a high level, conceptually, like, here's kind of what I want. You make it happen. This deep context database and a bunch of content is embedded, and that's helpful for search sometimes when keyword search doesn't work. Basically, now I'm just like, please go look at all the new embedding models, read up on them, see how they compare, look at the prices, look at our options, make a benchmark based on our own use cases, and then test all these things and give me a report. And it can even go do the computer use to sign up with new products I haven't even used before. That is like a multiple faster than it would've been at the beginning of the year when I felt like I had to be the project manager still.

I think the capability threshold has been crossed, but I think the price and the latency is not there. I think what's happening inside the Frontier Labs is that the Frontier Labs are seeing the full effect with the latency required to make it worth their while to run like 20 agents and be able to like work in flow with them. And I think outside the labs, unless you are in an organization which is willing to spend like $100 grand, you know, $100 grand a month, you know, in expenses, you're not, you're not really seeing that yet. And I think we will, we will get there. But it's going to take like, you know, 12 months or so.

Kind of honestly, though, was thinking somewhat the opposite direction this morning, because 5Five is repeatedly returning reports and proposals where I'm like, that was only like 2 minutes. Like, how did you do that that fast? I think it's fanning out and doing research a little bit more efficiently is maybe one aspect of it. You can spend more tokens to get faster, and subagents definitely help with that, especially if things are parallelizable like research. Over the last few days, my personal agent console— hey, can you add voice mode? Can you add better search? So when I'm searching through old threads, I've got different angles on search features that would be like, a week, you know, for a person to map out, figure out what tools they want to use, code the whole thing up. They're minutes oftentimes. It really is incredible how quickly you can ship this stuff.

One note on pricing. So SemiAnalysis did a price comparison. The same $200 Claude plan is worth $2,500 to $12,500 in API value. And so they compared Sonnet 5.5, Opus 5.5, and Fable 5.1. If you use the max amounts, uh, what, what your value that you could get out of them would be. OpenAI recently changed the way that they calculate the limits, and after their recent changes, the maximum value you can get out is about $2,900 out of a $200 a month plan. So right now, the best dollar value you can get is the $200 a month plan with Claude and using Claude Sonnet 5.5. However, I will note that I have had hallucinations with Claude Sonnet 5.5, so I have never stooped below Opus 5.5. At this point, I believe the $200 a month plan is kind of a loss leader for senior executives to implement at their company. You start off using it, It becomes so useful, you get caught up in this AI psychosis, and then you roll it out across the rest of the company in the next few months. So that's my guess on why we have such high values, such high value given for these $200 a month plans right now.

As Swyx said yesterday, we're all kind of token billionaires in the sense that at $200 a month, you get over $10,000 worth of API usage. What do I need to do to be processing a billion tokens a month? The limitation is much more often on, can you come up with enough good ideas to use your billion tokens? I'm trying to hold myself to a high standard in doing that, but can't say I always max it out. I mean, I've maxed it out from time to time, but more often than not, I'm probably leaving more value on the table than I should.

Researchers call it consent fatigue. Windows Vista asked permission so often in 2007 that Microsoft cut the prompts back for Windows 7 because users had learned to click yes without reading. Now AI agents ask before every command. Does anyone read the box?

Nothing more frustrating than, you know, coming back hours later and being like, oh, you got stuck on asking me to approve this one stupid command. That I wasn't really even going to read anyway. That is definitely, I think, a very unsolved problem. How do we actually get people to pay attention when they should be paying attention? Obviously, you know, I'm living proof that if you just ask for permission too much, then people just kind of get numb to it and just blindly approve everything.

How we've traditionally done it is not getting people to pay attention, but forcing the companies that deal with those people to provide things in an easily understandable manner. And if it's too complex, to step in with regulation to decide how the framework should work. So, for example, no one reads their insurance contracts. The insurance contracts are basically decided by the state. And the state will intervene in laws. And the state keeps the insurers kind of well-capitalized, changes the rules from time to time. Typically, because as a consumer, you don't have a lot of power against a large corporation. The state steps in and the state decides what the contract looks like, and then people don't have to care. That's what I think regulation kind of does. It's going to narrow what consumers see to an easily understandable thing that you can accede to very quickly without too much homework.

Meanwhile, we've got Haiku 5.5 was just released while we've been live as well. So the cost curve continues to come down.

I have some queries running to Fable on the API for the show. And I will note that Anthropic APIs are always slower than I would like them to be. So latency, latency is still, I think, an issue for the firm. And I think that's an issue around the capacity that they have. I would still be reluctant to rely on Anthropic in production. OpenAI API team has done a much better job and things are just faster, more snappy, and like you can tell.

Evan Miazono is founder and CEO of Atlas Ignota, a nonprofit housed inside Convergent Research that recruits people for the AI safety work nobody has claimed. Edward Hu is head of AI modeling at Merkur, where he trains models on expert data and keeps its professional work benchmarks honest. That was AI AM for October 7th.