A team serving 200,000 customers built autonomous agents that automatically validate, load test, and optimize AI services without manual intervention. The system, called Dark Factory, runs the entire software pipeline in a loop to catch failures before they hit production at scale.
How do we make sure that at our scale, when we're dealing with 200,000 customers around the world, multiple languages, very different kind of workloads, businesses that might have 100 customers or 100 million customers, how do we make sure that it scales for all of them? And then it's going to be performant. This is one of the things where we've actually written agents that I'll show you in a second, that once you go past prototyping, will actually validate The actual software, and then we'll run load tests for us so we can figure out where things are going to fail at scale. So one of the things that we've done is we've taken the entire software toolchain, right? Things I'm used to doing by hand, one by one, is once we've got a prototype, whether it's testing, load testing, testing against various edge cases, thinking about, hey, what are the features or the functionality that we're likely to need next? How do we then Build an agent that can plug into our agent loop to then go run all of those tasks. And the reason we did this, so as we did this, we found that when we started, a lot of our code, like, if you're an engineer, we found there's a lot of stuff that looked like on the left, right? You build out these prompts and you have one by one, and you kind of, you add on and you get this, like, if you're, if you've been an engineer, like, you end up with a lot of, like, spaghetti code, a lot of stuff that's, like, just an unmaintainable mess. Yeah. One of the things we started building last fall is, is a project, an internal agent that we have called Dark Factory. If folks don't know, in manufacturing, there's a concept of lights-out manufacturing, which is basically, hey, if you have all this machinery and there's people around that are using it, then you keep the lights on because they're the ones that are actually like going in there and fixing machines and all this kind of stuff. Increasingly, there's this concept of But the more you can automate the factory, you can actually just turn the lights off because that's not what the machines are after, right? They don't need that. So one of the patterns we built is this. What does it take to build a good— to build more complex applications, a software dark factory?
