Scaling ML Infrastructure Like a Pro with Elliott Clark

Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast, and my goal is to help you create DevOps toolchain awesomeness. Hey, in today's episode, it's a must listen if you're trying to scale machine learning infrastructure. Well, burning it all down. Hey, I'm Joe and our awesome guest today is Elliot Clark, someone who doesn't just talk scale, he's lived it, as you'll find out in this episode. If you don't know, Elliot has built one of the largest online transaction processing systems on the planet, helped shape Apache HBase Cloudr before the IPO, and most recently led the ad infrastructure team at Meta. Now he is also the founder of Batteries Included, Rethinking how Platform Infrastructure Works, but this time as his own boss and working with nodes. Are really excited to hear more about this. If you're wondering or really want to know how to do a work at scale and how ML systems survive real world pressure, or why infra teams keep repeating the same mistakes, this one's for you. You don't want to miss it. Check it out. Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts, that is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating and in my experience and many others I talk to on this podcast, frustrated users don't last long. But since slow performance is as sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback. But I always say try it for yourself. Go to smartbeard.com or use our special link down below and try it for free. No credit card required. Hey Elliot, welcome to the Guild.

Well, thank you, thank you for having me.

All right, before I get into it, is anything I missed in your bio that you want the Guild to know more about?

I think for people that are kind of DevOps and testing and stability interested, One of the other positions that I had at Meta was I was tech lead of a team called Dev Efficiency, which basically means how can we make Developers faster without breaking things. Right. So it was a lot of testing, performance and deployment related stuff. So that was also part of my scale journey, was learning and breaking those things quite well.

All right, so let's put you on the spot there. What are the top two or three things you implemented with that to help with dev efficiency?

Well, can't talk about all of them. Right. Meta keeps tons of it for themselves.

High level, high level, high level.

I think a lot of stuff that would test shadow environments with real production type traffic. And that was one of the big things that I've learned is just how powerful your production traffic can be in making sure that you don't break things in actual production.

Okay, so how do you get there though? When I used to do performance testing, I used to test in staging testing, go to production bomb. But that's been years ago.

Yep.

So I assumed you have a dockerization ephemeral environment, you're going to spin up and completely mimic a production environment.

Yeah, Facebook has a lot of environments and so they have something where they will capture incoming traffic for different services and then you can replay it. So while I was there, we had an open source project called Treadmill where you could take thrift type requests or other requests and replay them. Right. And so if you can replay what's going on in production, you then have a lot of different things you can do. Right. You can do performance testing. Do I get through this production run fast enough? Great. You know, my production won't go slower because I have, you know, introduced a new bug. You can also kind of sum up the total results of a replay. Right. I replayed an hour. In that hour, were the ads that we would have recommended what we wanted or were they X, Y or Z? You can kind of go back and look at what would have happened if production had have been using this version and that becomes super, super powerful. Right. You have a baseline that you can compare against before you even go in production. And I think the real insight is just how different production traffic is than what developers assume it's going to be. Right. You all write a test, a unit test that says, well, this is how someone calls my thing, but you don't really know. And there's always a new thing that you learned, a new something. And so keeping up with what's actually happening in production and using that in test environments was super, super useful and super powerful.

That's a good point. I guess if someone had like a microservice infrastructure, you may call service and I know it has Dependencies and other services. Until you use a tool like sounds like Treadmill to find these very cool.

Or your dependencies start sending different results. Right. They have a new version.

Right, Right.

And so what that new version sends to you, your behavior can be different. And you want to notice that before you go. Right. So you want all of those things to kind of surface and rise before it rises in production.

Interesting. So could you use some of these tools in production first to capture this? So before you even start building your requirements or developing to know exactly what you should be building? I don't know if that makes sense. A lot of times people build anticipating what it should be or how it's going to be used in production, but if they listen to production now, maybe they can anticipate some issues before they even start coding. Does that make any sense?

Yeah, I don't know about before you start coding, but definitely you can use it before you're too far along. Right. So a lot of performance, a lot of, hey, I'm doing this. You know, you replay it locally and does it hit the same problem? Right. Do you still see the same slowdown or can you get faster? Meta and a lot of the bigger companies I've been at all use compiled languages. And so figuring out what the compiler is going to do for you and what that end result of your code is going to be is a lot harder to reason about. It's not quite as straightforward as this will make me faster. It does make the whole program faster.

Nice. So you obviously inherited probably a lot of systems that are already in place you had no control over. I think you're known for more of building scalable infrastructure from the ground up. So if you're on a greenfield project, say, how do you know what infrastructure to use? Do you do a little beta test between different infrastructure or different. Different technologies? How would you get started?

I guess, yeah, sure. For me, the kind of way I choose the default infrastructure is I go with what the community recommends the most. Right. So you're going for a database pick postgres. You're going for, you know, ssl. You go with let's encrypt. You just kind of choose the default and it gets you out there enough that you can stub your toe. The real engineers change because they have a complaint about something. Right. That wasn't fast enough. It didn't solve my. Whatever. But you don't have that insight of what's not going to. What's not going to fit your problem until you actually go out and try it so choose the default. Use that to build enough where you can start saying this is what didn't work for me. And then you can go search for the kind of more cool or fancy or shiny things because you have a need, you have a thing that pushes you to it.

That's a great point. It's like doing a little poc, not just sitting there like playing around for months just trying to figure out the perfect infrastructure.

For sure, yeah, you won't be able to say what you need to tune until you know. You won't be able to choose and say this trade off is worth that until you know what the pain is. Right.

And if you're using more common tools, does it help you to know how to design for scale or does it all depend on your particular application? Is they're all different, I guess one to the other.

I really think that a lot of it comes down to learning your tools. And so choosing the ones that the community has already kind of banged on means you don't have to go learn that, hey, this thing can't actually scale for me. You can say, hey, I've had a problem with, you know, I use postgres right now for a lot of JSON stuff, right? And so I can go find a blog post or a developer that has talked about how you can optimize JSON storage on postgres. If I tried to do that on a database that has, you know, a hundred people using it every day, I probably don't go find that blog. And so my path to go find that optimization is much, much harder. Right. I have to start reading code, I have to do all of those things rather than finding an expert that can already kind of point me in the general direction. At some point you still are past the blog point and you've got to go deeper and deeper. That's still always there. But I'd rather that kind of softball easy automation be something that I can go have the best already kind of found and pointed out for me and learn from the giants who have already come before me, stand on their shoulders.

Nice. You know, I don't know how many people do you think to scale is easy now you just throw up more virtual environments and there you scale. I'm sure it's a lot harder than obviously there's other breaking points, I'm sure, but is that too simplistic to think?

No, I don't think it's that simplistic, but I think scaling often comes down to actually a problem of scaling people and scaling people's understanding. Right. We've gotten to a point now where yes, if I need a second or third read replica on a database, right, that's. There's no can this be done? Right. You know, this can be done, but the hard part then comes in knowing what all of the things you've just done really mean in production. Right now I have to wait for log replicas now I have to keep monitoring this and that. And those kinds of things are the things you have to teach people. Right? That is where scaling really comes in. Is teaching your team how to, how to move correctly and stably around different code bases is super, super challenging, I would think.

It seems the softer skills or the non technical skills are the hardest to scale. So how do you. Any pointers on how to do that with the, like with the team? Get them up to speed or get them all on the same page?

Yeah. So I think a lot of it comes down to culture, right? You've got to have a culture that encourages bringing people along and encourages people to be okay with failing. Right. So that means that your new engineers can come on and feel okay trying some experiment and bringing it back to the team and saying, hey, I tried this, it didn't work at all. Right? And they've got to feel acceptable saying this isn't going to work, I need some help. Whereas if you have a culture that says, hey, you failed, it's your problem. That's real hard to kind of scale systems because people end up hitting a lot of brick walls on really hard technical challenges and they need help. And if they can't go ask for help because they feel ashamed or whatever feeling inside, then you can't go teach them. And then the other part is just rewarding teaching, rewarding people that are the fonts of knowledge. Rewarding people that have go out and experimented and played with it. Let them kind of teach people. Give them an hour long lunch session where they get to say, hey, I played with this new JSON parser or I played with this new way to compile ML models to a new model format. Even if it never makes it into production, it never changes your software. The fact that your people are now thinking about these new things needs to be rewarded. And so finding a way to reward your engineers who have learned and then want to teach, that gives you a culture that is okay with scaling into these hard problems and doing that in a way that can go sustainably.

Love it, Love it. Great advice. I just want to pivot a little. I usually start off this question, I Don't know why I didn't today. I love speaking to founders. I'm always curious to know why start a certain company. So why did you start? Batteries included?

I wanted what I didn't have.

Okay.

So I was at Facebook before it was called Meta for seven and a half years. And while I was there they have an amazing set of infrastructure teams, right. I was on the database team for a while. So you want a database, you click a button that's multi continent, Paxos blah, right? And you just, you clicked a button to get it. And so I knew I was leaving Facebook and was starting to explore different startups that I ended up finding out were not my path at all. Right. I didn't want to build those, but in doing it I kept running into the same problems of, okay, I'm going to build this, I need a database. Well, I kind of want some monitoring on what I'm writing and then I want to be able to version it and all these kinds of tools that for me were just so intrinsic to how I built things. But putting all of the open source solutions together ended up taking so much of my time. And when I was trying my second or third go at a startup software, I got annoyed at doing the same thing just a little differently again and again. I wanted an automated solution that gave me all of this open source that I had at the big companies with minimal friction. And so that's kind of where I got to is I was just annoyed doing the same things over and over again to bring up a production environment. So I wanted to automate that like I would have as a PE at Facebook. So just did that and did it with open source and off we go.

Well, so what are some of the pain points then? And maybe current cloud native solutions that you're solving with this model? Obviously you can help easy to get up and running really quick, I guess. Any other types up and running.

And then also infrastructure does well when it has somebody that loves it and follows it and updates it and does all of those automations you get a new thing, but a lot of companies just don't have time for that. Right. The database team doesn't get days and days to try a new version and do those things for mid or small size companies, but at the large fangs of the world you get that easily. Right. And so I want to basically give all of that automation to everybody and say, hey, I can upgrade your postgres every time there's a new version for you, but we will already have done the integration Tests, we will already have written the automation so that version A to B goes through or SSL certs. Let's make sure we always get them renewed on time or like all of those kinds of things. Need a team that's watching it. And I want to give that, because the way that I've seen it given in large companies is just a bunch of automation that gets kind of sprayed to tons of computers. So why not be able to do that in open source and spray that, that love and that expertise to all of the rest of the companies that are wanting to build on open source but unable to.

Nice, nice. How do you keep track of all the open source packages, some that may have security, things that are built into it and you've already had a package full of like, how do you keep track of all open source projects, things that are breaking, things that have security risk, that are introduced to things like that?

We have a giant registry of every. We're batteries included. So of course they're batteries, because I can't pass up a pun. Right. So we have a giant list of all of the software inversions that go along with each battery. Right. And so we have just a single place to go up, upchange it. And then our full test suite runs, redeploy, upload, do all of the thing with a new version so that we can kind of test each one individually. And the hope is that we stay kind of like a rolling distribution. Right. We stay as close as possible and push those at a weekly ish cadence to you.

So this is great. I used to work in a regulated environment. Every quarter you have to write down all the open source things you're using. This is great. I'm using batteries included. They have a list right here. There you go. All set. Nice.

Yeah. We want to basically depend so closely on open source that you're able to say, here are the shoulders of giants that we're standing on and you can thank them. Right. Without Grafana, we wouldn't have our dashboards or without all of these things. And so being able to say, here's what you're running, off you go also gives you some agency to say that, you know, if something breaks, you can go to the Grafana project and write up a patch and say, here's the thing that would fix the version that I'm using. Off you go. And that kind of close tie with open Source just gives so much power and control to people that I think it's worth the community.

Nice. So what have you discovered after releasing what you thought would be why you envision what batteries included would be and what it's become.

I think I didn't really fully realize how many open source projects straight up depend on the cloud. Right. S3 being so, so ubiquitous to being able to run things in production was quite a surprise to me. Right. I kind of assumed I would get around to durability and storage as a later feature and then kind of found that like, nope, that's not possible. You have to start talking about S3 and where do you store large files or those kind of things Kind of early on, earlier than I thought.

So how does that work with redundancy? Say S3 goes down. I guess you could use it as a dashboard. Well, it went down because these three services you're using that. We're using it, yeah.

Right now we use S3 mostly for backups. Right. So if that happens, your backups are delayed or your logs aren't archived back to S3. We try and keep postgres on just kind of hardware, hard drive kind of things. So less dependencies in production is always better and less distributed dependencies are always better.

Nice. So how much is machine learning involved.

In this process for batteries included? A little less. That's. We're just getting to that right now. That was part of reason why I wanted to start. Batteries included is the tech stack that you need to get to a real. Well, running machine learning environment in production is huge. You need so many different things to be able to go back and find what happened, what changed, what's different that I didn't want to put machine learning in until later. So we're just now about to release a new version that will include the full ragstack so that you can keep it all on your hardware or in your cloud environment rather than sending your data to OpenAI or somebody else. You get to keep it all internally. So that one's just coming because I just now feel like we have enough. Right. We feel good enough about postgres that we can use that as a vector database. We feel good enough about our load balancers and networking that we feel like you can actually trace all of the calls to see where things were going. That you now feel comfortable enough to do the next step of complexity.

Nice. So do you have like a checklist? If you're going to start implementing ML into your software, you may want to try these five things first or go in this order or anything like that.

Yeah, I would say just. I'm always a fan of going and trying. Right. So for Me, I love sending people to try ML from jupyter notebooks. It's just so easy to get started. It's so easy to get a Is this working? Is this not? You get to the place where you can start complaining about what's working or not means that you have some perspective into what your actual end goal is. And so for me, that that's kind of it. I love those. And then a vector database has kind of become needed for larger enterprise workloads and off you go with those.

Are there any misconceptions you think teams have about implementing machine learning that makes them like, I'm not going to use machine learning for my large production environments because of X, Y or Z. I'm.

Actually not seeing them not implement it. I think the most controversial thing I'll try and say is I don't think most teams should be shoving LLMs in most of their stuff. They don't really understand the risk that comes with that and they don't understand the production performance implications of just asking someone to hold on and wait until the LLM is finished reasoning about things. And so seeing everyone rushing for the new shiny without really having a this is what I expect to get from it. This is the thing that will be better, not just a little bit cheaper or easier for me today, but what will actually be better is kind of one of the things I wish I could bring back and say no, but make it better, not just a little bit cheaper and easier for you right now.

Absolutely. So where do you see the future of infrastructure, especially with ML? Do you see like, I know it's hard to predict three to five years, but I'm always curious to know. You seem to always be learning. Are you looking at things down the road that you think, oh, this could be something that people should stop paying attention to now?

Yeah, I think I would start looking at different architectures for ML. Right. So right now everyone is kind of all in the we're text LLM with maybe some image that we can also but ask it in text. That's the kind of machine learning that's applicable to just kind of all of society. But I've seen how powerful ML can be when it's tailored to your business. Right. When it's recommending what you should see in your feed, when it's deciding which AD you should show, when it's choosing an automated placement group for where you should put your compute or those kind of things are much more specialized. I see the world going to a lot more than just LLMs. And so exploring what those other types of learning or statistical learning can do for you in other areas being combined with LLM. Right. So LLM does your describe or your whatever and then you hand a lot of that information off to another, more specialized model, I think will become a pattern that we see. But that takes a while till teams and companies can get the. How do you train a model? How do you version a model? How do you push it out? How do you revert it? How do you keep the data tagged of where each of those things came from and what they mean? That kind of thing.

Nice. I've been working on this with a lot of. I do news stories on modal context protocol. Everyone's coming out with an mcp. Is this applicable at all to infrastructure? Mcp, do you see that as something that's new, that could help better with infrastructure?

I think MCP could help. Infrastructure. I think MCP can help in that we spend a lot of time trying to explain things to a computer. Right. I'm going to write a ton of stuff into my shell to explain that. What I really want is a list of the running.

That look like this or. And so using LLMs and MCP to kind of bridge the gap of what information is out there and how do I explain to a computer what I want, what I want to see. Those things are much easier for humans to explain in text. Right? I can tell you what I want probably before my brain can even start saying, here's the five commands you need to pipe together to get that. Right. And so being able to give that to people that maybe don't have that command line experience or they're in an environment that they're not so familiar with, will open up a whole lot of new things. And I think the MCP in that kind of environment will be less useful for the super experts. Right. The ones that spend their entire day on one package, I doubt they'll be doing as much MCP day to day. But using it as a way to kind of interface with things that you're still learning or maybe don't have as much on the top of your head, I think those will be super, super useful.

So if someone's listening to this, like, oh, I need to give batteries included a try, is there. Is this a paid version? Are there free versions? Is it like a starter plan? How does it work?

Sure. So I try and say that I want to be the Robin Hood of infrastructure. I want to give everybody what the big boys have. So we wrote our own license. So under A certain number of computers, it's always free and is free by default. Then after that we will have a paid version. But for right now we're just completely free to everyone to go just try it out. We will be a usage based. So you know, if you have 700 batteries with a million computers running it, we're going to charge you a lot more than someone that has, you know, 10 computers running it. But it's also open source, right. So it's on GitHub if you really just want to go play around and try it. Our license has a time delayed Apache to it. So after a while whatever out and been released becomes just Apache 2.0 for you. So you can play with it on your computer on your own and make changes to it just like it was Apache. So you get kind of the best of both worlds. Which also means that we as a team have enough funding to come in and be that infrastructure company that goes and does those updates for you. We care about the new security patch, making sure ISTIO is fully TLS for you doing those things. You get that knowledge that we have to move you forward since we're giving it all away if it's old enough, while also being usage based so that we don't charge you a million dollars because you went from the free tier to the pro tier.

And that's probably a lot of the grunt work that no one wants to do. So it's perfect. Right?

Right. I think just integrating so many of those systems is the grunt work that doesn't get done well. Right. A lot of people will have something where they're like, I need to get monitoring up. Right. And so they get one grafana dashboard and they call it done. And then three versions from now that dashboard has metrics that are no longer in this new version and no one goes back and, you know, redoes it because there's nothing. There's no one that loves those dashboards anymore. They've moved on. There's no one that's checking to make sure the logging versions still work. So that kind of integrating and checking that the integrations still work is kind of what our bread and butter is what we do day to day right now.

Like I said, there's a lot of things going on with DevOps and ML. Are there any other cool things you see that could be applied to with machine learning directly into DevOps efforts that people are probably struggling with right now?

I think one of the biggest things that you can do to make your kind of DevOps and ML experiences Mesh well, is really talk about your data acquisition for what's happening in production and tying that in to the model version that you're currently using. Right. So I have a request that comes in, it shows some page tie in. That page had model X, Y or Z that was used to generate the, you know, help description or the whatever. Using and tying what models are running all the way back also gives you, oh, that request was super slow. Cool. Then we can tie it back to a model version or lots of people are complaining that they're not seeing the right content or whatever. Tying that back not just to, hey, it had some ML in there, but exactly what model, exactly where it came from allows you to see how models are actually impacting your customer experience without you having to go do a lot of, you know, finger in the sky, guess and check kind of which way is the wind blowing?

How do they know those ML models? Is it because batteries included keeps track of that as well, or is that.

Right now, batteries included doesn't do that. Right. We don't have a lot of that kind of ML stuff tied in yet. Right. That has to be a lot more specialized to the application. Right. What is my application that I'm doing running? And that's the kind of special sauce that companies will have to produce themselves. We hope to make that super easy. Right. So that all the logging and all of the tracing can easily go from there, but currently we don't yet.

How much does OpenTelemetry help with that? All those logs and machine learning to help?

OpenTelemetry works and does a lot of that, assuming that you put in the correct metadata for it. Right. So you've got to make sure that when you log your request ID that you also log that same request ID on your microservice that calls out to an ML model. Right. So it's kind of on the developers to make sure that they tie in the right stuff to OpenTelemetry. But if you do that, and when you do that, it really pays off a lot.

Okay, Elliot, before we go, is there one piece of actual advice you give to someone to help them with their DevOps infrastructure efforts and what's the best way to find or contact you?

Sure. The best advice that I can have is explore new ways to test. Right. Find new ways to break and find interesting things about your software. Teaches you what to expect if something happens, and also can proactively get you fixing it before you run into it in production. I've had great experience with, you know, throwing fuzzers at things, throwing replay and production traffic at shadow environments. Just doing production like runs on test things breaks so much. So go try it and best way to find us right now is we're on GitHub, so battery stash included. We have our all of our issues in open source right there. We also have links to Slack or email me any of those things. But yeah, GitHub's the greatest way to find us.

We have links to everything we talked about in this episode down below. All right, before we wrap it up, remember frustrated users quit apps. Don't rely on bad App store reviews. Use SmartBear's Inside Hub to to catch, fix and prevent performance bottlenecks and crashes from affecting your users. Go to smartbear.com or use the link down below and try for free for 14 days. No credit card required. And for links of everything of value we covered in this DevOps Toolchain show, head on over to test guild.com p183 so that's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing and DevOps. If you haven't yet, join our vibrant community at Test Guild, where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges. We're excited to Collaborate visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve O the tescill Automation Testing Podcast with lutes and lyres, the bards began their song. A tune of knowledge, a melody of code through the air it spread like wildfire through the land, Guiding testers, showing us secrets to behold.