MLOps and tracking experiments with Allegro AI

When we talk about MLOps, we talk about the ability to move from the data scientist own machine or laptop to training models at scale on some remote machine cluster. We're talking about the ability to orchestrate that and do that within a larger team, not just a single data scientist. And we're talking about the ability to automate that process. That in general is what we talk about when we talk about MLOps. How is it different than DevOps? Well, it's actually very different.

Bandwidth for changelog is provided by Fastly. Learn more at fastly.com we move fast and fix things here at Change Law because of Rollbar. Check them out at robart. Com and we're hosted on Linode Cloud servers. Head to linode.com changelog this episode is brought to you by Digital Ocean Droplets Manage kubernetes, manage databases, spaces object storage, volume block stor advanced networking like virtual private clouds and cloud firewalls. Developer tooling with a robust API and CLI to make sure you can interact with your infrastructure the way you want to. Digital Ocean is designed for developers and built for businesses. Join over 150,000 businesses that develop, manage and scale their applications with Digital Ocean. Head to do co changelog to get started with a $100 credit again.co changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@change.com community and follow us on Twitter. We're Racticalai FM.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host, Chris Benson, who is a principal AI strategist at Lockheed Martin. How are you doing, Chris?

I am doing okay. It's summertime here in Georgia and so it is hot and humid and so I'm just trying to keep from melting.

Yeah, not, not unexpected for, for where.

You are, I imagine happens on a regular basis. Hot human, not the melting part.

Yeah. So I think I mentioned this a couple times on on the podcast, but my wife owns a candle business so there's always this like during the summer you got to figure out the right shipping and tracking so that you kind of minimize the likelihood of candles melting on people's porch before they actually get into their house if you're sending them to like Texas or Arizona or that sort of thing.

That's a good point.

Yeah. So it's an interesting thing. On another, you know, shipping front, I've got a pile of boxes sitting next to me and a computer case. All the components for a computer are here at my house. So I'm about to build a first AI workstation of my very own. So I'm excited about that.

Very nice.

Yeah, it would be fun to have an episode, you know, detailing all of the mishaps that happen along the way as I hopefully. Hopefully you don't ruin it, but get this thing running, I'm sure.

So I'm curious, since you brought it up, I know in the past when we've talked, we both have typically gone to cloud services, especially for personal things that we're doing at home for our own interests. What caused you decide to go this way this time with a desktop?

I think it was twofold. I think partly it was like I haven't built a computer since I was in college, probably, which would have. I don't know if that's over 13 or 14 years, probably 15 years since I built a computer.

Maybe.

I thought it would be fun to just do it again. So that's partly just fun. But then also I'm getting into a lot more of audio models, so speech recognition things and spoken language identification and the data sets associated with those are quite large. And so sort of carting those around to various cloud machines and also running models for, you know, maybe days instead of hours starts to get fairly expensive. So I think those two things made sense to me.

Well, good luck with it. Yeah, we'll. We'll definitely have to get an update from you to share with us all what happened and what went wrong and what went well, for sure.

And today we're going to keep the practical train moving with some more topics that are extremely practical. Actually, I had seen what we're going to be talking about today, which is some tools from a company called Allegro AI. One of my friends pointed me to that, which I'll mention maybe a little bit later on, but I also saw Pytorch mention recently that Allegro Trains, which is one of the ML ops and experiment managing versioning things that we're going to be talking about today, joined the Pytorch ecosystem project. And I thought that sounded really exciting, also very practical. So today we've got with us Nir Barlev, who is the CEO and co founder of Allegro AI. Welcome, Nir.

Thank you for having me, guys.

Yeah, before we jump into all of those exciting things about experiment tracking and versioning and MLOps and all of that, it'd be great to hear just a little bit about your background and how you got involved in this field.

Sure. You know, I've been in the high tech industry for longer than I care to actually say, probably three decades. I started as an engineer actually and spent about a decade on large ERP systems and that kind of thing. This is way back. And then by way of an MBA at Wharton, I joined Google. And a decade at Google doing everything from working on the mobile team. This is right after Google had bought Android and you know, before the iPhone went out and to actually helping setting up Google's Tel Aviv RD center, to leading Google's European search, advertising, product and strategy and a number of other roles. And as role I was a GM of mobile payments. And yeah, when I decided to look for something else to do, I joined two folks who are, you know, actually my partners now to basically start Allegro AI. The way I came at it is I was looking to do something big that can impact the world and that would involve cutting edge technology. After being at Google and doing everything it did, you don't want to do anything less than that, really.

Yeah, I was going to say being at Google, if you're thinking of projects that make an impact, sort of worldwide impact or innovative, it seems like that sets a pretty good trend for your path forward or a high bar to reach.

For sure, that's correct. I don't know if be able to build a company as large as Google, but certainly that's the target you want to put for yourself. Yeah, yeah.

And it is interesting, I mean it seems, and I don't know, maybe you have a perspective on this as a CEO and founder, but it seems like there are a number of really innovative startups in the AI space that are kind of playing at the same level as the major players of, you know, OpenAI and Google and Microsoft and these at the same level at major research conferences you see kind of startups, I'm thinking of like hugging face or those sorts of startups that are really right there. And it seems like such a huge impact for a small team. And so I don't know as a CEO if you think about those things, but it, it seems really interesting to me that there, there can be these small, really focused teams that make a very large impact on that level.

Yeah. You know, being a Google, I have to tell you, you kind of think that you can probably do anything, right. The reality is that, and I've seen that personally on some of the big projects that I was involved with is that you Know, Google didn't execute as well as a startup or as fast. Google ended up, you know, acquiring them for less or even a lot money. Right. And there are a number of examples for that. And as a company grows larger, the targets get bigger. Right. And so doing anything requires a very, very high bar. I remember at some point, I'm Talking about like 2007, I think I remember pitching something to the time with Susan Wojcicki, who was at the time, I mean, now she's the CEO of YouTube, she was the head of advertising, and the bar was, you know, if it's not about $100 million revenue, don't talk to me about it. Right. And you can imagine this is back in 2007. So imagine today and this gives an opportunity for companies who are very nimble to identify opportunities. There's also a different perspective when you're outside of Google, as when you're in, especially, I think in the B2B space, there are opportunities where, you know, at least Google specifically is still relatively behind companies such as Amazon, for example, or Microsoft. Right. So you can identify small companies niches, and if you understand that those are going to grow, then that's opportunity.

So kind of curious, as you were at Google and you know, how did you come up with this idea for what would become Allegro? And so you're kind of, you had been doing that at Google for a while, moving through your position. So what made you think, I have this idea I'm going to make a major change in my life? You know, what gave you the motivation to go off and do a startup, find partners? Can you give us a little bit of that backstory?

So first of all, I can't really take credit for the original idea behind Allegro AI. That's actually one of my partners. I guess I can take credit in what we formed out of it and what it became. Because obviously, as in any company, especially startups, we change and we adapt quickly to find product, market fit. So obviously the vision as it was set or thought of by my partner, needed to improve and get better. And you know, that's something that I was involved in. But the original idea was not mine. It was more of for my position. You know, I had felt like I had, you know, I'm in Tel Aviv, Israel, and it was about relocating my family back to the US and at the time, this was about four years ago, it didn't make sense. And coupled with the fact that, you know, I joined Google when it was 3,000 people, I think it's about 100,000 now or so, you know, on that kind of scale, it's a different company in many ways. And I had felt like it was an amazing experience and I learned so much, you know, especially you know, being at Google in that time of growth. But you know, when I left Google, it was a big company with all the things that we all less like about big companies. And I felt like, you know, this was opportunity to do something different and really go out and try to build something on my own. And as I mentioned, I looked for something really big that could change the world. And you know, basically as a potential founder, I started, you know, quote unquote dating people, right, to find partners that we could, you know, come up with something that we'd like to do. And through that quote unquote dating process, I met my current partners and it was, you know, we hit it off, as you'd like to say, really quickly. There are amazing guys. When you're in a startup and you have partners, it's you're practically in a Catholic marriage for the time until the exit. And so you want to make sure that you have people that you can trust and that they're great people that you can work with and obviously amazingly capable and talented. And I found all of that with them. And basically as I mentioned, one of my partners was the one that was bringing that idea and it came to him. He's a longtime serial entrepreneur. He's a very interesting profile where he has both a very, very strong engineering background as well as a data science background. So the most prestigious lab today in AI in Israel, so it's run by Professor Leo Wolf, who's actually now at Facebook. And my co founder, his name is Moses Gutman, was his first PhD student, which basically means that they set up their lab together. And so he's really one of the pioneers of deep learning, machine learning, computer vision in Israel. And he basically saw what Allegro really is all about is the fact that we need to bring in engineering methodologies into the AI in the process. That was not the way that he said it at the time, but basically that's the idea. How do we actually scale things up?

And I'm kind of curious on that front and you've stated it well and I think that this has been brought up on our show multiple times from different perspectives. So I definitely think it is a theme that kind of surging through the community that we, we need to be more rigorous in terms of the engineering we put into, into our workflows and the AI driven products that we're building and putting out and the tools and all of that sort of thing. I was wondering, from your perspective, what you see as the challenges to kind of what are the sort of main challenges to getting people on board that are currently in data science and AI positions and kind of convincing them that they need to start doing things differently? What are some of those challenges? Does it have to do with this kind of variety of backgrounds that people come from, that it's not just engineers, or is it more than that?

Yeah, that's a great question. And the answer, actually, it's a moving target because our industry, right, the one that you guys are talking about and the one that I'm squarely in, is rapidly evolving and changing as we speak, at an amazing rate. I've never experienced that kind of rate before in my career. So generally say this, right? I mean, basically, right? It's a very different paradigm, right? It's a scientific paradigm, right? And initially people thought, well, you know, I'll get data scientists or research scientists, and that's what I need, right? And then they'll be able to do the job. Obviously, we all know that's not enough. The thing is that there's still, you know, a core and critical part of a team that needs to build something. But data scientists, research scientists have a very different, you know, mindset and outlook, right? I mean, they've been trained differently, right? I mean, they're at the end of day, they're scientists, right? And if you actually, you know, take it to the extreme, think of that, you know, mad scientists, nothing is in order. You know, everything is hectic. It's all about their creativity and finding the solution. And there's a lot of truth in that. Obviously, that's an extreme exemplification, but there's a lot of that, and that's changing. But we found, you know, throughout the course of the last three years of, you know, data scientists, research scientists have been very much against adopting any tools because, you know, they came out of university, they were focused on, you know, on the science tooling. They didn't understand the value of tooling, the value of processes. In some ways, you might say they, you know, maybe even, you know, were a little bit wary, right, of tools. And is that going to be good for them or bad for them? It was not. Wasn't even something that they were exposed to during the curriculum of their training. On the flip side, you know, they felt like, you know, I'm a, for example, Ph.D. out of whatever, Stanford. I mean, I should know everything A lot of times we saw relatively very junior data scientists leading AI teams. Not just in small companies, in very large companies, right? Because if you're not a Google or Microsoft or Facebook, you're not going to get the cream of the crop. And the last thing is, you know, their bosses didn't know what the heck they were doing. They didn't even know how to actually measure what they were doing. And as I mentioned, they thought that, you know, bringing those people in would be enough. And so a lot of that created the situation where, you know, the background of, you know, why do I need tools? And a lot of that still exists now, I think. But you know, a lot of people who have engineering background have actually started doing data science or you know, ML engineering and data engineering because, you know, because it's new, it's interesting, salaries might be higher, etc. Companies have realized that they're not seeing productivity out of the data science teams. And so that shift has been happening in the last, I guess year or year and a half. We've seen companies integrate, right, their data scientists and research scientists into a larger product team that has the engineers and the product leadership, et cetera, DevOps to really, you know, push them to ultimately build a product. Because it's not about coming out with a research paper, right? Ultimately, if you're sitting in a company, most of the time it's about building a product or a service. And so I think now what we've seen is oftentimes a situation where there's a very big under appreciation of what it takes to build a state of the art toolchain to support you. And I remember talking to someone that was way back in the day was pushing, you know, SQL databases. Imagine that, right? I mean, this is prehistoric times. And he was telling me how he had trouble pushing that into organizations because they thought they were going to build it themselves. Obviously, you know, anyone who tried to do that fell flat on their faces. Same thing here. And we've had situations that's changing a lot, but we've had situations where a couple years ago companies would tell us, what are you talking about? I can build this in three weeks. And they believe that after we showed them what we built today, a lot of these companies, because tools didn't necessarily exist or they weren't aware of them or they thought they could build it, have invested internally and built something. And you know how it is, right? Not invented here. And once you've built like a small toy, you're enamored with it especially. But for this sake, I'll say us engineers, right? I remember as an engineer I was enamored with some of the things that I built. And so that's kind of the hurdle that we as an industry that, you know, is building and pushing tools need to try to get around or over.

Foreignlog news is the best way to keep up with the fast moving software world. We track, log and contextualize the coolest projects, the best practices and the biggest stories each and every week. Make changelog.com your daily destination or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0 and you can subscribe right now at changelog.com weekly.

So Nir, I guess as we were, we were starting to get into tools and you were talking about, you know, whether organizations were starting to recognize the need for tools and how did they get productive and measure that productivity and that, you know, in a world that, you know, already has things like DevOps and ML engineering and data engineering and such, we're kind of moving into that area. I noticed that, you know, kind of front and center on your website you have this concept of MLOps and as you were mentioning DevOps and passing and the tooling before, it really kind of triggered that. I'm wondering if you can kind of tell us what MLOps means to you and the organization and kind of how does that differentiate itself from DevOps on the software side and other types of ML engineering and data engineering?

Absolutely. So actually that's a great question because it actually touches on one point where, you know, mlops itself as a term is not something that is set already and different companies are using it to mean slightly different things. You know, that's one of the, I guess, issues again with our industry so early on that terminology is not set. When we talk about MLOps, we talk about the ability to move from the data scientist on machine or laptop to training models at scale on some cluster, remote machine, cluster. We're talking about the ability to orchestrate that and do that within a larger team, not just a single data scientist. And we're talking about the ability to automate that process. That I guess in general is what we talk about when we talk about MLOps. How is it different than DevOps? Well, it's actually very different. So I guess let's define DevOps very high level. Basically the idea behind DevOps is that you want to make sure that a piece of software that usually is already tested, QAED and stable, right? That has left development and is now going into production to serve users or, you know, workloads, needs to work at scale and needs to stay up all the time, right? And you need to make sure that it can sell across many machines, etc. That, at the end of the day, is what DevOps is tasked to do. And so basically, what did we say here? We said that there was a single piece of software, that it was tested and it works, and that you need to take that and you need to scale that up and replicate that, right? And that happens only in production. Well, in AI, everything around that is actually different. So, first of all, as you guys know, machine learning, deep learning experiments can be very, very heavy workloads. I mean, you actually mention that yourself when you talk about building your own computer at the beginning, right? You're going to run things that are going to take hours or even days. And so, unlike regular software, you need to be able to run stuff on large machines from day one in terms of development. So that's one big difference. The second thing is what you're doing is you're running software that's not tested because you're doing it during development. The third thing is you're running experiments. And what you're trying to do, right, obviously, is run a lot of experiments, because that's the whole process, right? You're doing lots of experimentation until you reach your goal. And so with experiments, you're basically running pieces of code that are slightly different from each other. And that's a different thing than running the same piece of code, right, on lots of machines? And so basically, this is a very different problem. How do I, as a data science team, manage my workloads on clusters of machines? How do I handle lots of experiments that I need to run from one or more data scientists or a team of data scientists and do that effectively when we're talking about pieces of code that continually change, how do I actually take the environment that I built? Because in AI, again, the piece of code that you're running actually is much more complex on one dimension than a regular software, because really, it's an amalgamation of the model, right? The neural network, for example, the code that wraps it with the data. How do I actually take that environment that I built, my researcher and the model that she built, and then run it on a remote machine that has a different environment? All of these are different challenges. And this is the challenges that we attempt to solve. And this is what we call mlops.

Yeah, that's a really good summary. I like how you set that up in terms of the comparison to DevOps because it is kind of maybe a shock for people starting to get into this field where, like you say from day one, in order to actually make progress on their things, they might have to know about, oh, spinning up this GPU instead of in the cloud or, you know, Cuda libraries and running things in a repeatable way. It seems like a really high barrier for people to overcome, you know, from day one to get things working and also do it in a. In a repeatable way.

Exactly.

Yeah. I also wonder on the, like, you're talking about experiments and that sort of thing. I know one thing that is definitely true of myself and, you know, my wife could confirm, is that I'm not very good at remembering what I've done or what needs to be done.

Right.

In terms of the experiment tracking side of things, of course, there's like the running of things, which is definitely important. I think that's what maybe you focus on mostly. But there's also kind of this weird documentation almost piece of the puzzle that's not quite documentation because it's like a very specific type of documentation that's really documenting. Like, what have I done and what haven't I done? And how successful was that? And it's not really like you want to have a research paper necessarily, especially if you're developing these things as a product or maybe even a trade secret. Especially if you're on a team, want to have that common understanding of what has been done and hasn't been done. How soon do you see teams encountering that issue when they start working on this problem? And what are those kind of essential elements of, I guess more of the documentation or tracking side of things that need to be in stone somewhere over time.

Yeah, well, you know, as you were saying about documenting and how it's not exactly documented, if you come up with a term, please let me know.

Naming things is the hardest thing.

Exactly.

Doc. Doc Ops, maybe. Yeah, Doc Ops. That's probably already taken. That has to be taken.

That's a real issue. So, you know, that's a real struggle to name it. So I'll prefix and say. Actually, you're saying, you know, we were focusing more on the mlops. Actually, you know, one area where we're pretty unique is that we have a very highly integrated solution where we think that you can't focus on just one thing if you don't have a highly integrated platform that actually takes care of both of the experiment Management, the data management, the versioning and the mlops. You don't have the best scalable solution. But we can talk about that later if you'd like. The experiment management part of it, the documentation part of it. When do people realize that they need it? The answer is actually when someone in the team that usually has some sort of engineering background says stop, this is crazy, you know? Yeah, that's exactly the point.

I remember Doug, if you're out there, his name was Doug, he's a great engineer. One of the startups I work with, he was my wake up call.

And so, you know, I mean, it could happen with a team of one and it can happen. We've seen it happen. Well, we've actually seen teams of tens of data scientists that didn't have that. Right. And really depends you have that person who realizes that and has the influence and or power to actually say, you know, we need to change this.

Yeah, yeah. And I guess this is something that we've kind of talked about in passing, but that's this interaction between AI developers or data scientists and the rest of an engineering organization. So maybe a follow up question to Chris's question about differentiating MLOPS and DevOps. What is the kind of integration point, from your perspective, between the two worlds? Because if things eventually end up in a product, right, like I'm importing a model into some API handler in some code that is production product code, there has to be an integration point somewhere. Where does that exist and what challenges are at that integration point?

That's a great question. Actually the integration is something that happens continuously if you're actually running things well, so it's exactly as I said, right? I mean, ultimately what you want to do is you want to take this model, right, that you build to predict something or to solve something and then integrate that into, call it a wrapper or some larger piece of code that actually carries out the ultimate task of that product. The thing is, oftentimes you could test your model a lot, kind of like in a very environment that's kind of clean, but ultimately you're going to want to test it in the field, then you're going to have to have that wrapper. The other point also is that once you want to get into automation, right, and even if you're still within the data science part, if you want to get into automation and create lots and lots of experiment and you want to maybe you're actually fielding in continuously new data that's coming in, right. Let's say you're, you know, you're building an autonomous vehicle and you're, you're getting constantly new videos from your cars driving around, and you want to actually improve your models based on that, then that also creates an integration point. So the integration points is on those two levels. One is, you know, when you have to hand over the code so that it gets wrapped and to when you actually want to integrate those experiments within a larger pipeline that helps improve them. And there's another point that we actually try to facilitate with our product, which is how can I lower the barrier to entry? And I'll explain. Let's say you're a company and you're doing, you're building a solution to, I don't know, take computer vision. It's easy. I mean, let's say you're building something to identify cats. Right. Let's take the ultimate example. But you also need to identify dogs because you're building a pet detector or whatever.

You're speaking right to Chris's heart lover.

I was keeping my mouth shut this time. Yes, I am. Daniel can't normally shut me up on that, so go for it. Let's hear.

All right. As data scientists, you understand that if you've built a model, and I'm talking about the code narrator now, Right. That facilitates object detection for cats. Well, if you now want to do the same thing for dogs, what you need to do. Right. Is you need to take that code that you built for the experiment and probably the same neural network that is the one that you chose for identifying objects in whatever scenario, and now marry it with a different data set. That's it. Why would you necessarily need a data scientist for that? Why couldn't an engineer do that? And that's behind a lot of the stuff that we're doing. Also the ability to actually have the data scientists work on the core pet detector model and then have engineers facilitate optimizing that for the different objects.

Yeah. And I think that also actually that example itself illustrates another kind of unique feature of this. I think you're right in that those later stages could be kind of the popular word, I guess, is democratized to other people within the organization. Right. But also, it's still not quite the same as a normal DevOps in that if you're running with a different data set, somehow you need to have a kind of unique tracking that's going on with what data set was used to train this particular artifact or serialized model at what time? Because the code might actually be exactly the same. The difference might be in the data.

Exactly. Exactly.

Yeah. So I see so many people develop really sophisticated naming for their files and such, which you'd probably need your own documentation to document that. What about the data side? We mostly kind of talked about process and the operations infrastructure. What about the data side of things?

So the data is the holy grail at the end of the day. And I think that obviously experienced and senior data scientists get this. It's all about the data. Novices are focused more on the models, but at the end of the day, the difference between a product that meets the threshold of whatever KPIs you want it to hit and something that doesn't is about your ability to train it on the right data set and be on top of your data and be able to feed the exact, what we call data view to train that model. And so iterating on the data, right? Identifying the skews within the data and handling those, identifying the holes where you need to add more data or build synthetic data or augmentations around that, that is the key piece. And as we talked about this, right, it's an experiment process. And so being able to actually version that and track that, because as an experiment process, you know, you're going one track and then you realize, you know what, actually you want to go back to the model I built two months ago and actually take a different direction. You have to be able to version that or not just the model, you want to be able to version the data set. I mean, if you have enough experience as a data scientist, you know that you're always going to find data sets that work better for whatever reason. And you don't even know why, right? You don't know why. But whatever did. I mean, you know, there are so many examples of data sets that are quote unquote, wrong because the metadata on them isn't necessarily correct, but somehow they produce better results and data set, that's better. And so you have to version your data. You have to version the data, not just the files, but the metadata around that so that you can effectively go through that process and make sure that you're building the best solution that you can foreign.

If you like this show and you aren't listening to the Changelog, hey, let's fix that bug. The Changelog is our flagship show and we've been doing it for over a decade. Adam and I seek out and interview the people who are pushing the world forward with software. We dive deep into the hacks, the innovations and the leadership required to do what these amazing people do. One recent example is our conversation with Anders Damsgaard, a climate scientist from Denmark who gave us a peek inside his work and how he scratched a common itch he has when gathering academic research from around the web. Here's a dorky moment from that episode.

Are you trying to be right or are you trying to solve the world. World's problems?

Exactly.

Like, if you're a scientist trying to be right, well, then you're right may not actually be the right.

Yeah, exactly.

There's another saying.

All models are wrong, but some are useful.

I like that one. Then there's another saying. All models are wrong. Except for mine.

Mine's correct.

We had a lot of fun with Anders. He's a fascinating guy. Continue listening@changelog.com podcast 378 or search for the changelog on your favorite podcast app and find the episode called Open Source Meets Climate Science.

So, before we got to the break, you were. You were talking about versioning the data and wanted to kind of let you finish that thought. And then I actually wanted to also explore kind of how Allegro is moving mlops in a practical way, like what you're actually focusing on and how you're implementing ML Ops. But if you'd finish your thought on data versioning, we'd love to hear it.

Sure. You know, with respect to data versioning, at the end of the day, you know, we think that that's the, as I mentioned, the holy grail, being able to have a set of tools that enables you to effectively manage your data sets and their versions and effectively also be able to obfuscate the connection between the code and the data so that we can facilitate, for example, the ability to move from a cat detector to a dog detector. Because now you're using a different data set. And again, as a data scientist, you all know that taking one data set with a code and actually switching it to a different data set is not as trivial as one would like it to be. And so those are some of the goals that we set out about to do with Allegro Trains and really the ability to actually switch between the data sets and the code and the models. As easy as plug and play.

Yeah. So what is the kind of, I guess, range of things that Allegro focuses on in its actual offerings? So I know that there's the Trains project, which was mentioned in that tweet that got me interested, that joined the Pytorch ecosystem project. So how does that fit into the wider scheme of what Allegro is offering? And how does a data scientist interact with it? I guess, sure.

What we provide is we Provide a platform or a tool chain or a set of tools that basically takes care of the experiment process. The mlops part of it, the ability to actually scale and actually run things effectively and the data and the full platform which isn't completely available as open source, basically has all these key pieces together, highly, highly integrated. What we've open sourced or what's available as an open source project is the experiment management part of things, which is all about the documentation we talked about the ability to document things, version your models, your experiments, your hyperparameters, everything around that reproduce, compare, etc. And everything that has to do with the I guess, basic mlops. The ability to actually manage a cluster, whether it's on prem or on cloud or combination by a team of data scientists and really self help themselves with orchestration and scheduling etc. And automation on top of that and then some basic. Actually it's not basic because it's on par with whatever else is out there. But data management, or what we call data tracking at least is available in the open source where the enterprise version adds on top of that, much more sophisticated data management, more sophisticated scaling and data pipelining on top of the platform so that you know, companies can actually build a specific pipeline for what they need and obviously you know, the standard enterprise relevant features like you know, user management, permissions, managed services, all that stuff.

So I'm curious as you're kind of describing this and I appreciated you kind of talking a little bit toward what was open source versus what was the enterprise offering. As you look at different potential customers out there and there is a variety of ways they may implement, how they are allocating resources for their own ML ops prior to you coming into the picture with them. You know, some people are strictly cloud based. They may be doing, you know, Google or AWS or Azure, some people are or organizations is more specific, maybe buying like a bunch of DGXs from Nvidia and have a cluster set up locally or some hybrid form. Which of these scenarios does Allegro fit into and if multiple, how does it change how you would implement Allegro?

Actually we fit into every one of those scenarios. Any hybrid scenario that you can think of. And actually the more complex your environment is, the more Allegro Trains shines. And I'll explain. Basically the way that Allegro Trains is set up is you have a server backend that basically manages the processes and records and logs everything and then sets up the instructions for the clients that are basically what the data scientists connect with as well as the agents that run on the machines. That do the actual training. The system is built that you can set it up on any type of machine for training. It could be dgx, it could be any type of GPU by Nvidia, it could actually be cpu, it doesn't really matter. It can sit on the cloud on Prem, any combination on any cloud that you'd like, and it all works. In fact, a significant portion of our customers have a hybrid solution where they have on Prem systems and then they actually burst into the cloud when they have specific times when they need actually more processing power and that becomes really effective for them. We have other customers that are completely on the cloud, you know, and everything in between. Why Lego Trains actually shines, the more that you have a more complex environment is because. So on the first level is that the interface to manage these clusters is really, really simple. You can actually try it out. We have a demo server up on the web. The data scientists actually manage queues where they can set up, you know, the machines. I want one GPU or I need a cluster of eight GPUs or whatever, and it's completely invisible to them where those machines sit. With the enterprise version, we go even further and we provide three layers of software caching and what we call zero data move. So if you have a complex system where you have data in multiple locations, we'll make sure the data goes to the right machine to train. That's close by to it. We'll make sure that there's local caching to it, that it doesn't have to go back again and again. And so the data moves as little as it can. And we can actually, we go even further. You can actually do federated learning on our platform. And so you can actually have data being trained in multiple locations, geographically around the world, and then combined into a single model.

Really interesting. And I think you're kind of getting or hinted at some of these things. But just for my own understanding, it sounds like there's the Allegra Trains server, which kind of aggregates all this information, does that kind of experiment management is kind of, I guess maybe the central brain is a way to think of it.

Yep.

In my understanding, like if I'm, let's say, running, let's say I just have a machine, my own machine, and I have some code on it and I want that to be tracked by the Allegro Train server. I think based on what I was reading, you just kind of decorate that code with a certain snippet that connects to the centralized Trains server. Is that the workflow for that scenario.

Exactly.

Okay.

We try to make it as simple as possible. We dubbed it automagical. There's a snippet of code. It's basically two lines of code. You put just once in your code and the header, basically. And that's it, you're done. Everything is on track for you. You could potentially have the server, the training and your client be on the same physical machine if you'd like. It doesn't matter.

Yeah, actually, I'll reveal some of my cards now because a little while ago. So I have a friend here close by geographically in Indiana, and we. We kind of have regular calls to just talk about AI things because, you know, we both work in companies where there's not that many AI type people, and so we like to share things that we're learning and all that stuff. His name Will shout out to Will out there, if you're listening. But I asked him one of the first kind of times we were talking about his workflow and all those things. We got into this topic of ML Ops and all of that, and he's like, oh, I use this Allegro Trains thing. It's amazing. And I was just talking to him earlier today, actually, and I was like, hey, I'm going to talk to the Allegro Train, the Allegro AI people later today. What do you want me to say? And one of the things he said was, it's just, for him, it's super easy. Like you were saying, pretty low barrier to add this snippet to your code and kind of things happen automatically like you were talking about. And the other thing he definitely wanted to mention was that the team is super responsive and he mentioned raising various things on GitHub and all of that. And the team is very responsive. So great job. You got a very happy user in Will here in Indiana.

Well, thank you, Will.

Yeah, awesome.

But yeah, he was kind of telling me about how some of that works. And then there's also. You mentioned the agents. The agents, those have to do with kind of the more automated runs that happen across a set of shared resources or where does that fit in.

So the agents are. If you basically want to run your code on a remote machine, you basically set up an agent on that machine, whether it's a DGX or a GPU or whatever you have it. And that agent is tasked with. Basically, that agent is then associated with certain queues that you create. It could be associated with either one or more queues. So it's a little piece of code that sits on any machine that is potentially a target for running your experiments on.

So one of the things I'm curious about, and I meant to ask you this a little bit a while ago when you were touching on it was, was some of the motivation that you had for going with an open source business model that builds an enterprise business on top of that. And did you always know that that was going to be the approach you guys were going to take or did you consider any others? And how has that model worked out for you?

That's a very revealing question for us. When we started out, we probably erred on where the market. So I guess one of the things that you do in a startup is you try to time the market, which I think I saw several articles talking about timing being the number one critical aspect of startup success and actually one of the hardest to hit and sometimes even VCs call it luck. But we were trying to time the market because what we had built initially was around the holy grail about the data. And we basically built a system with the thought in mind of, well, companies are now doing development, but they're going to get to scale and they're going to have to be able to manage huge data sets that constantly change. You have to version that, you have lots of experiments, you've got these things running on multiple clusters. How do you handle all of that? We actually set out to build this really big robust system. Then we found out that very few companies were at the stage where they needed this or realized its value. We got back and started thinking, where is the industry now and how can we help the industry progress? And we figured that the right thing to do is to meet the industry where it is, which was, you know, before that skill, and come up and say, you know, all right, so what are the low hanging fruit of things that can bring immediate value to data scientists out there? And that was first thing was the experiment management. And then immediately after that the mlops or at least the mlops in its lighter form, right? Not don't think of a huge conglomerate running, you know, hundreds and thousands of experiments, but you know, even small teams. And we thought that the best way to do that, to really contribute to community, help spur that along, make that something that a lot of people can do stuff better in the way we think it would be better and helpful. And ultimately obviously we're a company, we're about making money. But being able to do that was something that we felt was the right thing to do. That will be a win, win for everyone, it'll be a win for the community. And Ultimately it'll be a win for us because when, you know, the larger companies that do have money to pay and do feel like they need to get, you know, more, they're going to come back to us.

Yeah, that's great. I think as evidenced by I think users and also attention and kind of joining in with the Pytorch ecosystem like in that blog post and other things, I think that that really allows people to, you know, solve a pain point that they really have really, really quickly and, and hopefully does eventually spur them on to, especially if they're part of larger companies or teams, you know, integrate more with your enterprise systems. But it's been amazing to talk today. The topic's very close to what I'm super passionate about. I think Chris as well. And part of the reason why we do this podcast is to talk about those practicalities of how people do their AI development. So really appreciate you joining. We'll link the demo server and the links to Allegro trains on GitHub and also your, your main website which talks about all of your offerings. We'll put that in the show notes for sure. And I encourage people to go there and check out those things and let us know in, in slack or on LinkedIn or other places what you think and how you like what they're doing. But really appreciate you joining near. It's been a great conversation.

Thank you so much. I mean it was, it was a pleasure, it was fun and really thank you so much much for having me.

Have you joined the free Changelog community yet? I'm not sure what you're waiting for. You get Changelog news, email notifications of new podcast episodes, access to our community Slack and Practical AI Channel where fun and interesting AI discussions take place all the time. All for the price of a free hot dog. Check us out@changelog.com Community we'd love to add have you practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by me, Jared Santo and our music is provided by the mysterious Brake Master Cylinder. We're brought to you by some amazing companies who get it thanks to Fastly, Linode and Rollbar. That's all for now. We'll talk to you again next week.

Sam.