OpenAI and Hugging Face tooling

Reinforcement. Learning in general is not like a model. It's a framework in which you can train agents or models. And if you think of self driving car or something like that, if you say I'm going to go from point A to point B in a self driving car, well, there's a whole bunch of routes you could take. It's not really that there's one perfect solution to that problem, it's more about the decisions you make along your route based on what you've done so far and the feedback that you're getting from your environment.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Linode. They keep it fast and simple. Check them out@linode.com changelog Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at rudderstack and we're calling all data engineers to check out rudderstack cloud and start building smart, smart customer data pipelines. Rudderstack is warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CDP with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes and because they're open source, you can see all their code so you don't have to worry about vendor lock in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500, 000 events and pricing scales transparently. From there, learn more and get started at rudderstack.com again. Rudderstack.com that's R U-E-R-S-T-A C-K.com.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where convers around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@Kingjob.com community and follow us on Twitter. We're Racticalai FM.

Well, welcome to another fully connected episode of Practical AI. This is where Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack I'm a data scientist at SIL International. I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

I am very well, Daniel. Looking forward to diving into some of today's topics and as you will explain to the audience, we get to put you on the hot seat just a little bit today.

A little bit, yeah, yeah. Talk about some of the things in, in my life. But yeah, it's exciting, lots of exciting things going on as we, as we wrap up. What is the last weeks of 2021 that seems like it went, it went by fast. I don't know if it did for you.

It has gone by incredibly fast the last two years. Just craziness. It's just like where did they, where do they go? But you know what, we got some new things happening here, some new things to talk about that you can guide us through.

Yeah. So I think maybe one of the things that we can talk about, which we've talked about several times on the show and has been a sort of developing theme that we Visit occasionally, is GPT3 and the OpenAI APIs. I don't know when you saw this, but recently I saw how OpenAI recently made their API available with no wait list. So previously you had to sort of apply on a wait list, they would approve you and then you could use some of their models. And I think originally, even when we first talked about GPT3, they released it and it was fairly closely guarded. I think mostly because of what they considered, you know, a safety process and making sure that people didn't misuse the model in certain ways. And I think based on the blog posts where they talk about them opening up some of the availability, they emphasize a lot of these kind of safety features that they have put in place.

And that might be actually before we dive into it, that might be a good place to start is why for those who are not already very familiar with it, why would this model that they've released need all of this careful vetting and slow rollout and such, because it's been quite a while since they did it. So you want to talk toward that a little bit.

Yeah, I think it's probably first of all, I'm not speaking for OpenAI, but I think in general what people's thought process is around these types of models is that GPT3, just kind of stepping back, is a sort of large scale language model that enables a variety of natural language processing tasks to be performed. And some of Those for example, like natural language generation are very. Performance is very impressive from GPT3. So one thought is, well, GPT3 could be used to do sort of malicious things like create a bunch of fake news type of stuff, misinformation, sort of distribute a lot of this kind of thing as well. GPT3 was trained on a huge amount of data that was kind of crawled from the Internet and other places. And I think the behavior of the model, because all of the biases and other things that exist in that kind of large corpus aren't totally probed out. So another question is how are biases showing up in this data? And if we just say, hey, everyone use this thing to generate text or integrate it in their applications, they might not take that sort of caution in mind if they're just kind of applying it wholesale across the board.

Yeah. And just to name at least the nine categories without all the detail, they note that they prohibit users from knowingly generating or allowing others to knowingly generate with the account the following categories of content. There's nine. They are hate, harassment, violence, self harm, adult political, spam, deception and malware. And apparently they have spent a fair amount of effort putting the safeguards around this so that it can do. It'll be interesting to see whether or not going forward they're able to put safeguards around other models. They release much faster now that they've kind of got some. The infrastructure in place. So it might speed up the ability to get onto new models. Because this has been, I mean, how long has it been now? Maybe a year and a half.

It's been quite some time. Yeah. So it's interesting. I mean they put a lot of thought process into this and I mean they're to be commended for leading a lot of the thought around these areas. I think there's different views on how the community should go about addressing these things and whether it should be via a, an access controlled API or via open source code and models and data sets or you know, how however it should be be approached. But they have put a lot of thought into this. They talk about in the blog post how they are putting things in place to review applications. So applications of the model, applications of the AI before they go live, monitoring those applications for misuse and you know, supporting better supporting applications as they scale and understand the effects of the technology. So very interesting developments here from OpenAI. Chris, have you logged in and tried anything with the OpenAI playground or anything like that?

I've just gotten an account now that they've opened it up. I am ready. So what I'm going to look for you to help me get into this. I have my. I have my brand new account. I have the site that they send you to open, where it says welcome to OpenAI. Start with the basics, I'm looking at their examples. Have OpenAI Codex open. And by the way, for our listeners out there, this might be a good moment. If you don't already have the account, pause the podcast for a second, go grab an account and follow along with us if you're not driving in the car or something. And I would love. I know, Daniel, you've been working with this for some time and if you could kind of guide us through a little bit about what to see. You know, I know you're a practitioner, you're not with OpenAI, but as someone who has used it, if you can maybe give us a little boost on things that you've already learned, that would be fantastic.

Yeah, so we can maybe just kind of look at some of the basic functionality that you can do here. And I mean there's. You can do a lot of things. So this is by no means a complete introduction, but if you kind of log into the API interface, one really nice thing is that they do give a kind of big blocks that give you introduction and examples so you can scroll through those. But they have this cool thing called the Playground. So they have documentation examples and then they have the Playground. Ultimately you can use the API in a variety of ways, including like a REST type interface. But they built this kind of playground to help you try out things and see like what are the types of things that I could do and what's applicable to my specific use case. So if you click on the Playground, it kind of opens up this, essentially a text box. The most basic of things that you could do is just start typing. I could say, Chris Benson is really cool.

Oh boy.

And knows.

I thought, no deception, remember, no deception.

AI. Yeah. So Chris Benson is really cool and knows a bunch about AI. And then I could click Generate and it's going to start generating a bunch of text. I don't know if you want to hear what it generated as a result of that, Chris. So I put in, Chris Benson is really cool and knows a bunch of AI. And then it started saying. I said, I think he's still a bit of a newbie, but he's learning fast. This is his first post. Enjoy it. And then it goes on. I've been playing with this new stuff called machine learning lately and I found it Rather fun. What I'm going to talk about today. So you can keep clicking that button and it's going to generate more and more of Chris Benson's new blog post about machine learning and how he's finding it rather fun.

Oh boy. I was about to put yours in, but since you put such an unrealistic thing about me, I just put in there was a guy named Chris who had nine dogs and I started with that. It took an interesting path. It says he wanted to go for a drive in his new car, a 67 Ford Mustang. It was a nice car and he was proud of it. He asked his neighbor if he could borrow his dog for a few minutes. The neighbor said, sure, take him for a few minutes. Chris went to the garage. So that's what I had. So this is, this is fun. This is. I have a feeling this is going to turn into my favorite Saturday night with a drink or two. Not too many, of course, because we're responsible and no driving. But yet this might be the new game with friends on a Saturday night thing to do after a beer or a glass of wine.

Yeah, but I mean, you can see. And I think the interesting thing is that there's a number of things that you can tweak here right along the right hand side of the playground. You can change the different sort of engines that are available to do this, but then also kind of response length, you can change various kind of parameters, hyper parameters about what's going on and you can show probabilities or not. So there's more here that you can get. And a cool thing is too similar to if you were writing REST API calls or testing them in Postman, you can generate code to make that call. Here in this playground it's similar. You can actually generate code. So if you click on View Code, you can then see, hey, here's the Python code that will actually call the OpenAI API and get the similar response from the engine. And that way you can sort of integrate. So it's telling you how to integrate this sort of text completion into an application and you can, you know, go down and see how to do that with Python or just calling a REST API, which is pretty cool.

Yeah. Or REST API or Curl.

Yeah.

So they give the JSON and the curl.

Yeah. Now the interesting. So this is cool, the text completion thing is cool. But I think the maybe more interesting thing, at least for our team and how we've looked at models like these is that they can be quickly adapted to a very specific task that you are more interested in than maybe just kind of general text completion. So if you look, if you're in the playground and you look at, there's a little dropdown called load a preset. Okay, you could load for example a Q and a preset. That's the first one that pops up for me. And you'll see what happens is the playground will sort of pre fill a little bit of example data for the model. So it's giving a sort of pattern. You can kind of think about this like a very small amount of prompt data or a very kind of few shot example type of thing where you're giving just a little bit of warm up to the model to tell it, hey, this is the sort of thing that I want to generate, right? And so when you pull up the Q and A thing for those that are listening, there's a Q and there's a question and then an A colon answer, right? Q question, a colon answer, Q question, a colon answer. And it provides a bunch of these answers. So the model all of a sudden is realizing, oh, it wants me to generate things that are like Q colon, some question and A, give me the answer. And so now at the bottom you can type in a new question. So we just recorded an episode about federated learning. So I'm curious if it knows what that is. So I'm going to say what is federated learning? Question mark and I'll generate. And it says federated learning is a machine learning technique that allows a single machine to learn from multiple sources of data. Which is actually quite relevant. How did it know how to do that? Well, it's trained on a whole bunch of text, data from the Internet. And you know the world, right? So at some point it maybe knows something about that or has been prompted.

I just want to share with you what I put in. While you were doing that. I put in for question, what is GPT3? And the answer was, GPT3 is a question answering system developed by IBM.

Well, they should work on that.

They might.

Maybe we should switch over to the IBM, IBM Watson API. This is a lot of fun. And actually you could scroll through these, you'll actually see there's examples that they give for summarizing text or text to a command or parsing unstructured data or classification. And so it basically gives you an example of, hey, this is the way that you can sort of prompt GPT3 in order to have it do a new sort of task for you. We've used this for kind of some data augmentation type things. So where we've wanted to generate data in a certain way for a purpose and actually use GPT3 to kind of help us generate that data, and that's been very helpful for us. So that's kind of maybe a general rundown of what is GPT3 and why you might want to to check it out.

Yeah, the just to call out some of the things in that drop down. There's chat, there's Q and A, which we talked about grammatical standard English summarized for a second grader text to command English to French parse, unstructured data and classification. And then it has a more example section. But yeah, it looks good. I'm looking forward to diving into this. Foreign.

This episode is brought to you by Me, Myself and AI. It's a podcast on artificial intelligence and business, and it's produced by our friends at mit, Sloan Management Review and Boston Consulting Group. The question is why do only 10% of companies succeed with artificial intelligence? That's the question they aim to answer with this podcast.

Each episode.

Sam Ramsbotham and Shervin Kotabonde talked to AI leaders from organizations like Nasdaq, Spotify, Starbucks, and Ikea. And as you'll hear, the show discusses tough topics like bias and AI.

We've built technological safeguards and prods. Some of it is actually having the technology prompt the human and say, hey, you're building a model. You have identified that maybe you don't want race as a variable in this model because it can introduce bias. But we see here you have a field that is zip code, and zip code can be highly correlated with race. There's a very human element to this question, but to address it at scale, you actually need to automate the solution as well.

All right, Me, Myself and AI is a collaboration between MIT Slow Management Review and Boston Consulting Group. It's available wherever you get your podcast. Just search me, Myself and AI.

Well, Chris, Hugging Face continues to be the darling of the AI world, does it not? New things all the time. Cool stuff. I've seen a couple of things come out from Hugging Face recently. So for those maybe listeners who are new to the AI community or aren't familiar, Hugging Face is an AI company and actually has a whole host of things that are quite relevant to AI development and research and application. One of those being a model hub where you can get models actually of all types. Now it started with natural language processing models, but now it has vision models and speech models. It has data sets that you can pull. It has spaces where you can host machine learning Applications. It has an accelerated inference API where you can serve inferences from your models. And you can kind of think about this almost like people post their code to GitHub. People post their container software containers to Docker Hub. You can post models and data sets to Hugging Faces Hub and those can be public or private. And so you can version your models and data sets there and serve your models and data sets there for your company. So it's really kind of a. It's becoming a one stop shop for a whole bunch of really useful AI tooling. At least that's kind of how I'm starting to see it.

That's totally right. Hugging Face is one of the names in the AI world that everybody respects and everybody recognizes. They just keep doing innovative things that are cool and they're super user friendly. And so it's. They're kind of the go to. One of the go tos in this space that you always are going to be using.

Yeah. And one of the things that came out recently which is really exciting is a first ML agents reinforcement learning environment on Hugging Face called Snowball Fight. Oh yeah, Yeah. I mean, very relevant for the Christmas holiday season, right, Is Snowball Fight. Maybe people are in some place where you don't have snow. You can now have a Snowball fight on Hugging Face. What was your first impression when you saw this, Chris?

I'm playing with it, right. I'm loading it up as we speak and it is cool. Good names, good graphics. And I am trying to load up the Snowball Fight demo here right now.

Yeah, I was playing it a bunch earlier and had some fun. If you load into it, you can actually play the game interactively. So just to give people a sense of what we're looking at, you sort of load into it. It loads this interface where a game comes up and that's kind of driven by unity and you can move around this little guy in a snowball sort of field and throw snowballs at another avatar on the other side. It's a lot of fun.

The music's playing in my earphones just so that you know, so you know you have a soundtrack going. Festive. Yeah. Oh boy, this is fun. Okay, since nobody can see what I'm doing, I'll stop.

Now we know what Chris is going to do the rest of the day.

Oh, this is it. This is taking over. So don't let my boss know that I'm onto this.

Well, one cool thing is I think we do have scheduled to have Thomas from Hugging Face who created this thing in an upcoming episode. So we'll dive into it more then. But I wanted to mention it here because I think it's cool that Hugging Face sort of made this transition. I mean, at first really centered around chat interfaces and then more broadly like open source NLP and then kind of into open source and kind of general purpose AI tooling and hosting services. And now we see this reinforcement learning piece coming in where the goal is that within Hugging Face itself you'll be able to kind of build and share reinforcement learning environments. Now Chris, I know that we've talked about it a little bit in the past. Reinforcement learning in general is not like a model. It's a framework in which you can train agents or models. And if you think of self driving car or something like that, if you say I'm gonna go from point A to point B in a self driving car, well, there's a whole bunch of routes you could take. Right. And it's not really that there's one perfect solution to that problem. It's more about the decisions you make along your route based on what you've done so far and the feedback that you're getting from your environment. And so in order to train an agent to execute decisions in that environment or make actions in that environment, you need to have a sort of simulated environment that will allow an agent to navigate in that environment, get feedback and rewards, and then be trained accordingly to operate in that environment. In this case to win a snowball fight. So that's ultimately, although the game is interesting, that's ultimately the most interesting thing apart about this is that it provides a route towards training reinforcement learning agents in environments that are shared on Hugging face.

It's a powerful approach. And at a previous company, back when deep reinforcement learning was still fairly new, we were using it for training robots on our team at this other place and was really good there. It's used in video games, it's used in the industry I'm in now in a non specific way. It's used to move all sorts of things we call platforms, things that move around and do things. And that's how we get autonomy to work these days. I mean it is a fantastic tool. There is one I think worth calling out that was a darpa. It's a public DARPA thing. And darpa, it's an interesting place. It is the Defense Advanced Research Projects Agency. And they, they do all sorts of government oriented and military oriented experimentation, very cutting edge. And about a year ago they did something called Alpha Dogfight. And I know I've Brought it up in the past, but they. In a simulated environment, they trained a model to be able to do dog fighting against other models.

And like in planes.

Like, as in airplanes.

Right.

I'm really glad you said that. Just to be very, very clear, I.

Didn'T want people to think about, like, robot dogs fighting each other.

Yeah, we're talking about. Think of the movie Top Gun. Okay. That kind of dogfight, like pilot Tom Cruise, all that kind of stuff. Not the same plane, though. And they trained him. And so. And at the end of it, this was not the Navy, which is what Top Gun is, but the weapons school, which is the equivalent of Top Gun in the air. In the United States Air Force, they had a weapons school instructor. This would be like the equivalent of the instructors at Top Gun go up against the model, and the model demolished the instructor over and over and over again.

Wow.

And so you're talking about one of the best fighter pilots in the world, period, getting demolished by the simulated AI that was based on deep reinforcement learning. Beating it in the simulator. They were using a real. I believe it was an F16 simulator. I watched the whole thing live, and it was just amazing. So this is some pretty cool technology here, and it's evolving rapidly.

But I think one of the things that I've always wondered as related to reinforcement learning is, like, you see the power of that, but then you see. See how hard it is to create these environments in which you need to train the reinforcement learning agent. So in order to train a reinforcement learning agent, you kind of have two choices. You can either, like, train the agent in the real world scenario. Like, if you're training something that's going to fly a plane, that's not very practical. Right. Because you're going to crash a bunch of planes and maybe kill some civilians.

People don't like that.

So you have to create this sort of simulated environment. However, creating that is actually. I mean, it's not my core competency in terms of 3D environments and unity and games and simulation and all that stuff. So I think the idea that there could be a place in which these environments are created and shared more broadly on Hugging Face, to enable people to share things, modify them, update them, train agents on that. I think it's a really interesting concept because I wouldn't personally. It would be hard for me to know where to start in creating this sort of environment if I didn't have a good way to kind of jump off.

So to generalize on that a little bit, it's a great way of accomplishing this in any kind of environment or industry that where the cost of that training would be prohibitively expensive. You know, you mentioned about we don't want to crash planes a thousand times and all the damage. Another area would be medicine. If you want to try new surgeries or alter surgeries and carry procedures forward and you don't want to kill patients in the process, this is an area where you can use deep reinforcement learning for that. There's so many areas out there that are just very expensive to do that. And I don't mean just financially expensive, but you know, loss of life and things like that. You're seeing it all over the place. You're seeing it more and more. And I think simulation is going to become ever more a part of businesses getting done what they are, organizations in general getting done the things that they're trying to get done.

So one more thing that I wanted to mention from Hugging Face, just before we're, we're moving on, they've had a couple of releases that are quite interesting. The other one I wanted to mention was what they're calling a data measurements tool, which is a open source project. This just was released as well. And you can kind of look through it. The Data Measurements Rule, they say, is an interactive interface and open source library that lets dataset creators and users automatically calculate metrics that are meaningful and useful for responsible data development. And so you can kind of, you know, this is new enough that I haven't dug into it, but I think this does certain things, everything from kind of basic things that you might expect from exploratory data analysis like figuring out missing values and description, descriptive statistics, all the way to analyzing biases in data sets maybe related to particular factors like gender or other things. And so I think ultimately what they're trying to do here is create a nice way for people that are maybe using data sets off of Hugging Faces data Sets Hub or creating new data sets to really understand a little bit more about them and document them a little bit better so that people aren't just kind of pulling whatever data set looks good without understanding the implications of that. So it's a pretty cool route that they're going.

I think even as we're talking, I'm looking through some of the, like the graphs they have that do analysis on the data sets that they're offering. I mean, the one that I'm in right now is hate speech 18 and you can go explore things like that. So it's interesting in a, in a bad way. Things I definitely wouldn't want my child to be. It kind of gives you all these different ways of analyzing and measuring. So very, very cool technology. A tool that is long, long overdue now that it's here. I mean, all kudos to them for doing it. Wish I'd had this for a while. So. You know, it's funny, Daniel, as we, as we look at some of these tools right now, it feels like this industry is. Is maturing a little bit in terms of not just having.

Yeah, we have nice tools now, right?

Yeah. Not just having the models, but having some of the tooling we need around it to make it safe and get to what you need to get to for a good output without some of the missteps.

Earlier in the show, you heard a teaser from our friends behind the podcast. Me, Myself and AI, MIT Sloan Management Review, and Boston Consulting Group came together to produce this awesome podcast. And every episode, hosts Sam and Shervin talk to the leaders that are engaged in the theory and the practice of AI.

I remember one project we had, we were training a chatbot, and it turned out we used raw, you know, logs, all privacy assured and everything, but we used these logs that a customer had provided because they wanted to see if we could build a better model. And it turns out that the chat agent wasn't exactly speaking the way we'd want another human being to speak to us. And why? Because people get pretty upset when they're talking to customer support. And the language that they use isn't necessarily language I think we would use with each other, you know, on this podcast.

All right, Me, Myself and AI is a collaboration between mit, Slow Management Review and Boston Consulting Group. It's available wherever you get your podcast. Just search me, Me, Myself and AI.

So, Chris, you had just started to kind of talk about how we're getting to a point where there's a good number of tools that fulfill a lot of the needs that an AI researcher or data scientists. The needs are so diverse, everything from analyzing data sets to serving models to dealing with infrastructure and tracking things. And there's a lot of good tools out there now. And I know that our team. And maybe this is a follow up from a previous conversation that we had about building data teams. Our team at sil, we've been in a sort of growth phase. We're building up a team that's doing NLP research and development, and we've gone through a kind of process of figuring out what tools work well for us and how to kind of plug all of these tools together. And it's it's kind of taken a year to work through a lot of, a lot of those things. But I know there's a lot of things that we talked about on this podcasts around MLOps and GPU servers and tracking models and experiments and all of those things. And I thought it might be good to follow up on that conversation that we had before and talk a little bit about how some of these things might tie together in a real world environment, because we talked about a lot of them individually.

I really want to hear what you're doing at sil. And the reason I say that is for those of us who have followed the show for a while when we've had some of these other related conversations. I know we talked about some of the infrastructure that we had at Lockheed Martin, which was a very large organization and kind of how we approach things, but that's a different set of business drivers and a different set of constraints about how you go and evaluate such software. And so you're coming from a different organization, different size, different constraints, different different budgets, all that kind of thing. I would love not only for you to share what you have done and how you arrived at those, but kind of what some of those constraints were that you had to learn to live within and yet make it work.

Yeah, for sure. So to give a little bit of context, we've now got probably a group of, depending on how you count them, probably 15 or so people between people on my team, plus kind of academic collaborators, plus other close collaborators that are working on kind of a similar set of problems for NLP tasks. And how we sort of thought about this was, you know, we need a way for this team to do kind of a diverse set of experiments. So we're working on all sorts of things from machine translation to spoken language identification and speech related problems, to chat and dialogue. And so we need people to be able to use a whole variety of tooling. But also we want to create some standardization and centralization around how we're tracking experiments, how we're running jobs, and how we're sharing models one with another. And so we're kind of settling down on some of that. Part of that was thinking about, okay, where are we going to run training, where are we going to run inference in our context and how and where are we going to store and track models, data sets and code. So some of that's a little bit easier than others. I mean code we version in GitHub, that's pretty standard for everyone. But also we use Google Colab, a lot because our organization, we use G suite for all of our kind of docs and drive and all that. So we use Google Colab a lot. So if we think, okay, some people are going to have code that lives in GitHub, some people are going to have code that's living in notebooks and Google Colab Colab of course has some GPU resources, but in order to train some of these larger NLP and speech models, we need other more robust GPU resources. So we did end up getting an on Prem GPU server, which is sitting down in Dallas. But that brings up a new set of questions. So if we've got 10 to 15 people and sometimes more distributed sort of all over the US but also in Europe, in Asia, basically all over the world, how do we get them all running things in a reasonable way on this On Prem server in Dallas? You might be saying HPC stuff. I know you worked in HPC stuff for some time.

I was going to say a scheduler, you know, is one of those things in terms of getting the jobs lined up and stuff. How did you approach it? I'm curious what's different from what we. How we addressed it in it.

So I think one of the things that we wanted to make sure was that our people running and supporting the server on the ground weren't really the ones that were going to like. Although they're installing the server, they're not really administrating the server. We don't have a large kind of DevOps and engineering team behind our NLP team supporting us. So we wanted a kind of simple solution that we could work with on our distributed team. So we ended up using ClearML for this. So ClearML, we actually had conversation on the podcast about a similar kind of tool which is in the same vein as weights and biases. That's another very popular one. But ClearML allows you to have a dashboard where you track all your experiments and your runs.

Yep.

And you can actually just by. So I could be running a Google Colab Notebook, import the ClearML library and register that experiment in the Clear ML dashboard. But I could Also from the ClearML dashboard, enqueue a job on the GPU server that will also be registered on the same dashboard and will be sent to the GPU server in Dallas. Sort of like a scheduler. It's not as sort of full featured as like these really robust big schedulers that are used on supercomputers, but it's enough for our team because we can say, well, we have this many queues on our GPUs and you can put things in there and they'll run in that order. We don't really need much more functionality than that. And all of that's registered and all of the input output data is registered in a backing data store in S3. For us, we're running code locally on our laptops, code in Colab and code on the on prem server. All of that is importing ClearML and all of those jobs are being registered in ClearML and all of the input data and the output artifacts like model files are being stored in S3 in a version way. So we know what data was used to train this model when it was created. We can look at the exact run and all of that type of stuff.

Without diving down like the alternatives that you didn't choose because I don't want to do that to them. But kind of abstracting that a little bit, what's some of the reasoning that you. That you chose clearml or conversely that you didn't choose a competitor from a capability standpoint or from, you know, how it was satisfying the need. What were some of the things that made you arrive at this solution being the right one for your organization that were not universal across all the solutions?

Yeah, I think it was a combination of one, just the simplicity of administrating the solution for not systems administrator people. Although we got their help right?

Sure.

The majority of people running and operating this thing are data scientists and NLP people. So we needed something that we could support and not some thing that we would have to know a lot about HPC systems to support. Yeah, so that was one thing and we needed a way to queue jobs and the fact that, you know, we could also integrate that with our, you know, runs on Google Colab and all that was. Was really nice. Now we do use also. So this is kind of for our training jobs. We do also use other solutions for kind of persistent data pipelines. We use Pachyderm for that, which actually allows you to kind of create and subscribe to data sources and pump those through to update data sets in a version sort of way. So we use that for other purposes. But clearml gave us that sort of experimentation for NLP research. And what we found is we can run our experiments there, but then we can also kind of. And this is where I wanted to kind of get with the connecting the pieces. So. So we're using notebooks, we're using GPU server, but then these models and data sets that we're creating in our experiments we can upload those now to hugging face datasets Hub models, Hub, those are versioned in that model hub. And then any inference we can do, we could either use the hugging face inference API or we could call down the model using the hugging face libraries into Python code where we could serve that model in a custom application of some type.

Right.

It provides a really then flexible route towards inference as well. Because if you can now store your models in a sort of standardized way in a hub and have a standardized way of serving them, it lets you have kind of consistency and efficiency around how you do that bit as well.

I'm curious about one of your constraints in general, aside from because I understand the solution you've taken us through so far. Are you able to keep things kind of cloud based or at least locally on that server in its immediate environment? Do you have any kind of edge considerations that you have to push to. Is that part of your requirement or not?

So it depends on the project. So some of our projects, the end target for deploying these models is it some inference server in the cloud. So that's part of what we deploy to. And that could be like I say either a custom inference server that we've built or something that we're deploying to an inferent hosted inference service. But we also deploy to edge devices, particularly some of our speech solutions we're running on edge devices that does have other concerns like you were talking about. But the nice thing is if you've got your models for the most part on our edge devices, they are connected to the Internet so we can ship a Docker container to those edge devices. And if we're for example downloading a model from model hub or S3, it can directly download that version of the model from there at runtime. So we have a little bit of flexibility there on the edge devices because they are connected to the Internet.

Gotcha. I'm curious if, and I know a while back you took us through some specs in a previous episode for a GPU server that you had. Is this the same one that we're talking about or is this a different one?

That's my one off DIY build that you're talking about. The previous one, which was an interesting build and I still use it more for sort of the one off things that that I'm doing.

Yep.

This server is. It's a rack mount unit with a 1/ hundreds in it. So a 100 GPUs which is another thing is nice about that solution is we can the a100 GPUs have this mig technology which lets you split up the GPU into sort of multiple virtual GPUs. Yeah, which is really great because we don't all the time, yeah, we don't all the time need to run like big jobs on, you know, whole combinations of our GPUs. We might need to run a whole bunch of small training jobs.

Right?

Yeah, you get a lot more utility.

Yeah, exactly. We can split them up and sort of slice and dice them the way that makes sense for, you know, the season of research that we're in. I think that's a really beneficial thing. So yeah, I would highly recommend people look into that technology if they're, if they're able.

It is. That was my single favorite feature when the A1 hundreds came out was that ability to do that instead of having, you know, having to use them all. It was otherwise once you had the A1 hundreds to look back, felt very inefficient in terms of how you were going and doing training. I'm curious, I want to go back though and hit a really basic question that people are facing a lot, and that is how did you determine your crossover for the organization for when it needed to have an on prem server versus when it could use cloud resources, whether that be Google Colab or AWS or any of the others that are out there. How did you make such a determination?

Yeah, I think it was basically when we looked at for the year, kind of an estimate of the sort of scale of training that we would need to be doing for our models. And we realized that like we would be making back our cost on the GPU server with the amount of training that we're doing. I think it's partially that. So there was a break even point there. But then also it was when we realized kind of we could develop also some operational efficiencies by having people kind of centralize their jobs on this server in these queues by kind of combining all of these people's work. Rather than like this person over here spinning up a GPU server in the cloud and this person over here spinning it up and this person over here spinning it up and they're not utilizing all of those GPUs to their full extension. Right. By utilizing like a more of like a job queue thing, then we could do that. Now you could also spin up a cloud server and implement like similar queues and such. But yeah, so there's a variety of options. There's also increasingly favorable options for running these things in the Cloud. So it's still not a kind of story that's, that's finished, I don't think.

No, it's definitely evolving. But we are still at that point where there seems to be a crossover is you get more capable, sophisticated, and you're running models on a more consistent basis and not just doing it for a short while, you know, each day kind of thing. And it's sitting there not running during those off hours. Once you get to that point, it definitely seems to pay in the current economics to go there. You go that way.

Yeah.

So as we turn to other, other things, do you have any, any learning resources we're sharing today?

Yeah, I mean, I don't have that many, but I did want to point people to a quick thing that I saw which I do think was really cool for people to Explore, which is pandastutor.com so pandastutor.com if you go there, this is a way for you to visualize your Python Pandas data transformation, which is really cool. So for those that don't know, Pandas is a way to construct data frames of sort of tabular data in Python. It's kind of ubiquitous in the data science AI world. It's very common to use, but it's so powerful. There's so many data transformations that you can do. Sometimes it's hard to visualize and strategize about like what your code's actually doing and some non intuitive things can happen. From my perspective, a really cool thing that this is addressing is helping people gain that intuition about what certain transformations and Pandas code does to data and it does that in a very visual way.

It does looks pretty cool. I'm looking through it now. Thank you for sharing this.

Yeah, for sure. There's also a bunch of things going on in the month of December around the advent of code and 27 days of Jax. I've seen that. So there's a whole bunch of every day of December trying to do some coding thing out there. So if you're into that, you could look up some of those things. Those are always good learning experiences. But yeah, I wanted to share this Pandas thing that I, that I ran across.

Well, thanks. You know, you've really taken us on a bit of a Tour today between OpenAI and Hugging Face and then the how you guys put together your current approach to training. So thank you for sharing that.

Yeah, for sure. It was fun, Chris, and hope you have a good rest of the week. I'm gonna put on my heavy code and go through the cold back to my apartment because now it's winter. But yeah, appreciate the conversation. Looking forward to chatting next week.

Absolutely. Talk to you then.

That's our show. Thanks for listening. For more like this, check out our master feed. It is all changelog podcasts in one easy to consume place. Let your podcast app snag everything we produce and then pick and choose which ones to listen to. Subscribe today@changelog.com master or just search for Changelog Master in your podcast app of choice. You'll find it. Special thanks to Breakmaster Cylinder for providing our music and to our longtime sponsors Fastly, LaunchDarkly and Linode. That's all for this week. We'll talk to you again next time.