Building a deep learning workstation

I used the air cooled CPU cooler. Not like I didn't feel quite comfortable enough yet to run like all sorts of liquid through my machine. I figured that I, you know, knowing myself I would break a tube and like ruin everything in my computer. And actually I looked, there was some study I found I actually probably couldn't find it again. And maybe it was just confirmation bias, but I found like, I think it was some YouTube video showing how the air coolers, the good air coolers like actually cool better than a lot of the cheaper water coolers or radiators. So that gave me some confidence there.

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com Our feature flags are powered by LaunchDarkly. Check them out at launchdarkly.com and we're hosted on Lino Cloud servers. Get $100 in hosting credit@linode.com changelog what up friends? You might not be aware, but we've been partnering with Linode since 2016. That's a long time. Way back when we first launched our open source platform that you now see@changelog.com Linode was there to help us and we are so grateful. Fast forward several years now and Linode is still in our corner behind the scenes, helping us to ensure we're running on the very best cloud infrastructure out there. We trust Linode. They keep it fast and they keep it simple. Get a hundred dollars in free credit@linode.com changelog Again, a hundred dollars in free credit@linode.com changeLog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com and follow us on Twitter. We're Racticalai FM.

Welcome to another fully connected episode of Practical AI where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson. I am a Principal Emerging Technology Strategist with Lockheed Martin and with me, as always, is my co host Daniel Whitenack, who is a data scientist at SIL International. How's it going today, Daniel?

It's going great. It's a little bit interesting. You know, Chris, we haven't done one of these fully connected episodes in a while where it's just you and I so for those listeners that have been with us for a while, you'll remember Chris and I used to somewhat frequently, maybe every few episodes have an episode with just Chris and I and no guest and discuss some news things or learning resources or random topics. But we haven't really done that in a while, so it seems so nostalgic.

So it is. But this is a good one. This is one we have been talking about doing for quite a long time because you were working on developing your own AI workstation and you built it from scratch and we heard a bit of progress across several episodes there.

Yeah, in the intros to a few episodes I mentioned that I was building an AI workstation and actually ended up. So I have a few computers sitting around me that one of those is that workstation I was mentioning. Another one is. We can talk about that one as well actually. But yeah, it's been a bit of an interesting journey and I think, you know, at least news wise, maybe some people are thinking about this these days, but Nvidia's latest release with their 30 series GPUs. Actually even today I was talking to someone who was trying to get their hands on one and there's like none anywhere. So this is. What is this? November 10th as we're recording this of 2020. And I forget when they released those. I think it was in maybe September, but they've been pretty scarce. And I think what happened is the. So there were the lower tier ones I think primarily geared towards gamers and those basically sold out instantly. And so then the ones that were like, I think the 3090, the higher tier ones that were geared towards like AI training and stuff, the gamers then just started going to those and basically everything sold out for anyone. And this morning I was on a call with a guy and he was like, we were taking the call while he had Best Buy pulled up and like he was getting notifications when they would like release some on Best Buy but he never got one. So like he, you know, went into the thing and pushed the add to cart and then they make you wait and then by the time you're like they're waiting in the queue is done, then you know your cart opens and it's sold out and no longer available. Yeah, it's kind of a vicious cycle like that.

So you went virtually shopping with him on his.

Yeah, yeah, he screen shared and everything. So we got the experience together and that was.

You don't just have your own frustrations, you get to share in his frustrations. Exactly. Very Good.

So yeah, for maybe listeners that are newer to AI, which we'll go into this a bit with the workstations but a lot of people use GPUs and specifically Nvidia GPUs to accelerate their workloads. In particular, training is a big thing to accelerate inference as well, but training mainly. So when people are building their computers for training deep learning models or AI models, a lot of times they're in the market for a GPU and such are not available at this time as we've been discussing. So that makes things a challenge. Actually I mentioned I have the one workstation, so I guess I can kind of talk now about the things that got set up. Go for it. So I have the one workstation that I built that has two GPUs in it and I've been using that for training models. And then I've got a separate one which is it doesn't have a GPU in it, but I've been using it. We've been training models for intel based edge devices. And so I've got this intel machine which actually is fairly beefy, but to test those models on an intel based architecture without a GPU for inference. But also it's got a bunch of RAM for like pre processing and things like that. And I can kind of talk about how I've experienced both of those, but those are the two things that I've been using recently. And so I really haven't done much other than connect from my laptop to those machines while I've been working recently. And that's some interesting characteristics, I guess is the right word. I don't know if I would call them features, but interesting workflow implications as well.

Well, you're diving into that now, so you gotta, you gotta tell us a little bit about what.

Yeah, I mean this is not really AI or computer. Well, I guess it's. I guess it's computer related. But so I SSH into either one of these machines or actually on the workstation. Sometimes I SSHN and just run jobs and sometimes I'll connect to a Jupyter server. The other machine I can SSHN or actually even just connect a monitor and work on it directly. Yeah, we have Comcast Xfinity Internet here and you know, just like the home Internet network, I have the Xfinity router. So just the one that comes from the isp. And I guess they assume that most people aren't going to be wanting to like connect in that way or like when I am at my wife's business, she owns a manufacturing business. And when I'm there and I want to connect back home to my AI machine, then I have to open up a port in our home network. Right. And then do this port forwarding thing. And all of that's kind of, I would say, difficult in the Xfinity firmware that's on that router. So our listeners may also remember that I installed a new network at my wife's business, which has some really great network appliances and ubiquity gear. And so I'm thinking actually of taking both of these machines and maybe just like putting them in the IT closet at her work. And then whenever I need to connect just SSH into that network and find the computers, I think that's actually the better solution. That's like the workflow things that I mentioned in terms of connecting to them.

Gotcha. And for what it's worth, for those out there who may not be intimately familiar with that, the two seconds explanation on ssh, it's a secure connection to a remote computer. And you talked about going through your router and having to open ports. And those are like doorways you might think of on your router where you have to, you have to open them, you have to have access to that and they have to be able to connect to each other. So there's a lot of plumbing. It's. Yeah, it's Internet connection plumbing to make work.

Yeah. And when you spin up a Jupyter server, let's say locally type like jupyter space notebook on your. On your terminal, that spins up a local server. And you might notice if a tab pulls up in your browser, normally it'll say like localhost8888. Yep, that's the host and the port. So I'm just connecting to that host running the jupyter server. But it's a remote computer, not the local host on the. On the same machine.

Gotcha. And I have a question before we move off of this. I know this isn't the primary topic, but do you ever have a reason to get on and do the actual work on the workstation itself locally, or do you always come in from your laptop through a port and use the workstation as a remote server in that way?

Yeah, I haven't really done that much. I don't really have much installed on the workstation other than Ubuntu is the os, Linux os, and then I have Docker installed and then the like CUDA libraries and such for the gpu. And so there's really not much. It's not meant to be a, a primary work. Primary workstation with all sorts of things installed on it necessarily. So yeah, I haven't really connected to it in that way outside of when I very first set it up and booted it up. Before all of that, I connected up a monitor and keyboard and mouse and that sort of thing.

Gotcha. So you're kind of using it as a remote SSH server to do the work and you're still on your laptop when you're doing that, but it's providing those resources.

Yep, exactly. And I mean there's no reason why you couldn't use it in that way. But also if you're doing like, if you're using this workstation primarily for those types of workloads, like training workloads and that sort of thing, if you're pulling up a monitor and you have like 17 Chrome tabs up and like it's eating away at the memory and then like you're also have, you know, the GPUs then potentially supporting various graphics operations. Not that it's consuming a lot in terms of the same amount that training consumes, but it's just not optimized. So yeah, there's really not anything running on there other than those few things that I mentioned.

Gotcha. So I guess what was your motivation for doing it this way? Because we have talked and anyone who's listened to a number of our episodes prior to you doing this, we kind of for a while were talking about really focusing on you. We have all three major cloud providers out there with all these services. I remember us having conversations going why do it elsewhere? And stuff. Clearly you had a change of heart somewhere along the way. So what was it that motivated you to do this?

That's a great question. I think there were a few things. It wasn't any one particular thing. The first, which is not really practical in any way, is just because my brother in law was building a new gaming computer and I was kind of watching him do that and we were talking about it and I kind of realized that, you know, back in college, I guess it would have been, would have been the last time I built a personal computer of any kind.

Right.

And I used to do that, you know, fairly frequently, especially since I was in. I worked as an IT person at my college and so I would do that fairly frequently or dig into the guts of computers fairly frequently and switch out parts and whatever.

I did two in college, but that was a different time.

Yeah.

Yeah.

I won't ask about that.

Yeah.

You don't want to, you don't want.

To go back to the age of dinosaurs, you know? No, no, definitely not.

I really enjoyed doing that back then and I realized I had, I just hadn't done it forever. And so part of it was like all of these new parts and how things have updated. Everything's different now. There's a different spin on all sorts of things. You know, hard drives are different. They don't have like these spinning bits in them, which they always did.

Yeah, platters.

Yeah. I remember in high school, college, we would take old hard drives and take the tops off and then you had like action figures and we duct tape the action figures to the drives and then like put little swords or something in their hands and then, you know, spin them both up at the same time and the last one standing would, you know, win the prize or whatever.

So kind of a centrifugal force contest, right?

Yeah, exactly. Who survived the longest can like. Yeah, yeah, got it.

Very good. At least you found a use. You're the only person on the planet who has found a use for these old hard drives rather than just chucking them.

I will say that the network video recorders and a lot of file servers now, I think still use those types of drives. They'll have 12 slots for regular, you know, spinning hard drives. You know, when you have like that much data, it doesn't need to be accessed quickly, just cheap storage.

Gotcha.

But I totally forget where we are. Oh, the motivation for got into action figures on hard drives, fine.

You know, come back from the action figures. Why build.

Yeah. So the second reason was a lot of the stuff I'd been doing up until this summer was natural language processing. Text based though. So text corpora or data sets are generally fairly small in terms of size.

Compared to audio or, or video. For other non nlp.

Yeah, compared to audio or video. But I've been doing a lot of speech projects this year and so also having a system where I could run something, you know, for four or five days and do that over and over and over and over without, you know, incurring the cloud costs.

Yeah. So there's a crossover.

There's a break even point. Yeah, there's a break even point. So even I think when I did the research, there are some really great affordable. I mean, depending on your definition of affordable. But there are some cheaper manageable solutions for using GPUs in the cloud. One of those was paper space, which I was using before for longer training runs. Um, that's a really great system that integrates with a lot of things now, I think even like hugging face Transformers. I saw an article about an integration there and that's easy to use. I like the design pattern there. Yeah, and they have integrated notebooks and all of that. There's also Google Colab and things like that where you can even get free GPU usage. Yeah, our organization uses G Suite and so we have access to team drives and all of that. So I would occasionally save, you know, collab notebooks and other things and work on them with people in that setting. Yeah, so there is a break even point and I think I kind of calculated out a bit of what we were going to need to run and we were sort of right at that break even point and I knew, you know, we were kind of pushing our limits in terms of the cost, so.

So you actually did an analysis though, at least a back of the napkin kind, right. To figure out what your breakeven would be.

So like it was something like, you know, over a summer, you know, we were going to spend at least, you know, 8 to $12,000 on GPUs in the cloud for training. That would be kind of the lower range. And you know, when I started looking at blog posts about now some deep learning workstations that you'll get from that are like pre built, we can talk about that too, are not going to be a cost savings with respect to that. But you can also, if you sort of roll your own and build your own, you can get into a lower cost zone with that and that's one of the things that we can talk about moving forward.

Gotcha. So in the end it was just a simple. But was that the whole enchilada, so to speak, was that was basically you figured out that your crossover, your break even point made it make sense after that and then as soon as you got to that point, that was it. Anything else that contributed, I think those.

Were the two main things. I also knew that there were a couple other people onto our team that would need this sort of resource. So, you know, when I'm, you know, not running things, it could also benefit others on the team that needed GPU resources and you know, we're either going to pay for those or get their own machine. So.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal and helps sustain our production of practical AI into the future. Simply follow the changelog link in Your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog is better.

Okay, so now that we've kind of talked about why you chose to do it, let's actually get into the machine itself a bit. Talk about what you built. So, you know, in any way you want. Just take us through a rundown of what it is that you've done and maybe some of the choices along the way.

Yeah, sure. So I'm actually going to, while we're talking, I'm going to pull up a couple links where I started my research. So originally when I was looking at this, I was like, you know, do I really want to sort of roll my own workstation and get all the parts and do all that or just get a really cool workstation that is still going to be really cool and sit by my desk, but you know, isn't going to require sort of all of that work? So I started looking at places like lambda Labs or System 76. They have these sort of pre built deep learning workstations with, you know, four, eight GPUs, different, you know, configurations. And I really like the look of those and what you could do with them. But when I started calculating out the cost, it was going to be, you know, pushing 10 to $12,000, which rightfully so they need to make, you know, profit on those. And there is a significant amount of work in terms of, you know, fabrication of what they're putting together and like the design in that and making sure it's maintainable and all that. But I was looking at definitely a lower price tag than that in terms of what I could accommodate. So I found a couple of blog posts that appealed to me right out of the bat. The first one of these was from a guy named Jeff Chen, who is out in California I believe at the time he wrote the blog post he was at university doing research. This from 2019 and the title is how to build the Perfect Deep Learning Computer and Save Thousands of Dollars. Which is obviously very appealing to me. The next set of blog posts was from a guy named Curtis Northcutt. Actually he has a few different builds, but one of them he talks about, the best four GPU deep learning rig only cost $7,000, not $11,000. So if you remember, I was kind of looking at those pre built ones around $11,000. I didn't think I needed four GPUs in terms of the initial workloads I was going to run. So I thought maybe I could also Bring it a little bit under that. So under that price point. So my thought was I could follow something like these blog posts talked about and get a workstation up and running for much less than what we would be spending over the summer in cloud costs if we were to run our workloads there. But also I wanted to get it expandable for the future, I guess. So start out with maybe two GPUs, but also have it expandable to maybe four GPUs because I knew, you know, our workloads might expand or more team members might come on the team eventually and that sort of thing as well.

So quick question there. You, you mentioned, you know, kind of that you knew that you really only needed two right now, you wouldn't need the four. As an aside, real quick, how did you make that evaluation?

Yeah, not in a high tech way.

Okay.

And so my logic on that front was as follows, which is maybe not the best logic to follow, but.

Oh no, I can hear him caveating all the way through this. You just finger in the air kind of thing.

Yeah, yeah. And I should. Maybe I'll just make a general disclaimer that I'm not a hardware person. So a lot of the choices I made, I'm sure I'll have people come into our Slack channel and be like, why did you do that? Why did. I'm sure. And I already know a couple of things that I would change if I could, which we can go into those later.

Yeah, we'll get into that in a bit.

So this is a learning process for me. I'll just disclaimer that. But my thought process in terms of the two GPUs was that I didn't yet want to worry about multi GPU trainings consistently. I've found in the past that generally the time I sink into getting multi GPU accelerated training working, I mean, that's. It's more complicated and the time I put into that can be saved by just sort of loading that workload into a single higher tier gpu. So a lot of times I find that it's most easy to just do single GPU training but on a higher tier GPU than do like multi GPU trainings, all with lower tier GPUs, if that makes sense.

Got it. So, yeah. So let me ask this. Do you. So is it fair then to say that your training runs are, since you have two GPUs, you're doing two concurrent training runs versus doing a single one where you're using both of the GPUs there.

So this is the second logic with this is that I knew that primarily on these speech projects I was working with a single other main developer on this work. And so my thought process with that was while I'm running something on GPU 1, he can be running something on GPU 2.

Got it.

So that's where the two came from was the fact that I assumed we would be doing mostly single GPU runs and that there were two of us.

Okay, no, fair enough.

Yeah. And you know, at some point it would be interesting for me I would like to try some multi GPU stuff. When I looked into this, there's a whole like rabbit hole you can go down on this front because like certain cards. So I actually have two different GPU cards in the machine. One is different from the other one. There's a 2080 Ti and a Titan RTX and those you actually. So there's a special link between GPU cards called an NV link, which is I think in my understanding, the best way to communicate data between the two cards. Yeah, the fastest way. Yeah.

And that's an Nvidia technology between the two cards for networking.

That's my understanding.

Just for those who aren't familiar with it, it's proprietary technology from Nvidia that gets you that great throughput for training purposes.

So eventually I'd like to try something with that. I think my understanding is that if you don't use that link, the sort of standard link through the motherboard, you don't always get the sort of boost that you might expect because of the communication bottlenecks between the cards.

Yeah, and that's my understanding as well.

Yep. So eventually I might like to try that, but it may depend also on future workloads that I'm running and what's needed there.

Gotcha.

Yep. So that, that was the logic behind that, which meant that I needed a motherboard in the workstation that could support two GPUs. But I also wanted it to be expandable.

Expandable. I remember you mentioned that.

Yeah. So the blog post by Jeff Chen.

Who?

Jeff. If you're listening to this, thank you for your blog post. Reach out if you are listening. We'll grab you a shirt. But it talks about all sorts of things and probably some of it is. So I noticed some of the things, like some of the parts had even been updated since he wrote the post, but most of it was still relevant when I went through. And he also talks about trade offs and he goes down and he talks about expandable. So making sure. That you build things expandable and that sort of thing. So that's where I kind of got that idea. What I ended up doing was getting a motherboard. Which actually this reminds me. So in our show notes, I think one of our listeners asked for this at some point in our Slack channel too. I'll post the part list that I used in our show notes if anyone's curious about the parts that I use. So don't feel like you have to frantically write down part numbers or something. And I would probably mispronounce the names anyway, but I'll put the parts list in our show notes. But it was from Gigabyte motherboard and I don't know how to say this. This is going to be where my ignorance comes in. A ros aorus A O R U s Good enough.

I have no idea.

And so that. That supported actually four. So I could. I could run four. Although it seems like a tight fit. I don't know if it's just like me and my inexperience and not knowing that, like things are that tightly fitting, I could definitely fit four, I think. But right now I have two spaced out. So there's a slot in between the two that I have in there. There's actually a slot on the bottom and then I have one in the next slot up and then there's this empty slot and then the final one. So they're actually spaced out a bit. If I were to put all four in there now, seeing that they would be pretty tightly packed, so depending on ventilation stuff, that may be an issue.

That's what I was going to ask, you know, coming at it from someone who also does not do hardware and has not built a system in quite a while. But, you know, having previously put Nvidia cards in that had big fans and things like that, is that still, when you talk about slotting it in, is that including those ventilation capabilities or is that without that?

Yeah. So as I learned, there's a bunch of options here as well. And there's two main types of cards. There's actually a blower style card which pulls in air from one side and actually shoots it through the card and out the back of the computer. And then there's another style which people might see in pictures where there's like two or more fans on the bottom and it just sort of blows down. Like it blows from top to down and draws air away from the card. So this is actually really interesting. On Curtis Northcutt's blog, he talks about double deep learning Speed by changing the position of your GPUs. And when he's talking about position, he's actually talking. He has a mix of blower style and non blower style GPUs. And so what happens is when you put one of those non blower style GPUs on top of another one, it actually just blows the heat into the other one and heats it up and it degradates performance. So the positioning is actually really important here in terms of managing the heat on those. So you kind of want to shoot for those blower style. At least depending on how many you're going to have and how you position them, you're going to want to worry about that positioning and the flow of air, obviously. Actually, you know, this is one of my new moves as well, but I ordered case fans and I thought I was just getting like a case fan, but the, the order that I did on Amazon, it was like actually a pack of four. So I, I ordered what I thought was two case fans and I ended up with like eight.

Okay.

So I just plugged in as many fans as I could into the motherboard and yeah, I, I forget how many I ended up with, but there's, there's a lot. So that's it. Yeah, the, the ventilation and all of that is definitely a big deal with the GPUs.

So are there variances though, in what you saw? And maybe you didn't have a big enough sample, but in motherboard design, I mean, it clearly that can have a fairly profound impact on what your GPU can do or GPUs can do.

Yep, that's a great point. So I'm speaking a little bit outside of my domain of expertise here, but this is what I learned off of sort of blog posts and that sort of thing. One of the things you want to look for with your motherboard is what it talks about with PCIe lanes.

Yep.

So this is, in my understanding, the sort of lanes of communication between your CPU and various other things that are connected to your motherboard, like your hard drives or other things. So, right. What's going to happen is when you start adding GPUs, you're going to start filling up those lanes. So according to, you know, the Jeff Chen blog post, as a general rule, this is probably not true, maybe across the board, but as a general rule, you might think that each GPU requires at least eight of those PCIe lanes. And so what you have to do is you have to find a combination of cpu because that determines some compatibility with the motherboard. So combination of CPU, then a combination of motherboard that gives you enough PCIe lanes to support the number of GPUs that you need to support. So there's a little bit of arithmetic there, nothing too complicated, but you have to be sure because some CPUs and motherboard combinations, the pairing will not support the needed number of PCIe lanes that you need to run those and then also run. So I added a solid state drive for my hard drive and I've actually, on my desk I have a second one because I already filled it up. That's one item on my didn't do it right list.

We'll get to that in a few minutes.

So I have a solid state drive and that, that also, you know, takes up some of those. I have an M2 solid state drive which is just a little bitty, you know, gum stick sized hard drive and that requires a certain number of lanes. You also want to think about, you know, your storage and that sort of thing and how much RAM you need to support in the machine as well. All of that stuff went into the motherboard and I just kind of took some recommendations from various blog posts on that. Again, I'm not, I don't follow this stuff religiously, so I don't, I don't know all of what's out there, but I was able to find a combination that worked.

Do you have a current list? I know you're going to put it onto the show notes, but I'm just curious if you can highlight each of the, you know, what the RAM was at least size wise, that kind of stuff and just give us the highlights without going too deep.

Yeah, here's the highlights. The motherboard I mentioned, the CPU I got was actually. So this is one thing that probably looking back, I would change for some reason at the time I got a Ryzen Threadripper cpu. And not that it's bad, it actually works quite nicely. So it's an AMD chip. I think looking back, especially with my experience with the other intel machine that I have, I would go back and change that to an intel cpu. And we can talk a little bit more about that here in a second. But that was the CPU I used then I ended up using a total of four sticks of RAM for a total of 64 gigabytes of RAM. And then I have one of the M2 solid state drives, which is a terabyte Samsung drive. One of the other things you have to be careful about with this is the power supply. You need enough power to power the GPUs and all of that as well. So that was actually one of those things that turned out to be more expensive than I thought in terms of looking around and finding an appropriate. I ended up choosing one that actually someone already gave me some feedback I think in our Slack channel. Maybe it's not the best power supply, but it does the job. A rosewill Hercules and then some case fans. I use the air cooled CPU cooler. Not like I didn't feel quite comfortable enough yet to run like all sorts of liquid through my machine. I figured that I, you know, knowing myself, I would break a tube and like ruin everything in my, everything in my computer. So. And actually I looked, it's. There was some study I found. I actually probably couldn't find it again. And maybe it was just confirmation bias, but I found like a. I think it was some YouTube video showing how the CPU or the air coolers, the good air coolers will actually cool better than a lot of the cheaper water coolers or radiators. So that gave me some confidence there.

So now that you've kind of talked about these components and you started to tell us a little bit about some of the things that you've learned and might change as you've gotten some education, kind of go back through and what would you do different? You know, you've already started on that line, but I'm kind of curious. There's a big learning process that you had to go through this and neither one of us are hardware people. So what kind of what do you know now that you, you didn't know before?

Yeah, so there's a couple things that probably the funniest thing is my case doesn't close.

So does that affect the airflow then?

I think that, I mean, if anything it improves the airflow or at least that's what I tell myself. So I got a case. Actually, I think it's the same case that, from that blog post that I keep mentioning and I was like, well, I don't want to worry about like finding a different one. I'll just use the same case. But what I didn't realize is so that each of the GPUs, at least the certain GPUs require power directly to the side of the GPU. And the power cables I got with my power supply are fairly rigid.

Yeah.

They're not like super bendy. So when I bring them in the side, they kind of stick out the side and I can't close my case.

They're that rigid. Wow.

Yeah.

Yeah, so I mean they're, yeah, they.

Do make these little like 90 degree connector deals you can buy. Basically I just, I got it up and running and set the side of the cases, you know, to one side and I haven't really looked back. Probably at some point maybe I would look into those but. Or like bendy cables or just a larger case. It's a pretty big case, but it's that point like between the motherboard and the GPUs and the in the case that width, it could actually expand out in the case the other way. So that's one thing, just that spacing. But I mentioned this other machine that I've been running which is I actually really enjoy doing stuff on this machine. It's a Intel based machine that we're using to as a target system, you know, to test out certain of our pre trained models for inference and test out our applications on there. And we have some speech applications or testing and that sort of thing. It's one of these intel based nuke computers. They're like little small computers and they have, I think the one I have has a i7 intel i7 processor in it, no GPU in it, but it has a lot of RAM. So 64 gigabytes of RAM. And I found a lot of the sort of speech processing and pre processing stuff on there is super fast. I really like the performance on there. Not that it's amazingly slow on my workstation that's sort of our target system for our applications is an intel based system. And so at the time when I was building the workstation, looking at what was available online for building these, you know, I could get an intel based motherboard and processor, but it was a price hike.

Right.

And looking back I would tell myself to just go ahead and take that price hike because if that was the case, I wouldn't constantly be moving models around between these two systems to do my testing. And you know, the intel, the way I understand it, like the Threadripper has, the AMD chip has more cores. And so if you're doing a lot of multi threaded stuff, it's really good. But the single core speed on the intel processors is higher. And so depending on what your workload is, then you could actually get a performance boost even though you have fewer cores but with a higher core speed with the Intel. So all of that stuff, it seems to work a little bit better for me. And this is just my own personal experience.

This is all inference you're talking about here, right?

So inference, but also pre processing so cpu, I'm doing all my pre processing on this machine too. And pre processing is also part of that CPU workload that you have to consider. And I use Intel's OpenVINO toolkit to optimize the models for running on Intel CPUs. So if all of that was in a single workstation, like I had the intel cpu, the motherboard, and I could do the pre processing, I could do the model optimization, all of that together, rather than sort of swapping back and forth or doing that in separate places. I think just the saving in terms of my workflow would be useful. So that may be one thing to keep in mind. Is like those other pieces of your workflow, like optimizing your model for certain hardware or pre processing as well. Those are things that you need to think about, you know, those are equally valid things to think about when you're designing a system.

So before, just as an aside, since you're kind of touching on it for a second, if you look at what your existing workflow is from beginning to end and the choices that you've made, what are each of the big pieces, like what frameworks are you using and all that, just to throw it out there so that people can get a context for it.

Yeah. So we're using both TensorFlow and PyTorch in our work. Depends on the certain application. But we have a couple of different products that we're developing that include multiple models in each of these products. Some of them are TensorFlow models, some of them are Pytorch models, and we have both text and speech pre processing elements that are in there as well. We generally save most of our training Data in Amazon S3 or Digital Ocean Spaces, some object store. And then we'll pull that down to the machine and pre process it and then save that pre processed data for training and testing and evaluation. We'll save that actually on the workstation. That way we don't have to constantly be pulling from some remote source for the data. And then once we have that locally, then we can iterate on our model training, whether that be with TensorFlow or PyTorch. That's all done within Docker. Like I mentioned, we don't actually have really much at all installed on the machine itself.

Right.

So we pull down TensorFlow or PyTorch or our own custom Docker image run that. We haven't had any issues in terms of connecting to the GPUs with Docker. So that that's worked out great.

Are you basing it on. On Nvidia Docker containers?

Sometimes, yeah. So sometimes we have our own customers containers we're using as well. Or like I mentioned, sometimes we also have containers for certain optimization frameworks that we're using, like Intel's OpenVino or others that optimize our models for certain targets. Those are sort of a mix. And then after that, in the products we're building, we have a custom way of bundling our models together in what we call a model bundle for a specific deployment. And so that model bundle needs to be built and encrypted and then that's actually pushed back out to our Object store in S3 or DigitalOcean spaces, depending on the application, and that model bundle is saved there then for deployments of that system that would then pull down that model bundle from the object store and run with that.

So earlier when we were talking about motivations and you talked about the break even there, you know, you kind of figured out what made sense. But obviously when you made the choice and you're not going with cloud because the break even in that case, what are you thinking in terms of your upgrade? You know, because we talked about, you know, being able to fill in with more GPUs.

Yeah.

And enhance the capability of the box. But what about over time as GPUs start to age? What's your thinking there? And you know, that makes that a. A good choice from your standpoint.

Yeah, I mean, right now I have no choice there because I can't get my hands on any more GPUs because they're all. It's a good answer. They've all ceased to exist on the face of any website on the Internet. So I, I can't get my hands on any. But I think right now we're in a pretty comfortable spot in terms of the workloads that we're running. However, we're ramping up some of the work on these prototypes that we've built. And I'm guessing if those kind of catch on and are productized a little bit more, then we'll have a need for an interesting new workflow which is a bit more automated. So right now most of what we've been doing on this machine at least, is prototyping work and preparing very custom deployments for very specific applications. In the end, we're going to build a bit more of an automated workflow where, for example, one of the things we're doing is spoken language identification. So audio comes in and then we detect what language is being spoken in that audio clip and so if, for example, in a certain application we want these four languages instead of the six languages that we originally had in our model, we want to be able, in a very quick way to spin up a pipeline that would redo the training with sort of different combinations of data. And so that's something that I think would be interesting. I know we've had a couple of conversations recently. There's a lot of ways to do that. One of those, we've talked to the Pachyderm crew before on the podcast. They have a great pipelining system.

Definitely.

We've also talked to. I remember Allegro AI has a sort of training and automation system that you can run on an on prem instance. And so that may be another option that I can install that Allegro system on the workstation and run it that way. There's a lot of different options there that we could possibly use. And also upgrading the sort of monitoring there, right now, that process is fairly manual as well. So maybe utilizing either, like I mentioned, maybe something more than Tensor Board. So we just had a conversation with Weights and Biases CEO, sure did, Lucas Bewald, and he talked about the great stuff that they're doing. And, you know, all of that's, I think, not too hard to implement in that workflow. So those are some of the things on my mind.

So, you know, as we're closing out here, you know, at this point, would you do the whole thing over again? Because it sounds like you've learned a lot. There's definitely some cost advantages to what you've done. But I also know that as we recorded various episodes over the weeks that you were engaged in this, that, you know, there was a lot of work involved. There was as you were kind of staticing along the way. Was it worth it?

I think it depends on, I mean, of course, your need, but also your personality. I really enjoyed this. It was something that I found a lot of enjoyment in, and I think that was worth it in and of itself. But if I was to look back, I would definitely do it again. And I think, you know, the things like I mentioned that I would kind of tell myself in retrospect is, you know, don't skimp on the CPU or the RAM and over project your data storage. So go ahead and get yourself, whatever it is, 4 terabytes or, you know, whatever it is, something that you're not going to fill up quickly as soon as you start working with speech or video data or something like that, one terabyte, it seems like a lot, but it's not. It Turns out, you know, it's easier than you think to fill up that space. So get yourself, even if it's some cheaper hard drives that can be just sort of cold storage, do that. But yeah, it was a fun experience and I would definitely recommend doing it. You know, it's, it's fun to have that sitting, you know, by me and my now my home office. And now we don't have to heat our house quite as much. It's just winter, so it's good timing.

Side benefits. You get there. Okay. Well, that was pretty cool. It's a long overdue this conversation. I'm looking forward to seeing what listeners inform us because both of us are. This is not our specialty. Be gentle, be gentle on Daniel. It'll be interesting though, to see what we get in the Slack channel and other social media, the LinkedIn group group and stuff and see if anybody can school us a little bit. Maybe this is one of their, their things and they have that knowledge. So looking forward to, to hearing people share that. Well, Daniel, thank you very much for taking us through this. I'm glad that you did and thanks for teaching.

Yep. Talk to you later.

Talk to you next week. Yeah.

Come hang out with Daniel, Chris and hundreds of other AI practitioners centers in our community. Slack. It's a cool place to be. Not a lot of noise, some great signal and best of all, it's totally free. Check it out@changelog.com community and don't forget to follow the show on Twitter for AI news and links, highlights from past episodes and more. We are PracticalAIFM. We'd love to have you following along. Thanks to Daniel and Chris for hosting Practical AI week in and week out to the mysterious Breakmaster Cylinder for the excellent beat you here on all changelog podcasts. To our sponsors who have our back fastly, Linode and launchdarkly, and to you for listening. We appreciate your time and attention. That's all for now. We'll talk to you again next week.