Going full bore with Graphcore!

This is one of the reasons that we're kind of quite keen on having a very kind of flexible programmable software stack because there are all these choices to make. It's a really great example you've made there of GNNs because it shows that what we might be doing in two, three years time in this field is going to be quite different to what we were doing two, three years ago. And if you think like five years before that, then it wasn't your networks, right? So it wasn't deep learning. So I think key for us as much as having the right processor and having these graph techniques, if we can call them as a family, the important thing is to have the capacity to do innovation.

Big thanks to our partners Linode, Fastly and launchdarkly. We love Lino. They keep it fast and simple. Check them out@lino.com changelog our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by LaunchDarkly. Get a demo at launchdarkly.com this episode is brought to you by our friends at O'Reilly. Many of you know O'Reilly for their animal tech books and their conferences, but you may not know they have an online learning platform as well. The platform has all their books, all their videos and all their conference talks. Plus you can learn by doing with live online training courses and virtual conferences, certification practice exams and interactive sandboxes and scenarios to practice coding alongside what you're learning. They cover a ton of technology topics, machine learning, AI programming languages, DevOps, data science, cloud containers, security, and even soft skills like business management and presentation skills. You name it, it is all in there. If you need to keep your team or yourself up to speed on their tech skills, Then check out O'Reilly's online learning platform. Learn more and keep your team skills sharp@orally.com changelog again o'reilly.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changelog.com community and follow us on Twitter. We're Racticalai FM.

Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a Principal Emerging Technology Strategist at Lockheed Martin. How you doing Chris?

I'm doing good. It's a beautiful spring Day. And we're about to talk AI with interesting people that you can't ask for much more than that, my friend.

Spring has sprung.

Spring has sprung.

Since you're in the south, someone, one of my coworkers this morning was talking to me about. Every 17 years, the cicadas, like the cicada brood.

It goes crazy.

Entomology is. Wait, is that what it is? I forget the name. The one that starts with an E. That scientist. I'm not one of those people. The one that studies bugs. But are you aware of this?

I had heard that actually before, and it affects me in a fairly direct way. So longtime listeners and certainly Daniel knows that I do a lot of wildlife stuff. And so on the side, separate from this whole AI thing, I do a lot of wildlife stuff. And one of the things I do is I. I am a snake wrangler, among other things, as part of that wildlife thing. So I am commonly picking up copperheads with the appropriate safety equipment, by the way, just so no one gets the wrong thing. And copperheads love cicadas.

That's a poisonous snake. For those that aren't in the U.S. chris is very adventurous.

If you're not in the southeastern United States, a copperhead is a venomous snake. And so I have a lot of experience with those. And so totally off topic, of course, but the cicadas. Yes. Favorite meal of a copperhead.

Oh, okay. It's their prime time.

That's right. Who knew we were going here at the beginning of the show?

Yeah.

Yeah.

Well, I mean, I'm sure that there's all sorts of interesting AI to, like, predict the cicada brood numbers this year, but that's not the topic of this year.

That's not what we're talking about today. I'm so sorry, folks.

Yeah, actually, I think that I'm pretty interested by the topic and guess this week because it seems like we talk a lot on the show or GPUs or accelerators or specialty cards for AI are mentioned, but a lot of times they're mentioned just in the context of the accelerators, not the software component that goes along with them. Of course, I found, and I don't know about you, but I found that a lot of times when I, like, oh, I'm really have access to this really cool card or something, and I want to use it, but the problem is not having access to it. It's the software that I write for. It has all of these issues. So I'm really excited to talk about that software hardware interface today.

Real life AI right there.

Yeah, exactly. Real life practical, very practical. With Dave Lacy, who is chief software architect at Graphcore.

Welcome, Dave. Hi. Pleased to be here. It's great, Great to see both of you.

Yeah. And before we jump into things, do you want to give us just a sketch of your background and how you got into doing what you're doing now?

Sure, yeah. So I'm a computer scientist and particularly specialized in compilers and that kind of. That area of computer science my life. So I did research in that in academia. And then over the last so many years, I don't want to say really, but after the last quite a few years, I've been working in various companies, working on the software stack to target new types of hardware in different areas. Most recently, we'll talk about today in Pathcore and AI, but previously in HPC and in embedded processing and network processing, all kinds of things. So very much software guy trying to make sure that as we push the field forward and get new hardware to enable new things, we have software that can support that and let users actually program these new wonderful things that get created.

Cool. Yeah. And you mentioned a few different things. Of course, there's AI specific hardware, there's high performance compute computing or hpc. I think a lot of times when people think about like AI hardware, maybe they're thinking about GPUs, but I mean the story is a lot more diverse than that. So maybe you could give us a little bit of a sketch of this sort of landscape of AI specific or hardware that targets AI workloads, I guess is a good way to put it. What are those categories of things out there?

Sure, sure. So I think the ones everyone knows about are. Are CPUs, I mean, clearly. And they definitely have their place in this landscape for specific tasks. And then obviously more recently GPUs, because of the nature of the kind of highly parallel compute and what they target, they've had quite a bit of success. I think you see quite a few companies now and I think ours is graphical where I'm from and obviously unbiased is. I think one of the leading. Is probably the leading kind of new type of chip in there where what we're trying to do is produce a chip that's specifically for machine learning and AI. So one thing about other chips is you'll find that they've been designed for something else, CPUs. We know they do all kinds of things, GPUs, originally graphics. And our chip is something where we've gone, well, can we design a Processor that's just for this space. And this space is really important. You've got so much application that it's worth designing a processor specifically for it. I think that's the new kind of breed of process you'll get. And I think the IP is the one obviously that really kind of fits that slot that allows us to do more in this space than other processes can because it's. Because we thought about from the beginning the kind of attributes you need for an AI chip. So kind of what kind of data are we dealing with? What kind of data patterns does that kind of involve? What ramifications does that have for us? For example, on the memory hierarchy, Our memory hierarchy is very different to other processes and so on, just so we can kind of process data in a different way or the compute units and so on. That's where we're finding, we're finding a third place now, where we have specialized hardware, ML, AI, intelligent compute in general, where the IPU sits for us in graphcore.

So as we've talked to different folks through different episodes and, and everything is trying to understand kind of where you fit in and you're kind of laying out the landscape kind of still at a high level. Could you talk about a little bit about the different categories before we dive fully into graph core? Just so that we kind of know in our heads where to position it, how that fits in. Because people come into this conversation with different levels of understanding and knowledge on that. What that is. Could you kind of differentiate the different pieces of the landscape then we, as we focus in on where you're at, we'll kind of have some context.

Yeah, yeah. I think, if I may, if I can take that landscape and talk about it from the types of processing the different processes do.

Sure, sure, that's fine.

Yeah, that'd be great.

Yeah.

So like kind of a CPU does, is very much deals with scalar processing. So scalar processing is dealing with kind of individual numbers, maybe kind of small groups of numbers together, like small vectors and things like that, and they'll do it. And that's very good for the general compute. Right. Because actually nearly anything could be split down into individual numbers. Right, sure. So that's kind of the CPU kind of space GPUs, what they deal with primarily is large vectors of numbers. I think that's very clear. Right. So big contiguous blocks of numbers that come in and you're going to deal with them in parallel. Because they're dealing with them in parallel, they can get better efficiency. So they can go Faster on the workloads to the same power. I'd say what you could call the ipu, say our graph or one or other one. So you could call them graph processes in the fact that they're dealing with data. That's where you have to deal with a lot in parallel. So they're highly parallel processes, but the connections between them mean they don't come in big contiguous blocks of vectors. There's actually kind of, you're going to have to take one number from over there, one number from over there, one number from over there and then kind of deal with them in parallel. And that's kind of in some ways what you, a way you could characterize a graph processor. One of the aspects to think of in categorization, another categorization, if I may, is also about the type of number you're dealing with. So again, both CPUs and GPUs have a wide range of kind of number formats, if you like that deal with integers kind of, and also fixed point numbers and floating point numbers with certain different precisions. But in AI I think we're finding a lot that what you need to deal with floating point numbers, so kind of fractional numbers to represent, you're kind of representing probability distributions most of the time. Right. So you don't need a lot of precision. What you need is a lot of range. Like, kind of like if you think of probability distribution, that's kind of what it is. So again it naturally you're dealing with numbers that are low precision, floating point with kind of ranges you have to deal with and kind of randomness involved in there as well. Injected it so you can think about the shape of the data and the type of the data and kind of which processor deals with what. That's kind of probably a reasonable way to describe that landscape across the chips. I expect there are other ways you could split it up as well, but I quite like those as a way of kind of splitting, splitting these things out. And it kind of leads you to think, oh, what kind of applications could these be good at? Kind of by thinking about them in that.

Yeah, I see your point on that. Absolutely.

And maybe you could connect. So you talked about like graph nature of the iq, the graph core processor and like taking a number from over there and how that works. Could you sort of map that onto the typical like AI types of tasks that we're wanting to perform and, and how that connects people that for example, are started using TensorFlow before TensorFlow 2 or maybe familiar with saving quote graphs or Something like that or loading them. Does that connect to what you're saying or what's the connection?

Yeah, yeah. So there are several and they're kind of some different levels there. So one of the graphs that gets anything's a graph. If you look at it hard, it's one of the things. But one of the graphs that people talk about in things like TensorFlow and so on is the compute graph, which is the way data flows around between big tensor operations. And that's certainly one of the kind of larger coarse grade things you can think about. But I guess I'm talking more about the graphs that connect the information together. So if you think about what are you doing operations and quite often that matrix multiplies or their convolutions. Now let's look at say convolutions for image processing. That's a really good example. Right. Because there, what you have there is you don't have quite the all to all connectivity of a matrix multiply operation. Like you have a kernel which is going to be applied the same thing to lots of different parts of the image in a kind of across the image axis the one piece of data is being kind of scanned over the day. So you have a kind of a particular fan out graph there and then you can get in the operations you can get into as you get into other neural networks. Say if you look at things like ResNext or EfficientNet in image based neural networks, you can see that actually you get kind of grouped convolutions or separable convolutions again where you're kind of splitting up the data and then recombining it in kind of finer and find a way. So it's those kind of things that require hardware that can cope with efficiently moving that data around and also a software stack that can target that efficiently as well. So we shouldn't get the software stack. That's kind of fine. I always like to remind people about you need the software to target as well.

It takes them both.

Yeah, yeah, yeah. And then if you get into even more kind of recent things that people are doing where you have imagine doing a lot of what we do at the moment when people use TensorFlow is linear algorithm matrix A times matrix B and that gives you some result, kind of take some data coming in and we do a big weighted multiply between. But really that can be quite inefficient. We've got this trend to go really big, really big reason, which is great actually. It kind of is pushing things forward with these really big Kind of you see in language models and so on, but, you know, it's not very efficient. And you'll see that the shape of that data means there's actually quite a lot of sparsity in there. There's a lot of that data could be zero. And that leaves you with even more kind of complex data patterns. And I think kind of that push for efficiency is another space where you'll get kind of naturally a graph structure, natur. The kind of graph structures anything where you have kind of almost in my mind, irregular connections between the data. So I think there are lots of different ways of looking at a graph within a kind of neural network or any kind of machine learning program. But I think the key ones for me is like, how are the data? How's the data connected together?

Yeah. And I know that at some point Chris and I were talking about an article about trends. I forget which of the major AI conferences it was, but they're talking about trends year over year and in terms of what people are focusing on. And I know that there was a big boost in people looking more and more at what's called graph neural networks. Does that connect directly to what you're saying, past the other things in some way? Because I know a lot of people are talking about these things, and my understanding is it's, you know, in some ways people say a graph neural network, they have just a way of encoding the graph into a tensor, and then you still have the tensor operations, but in other ways there's actually graph native sort of operations that you can do within the network itself. I'm not an expert in these things yet. Have those connected with what your company is doing at all?

Yeah, yeah, it's something that we're quite interested in. And actually some of the research groups we work with kind of are interested in that kind of network. It's yet another type of graph, but the graph is not the model itself, but the data that the model is consuming. What we found is that there's a lot of choice there. There's a lot of choice in how you represent the data structure. There are many ways to do that. And then the actual kind of data processing operation you'll do will be different depending on that. So do you represent it as a. As a list of edges or dense matrix or a bit vector or things like this? Actually, I think what people really want is a way to explore through those choices at the moment, actually, to kind of push things forward. This is one of the reasons that we're kind of quite keen on having a very kind of flexible programmable software stack because there are all these choices to make. I think kind of is a really great example you've made there of GNNs because it shows that what we might be doing in two, three years time in this field is going to be kind of quite different to what we were doing two, three years ago. And if you think like five years before that, then it wasn't your networks, right? So it wasn't deep learning as such. So I think kind of key for us as much as kind of having the right processor and having kind of these graph techniques, if we can call them as a family, the important thing is to have the capacity to do innovation. And part of that for us I think actually does come down to the software quite a bit actually. It comes down to making sure that people can modify it in a good way, can extend it, as well as having the kind of easy out of the box thing where they just, I just want to run a TensorFlow program or something like that. I think that the GNNS is a very good example of kind of where you want that kind of look.

This episode is brought to you by our friends at rudderstack and we're calling all data engineers to check out rudderstack cloud and start building smart customer data pipelines. Rudderstack is warehouse first. No more silos. Rudderstack builds your customer data lake on your data warehouse, not theirs, enabling all functionality of a CD player with more security and retaining full ownership of your data. It's open source and API first. Rudderstack can be easily integrated into your existing development processes and because they're open source, you can see all their code so you don't have to worry about vendor lock in or black boxes. And best of all, they have transparent pricing. Stop paying your CDP a premium to store your data. Rudderstack is free up to 500000 events and pricing scales transparently. From there, learn more and get started@rudderstack.com again. Rudderstack.com that's R U D D E R S T A C k dot com.

So my head is also still spinning a little bit on something that you were talking about a little while ago and that is, you know, the software and the hardware, you need them both. Over the decades the world has been very, very CPU focused, you know, up until this most recent time period where we're really digging into AI and, and all of these new hardware architectures have come into being. How do you approach given the fact that we're having all this innovation in how these hardware architectures are coming out. How do you approach writing the software that goes for that hardware, that makes that hardware run? Especially when you think about the fact that we have a long history of being specific to different types of CPUs, but now the sky's the limit on what's happening. How does that change the act and the thought process and the planning for building software?

I think it does a lot, actually. I think CPUs are amazing. I think they've been so successful in fixing an architecture, fixing all aspects of the architecture, the way the instruction set runs, the way the memory hierarchy works, and then applying it to lots of different fields and having that success. I think it's a remarkable thing. I don't know if it works for fields though. And I think what we find in AI is one thing that we've been very keen on right from the start, and I definitely see other people in the field talk about it, is this idea of co design. So co design is this general idea where you want to design several aspects of assist together. And so clearly one of the big aspects of co design is design your software stack and your hardware architecture at the same time. And that's something we're very keen on at graphcore. Actually we kind of, right from the beginning, we're very keen on making sure we worked on how the software was going to target the chip, make sure we kind of modeled everything right up to the kind of full neural network applications as we designed the chip and pushing that all through, I think later on in the company. What happens then is you need to extend that software stack and kind of really, kind of invest in kind of building that core to a really robust solution. So now we've got kind of more software people than hardware people. It's a really big part of what we do. We're as much a software company as a hardware company. But even right at the beginning it was very much let's design these together. I don't think it stops there actually. I think actually you could also say you co design and we'll see more and more of this. The machine learning algorithm architecture itself, along with the software and the hardware and the system. So the kind of way you, you know, we're not just particularly for training, right? For large scale, large data machine learning training, which is the world we're often in. At graph core, you're not talking about one processor, you're talking about hundreds of thousands of processes working together and they all need to be connected via networking Cables and put into racks and the power management and all this kind of stuff has to be around it. And really you need to co design all of that together and that's good. I heard someone kind of use the term algo texture before to describe this and I think, yeah, why not? That's a fair enough term.

That's fascinating what you're saying. I'm not trying to cut you off, I actually want you to expand on that. The idea of doing software hardware, the actual ML architecture and the system, that seems daunting because all of those individual topics are. People spend their whole careers doing that. How do you blend that in a cohesive way?

Seems like your team makeup is very important for that.

I would imagine so.

Absolutely. I think for us, yes, team makeup's important in the company. We've really kind of stressed that. I think partnerships are very important for us. So we've always been quite keen on working with commercial kind of Internet, consumer Internet companies that want to do large scale deployments behind search engines or whatever there is. So that, you know, the kind of thing which we also want to work with research groups, both kind of university and company research groups to kind of see where that's going. And we're not alone in that. A lot of companies will, you know, that's why I think there's been part of. Why that there's this keenness to bring kind of ML researchers into within companies to work on that. So I think that makeup and the communication between those is really important within company to do that. And partly it's just an awareness to want to do it. It's as having that communication like freely flowing and actually to have the right kind of culture within a company to Foster your software AIs and Gate. Well actually no, we really feel that actually to get this algorithm working you could work it this way. Way. There is a flip side, I have to say, in that you've got to be careful not to design out the future if you like, because you could interpret this idea as being we'll, we'll harden a piece of software into a chip. So we'll take like, oh, I don't know, Resnet 50 or something to give a good example from and say we'll make a chip that will do that really well. I don't think that's really co design because it, it doesn't. Because you have to also design for generality. As I say, design for not just the algorithms of the day, but the algorithms of three years time or four years time as well. Because a chip has to last that longer and people want that flexibility.

Is that a little bit of an artifact left over from an earlier time when you were still thinking about CPUs and there would be. Things would start in software, you know, you'd solve a problem in software and it would stick and be maintained over the years and finally it would get incorporated into chips. Is that a legacy mindset that maybe has been brought forward but doesn't work when you're advancing on the ML architectures as rapidly as we are? I mean, if you, if you bind it into the chip and you're stuck with that, whereas we're seeing, you know, rapid advancement over the last few years in terms of where things are going.

Exactly that. I think that's exactly right. I mean, you just need to look at, as everyone who subscribes to like Archive for ML will see, the amount of the fire hose of innovation here is just huge and you have to plan for that. I think good flexible software is the key to that, but. And then you have to design your hardware to kind of match that as well, so. So. Exactly. I think that's hit the nail on the head. We're in a real fast moving space here. It's not stopped. Right. There's still a long way to go before we kind of settle down on exactly what the algorithmics are.

As Chris knows, I love to get into the practicalities and I guess what I'm thinking of is you have your graph core processor, the ipu, over here on this side and over here you have already established community frameworks like TensorFlow, PyTorch over here on this. And obviously people are, they want to use those, they want to use Keras or whatever. And I'm sort of looking at a diagram which we'll link in our show. Notes about your popular graph framework software, which is very well written, but you've sort of got the frameworks on this side and you've got the processor on this side. Could you explain just for people, like what it takes to connect something written in TensorFlow or PyTorch, like sort of people are used to, to this new way of processing on the graph processor. What's in between there?

Sure. Let's take TensorFlow as an example and I'll see if I can walk through that diagram from left to right. So kind of you have your TensorFlow program written in Python usually and what that describes is, it will describe, usually it describes some model that you want to optimize and then you apply an optimizer to it and that gives that model really is we're back to this kind of compute graph, right? So it's a series of linear algebra operations kind of connected together. And out of the end pops an answer which you can evaluate on how close you're getting your optimizer to learning a good solution to that. That graph gets explicitly represented in TensorFlow. So a data structure gets created, which is that graph at that stage. It's called the core graph in TensorFlow. So it's pretty much what you've written, more or less. And then what happens is, well, the first thing that happens really is that that gets differentiated. So we kind of create the backwards pass to say how do we calculate gradients on that? And it gets wrapped in a kind of, that whole data structure kind of then represents a big loop outside that says feed data in and kind of update my weights. As I, as I, as I calculate these gradients, we take that whole graph and then it gets passed first through the TensorFlow compiler flow. And that's not kind of a graphical thing, that's the TensorFlow kind of upstream developers, they'll take Graph, they'll canonicalize it into kind of smaller operations, they'll do some optimizations on it, and they'll convert it into what's called HLA Graph, the XLA graph. So that this goes through their compiler infrastructure, which is called the TensorFlow XLA. So the end of that you have a kind of slightly lower level split up graph where we're still talking about kind of quite big linear algebra operations like matrix multipliers and things like that, but it's been kind of reduced down and tidied up and made a bit more kind of closer to the hardware. So at that point, Our graph Core TensorFlow backend takes over and the first thing it will do is it will do a few more kind of optimizations on that data structure at that level. For example, in the chip we have a hardware unit for doing exponentials and sigmoids and the kind of things that come up in certain nonlinearity. So it will recognize those patterns in that graph and say, well, we replace them with one special operator that will kind of go down to the hardware and so on. So we'll do that kind of thing and then it will basically convert those operations into an even lower level form of graph which is much more fine grained than that. So we have something called POP Libs, which are libraries that implement things like matrix multiplies or nonlinearity operations or things like that in Poplar. So let me Talk about Poplar briefly, because I realized I just introduced that without saying what it is.

Absolutely.

So Poplar is popular's R graph programming framework. So that is a way of representing graphs that run natively on our device that do these kinds of of operations. And in Poplar we have graphs that kind of break it down to the individual processing units. So on each of our chips we have kind of about 1400 cores processes on the car, each of which has kind of hardware threading in there. So you've got about 7,000 parallel compute units. And the Poplar graph kind of represents like the graph at that kind of level. And Poplars is what kind of then says, well I've got this major spotlight to do. How do I split that over those parallel units in an efficient way? So that's where it does kind of partitioning and access splitting and stuff like that. Then we have the popular Graph compiler which then will take that fine low level graph and create kind of actual code for the device which then goes into the graph engine which then kind of runs it. There are quite a few levels, you notice quite a few compilers involved and we have like, kind of. So we counted them and there's like five or six different compilers that have to interact to get that kind of efficient implementation down on that device. There are some other things that go on like sometimes you might want to multi chip models. So at a higher level you'll do kind of kind of model pipelining and things like that to get a kind of efficient model spread over multiple chips and things like that. But fundamentally that's the flow.

That was great.

And by the way, I don't think I can recall anyone ever taking us through even a genericized, you know, not specific to your system system but that. So I appreciate that very much.

That was super fascinating. And I think as you mentioned Dave, it's like there's all of these layers and of course you're connecting to different frameworks that are out there. It's intriguing to me from a software development standpoint with these frameworks updating and new architectures and operations being added and all of that. I guess one, how do you keep up that pace and test that whole pipeline of things? Is it a matter of having like reference implementations of all of these different models and essentially running like tests against the compilation of those upon new versions of your framework or how does that work?

Yeah, exactly that, you know, there's a lot of investment. Well, one is the development that kind of not a lot of software developers and then Like a lot of investment in regression system and test infrastructure. That all has to be done because you have to have a kind of really robust kind of comprehensive software product that there's no way around that to be usable. And I think to kind of blow away in Trump a bit here. I mean I think graphcore amongst the kind of raft of AI startups is really well advanced in that space. All the elements you have robustness, like kind of documentation. You go to our site, you can see the tutorials and the documentation, how you do that and so on. So I think we have to kind of try and keep on top of that, keep working internally graphcore to keep those expressions going. I think the other thing I'd say that kind of really helps with this is being very open so those so making sure that people can we document and make sure that people can access. Well this is something that you do get on other platforms but I think we try and be real leading in this and kind of being able to know what popular the low level graph compiler kind of explaining that making sure that people can add custom kind of operators in that those libraries I talked about, they're open source on GitHub, that kind of thing. So trying to make TensorFlow, PyTorch backends are open source and everything as well. So by having a more and more open infrastructure, it makes it easier for the community at large to help you adapt to new things as well. And as we find as we're kind of getting more popular, more people are using us that we kind of get more kind of community involvement like that as well. And I think that's an important part of it as well. Foreign.

We deserve a better Internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome. Keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in add and tracker blocking by default. Quick access to the Tor network for true private browsing and an opt in reward system so you can get paid to view privacy response respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give Tippian a try on changelog.com.

So Dave, I'm curious, as an AI practitioner, one of the things I would love to know from you is as you've spent all of this time making AI programs, sort of be sympathetic to a certain architecture and for certain tasks, do you have any sort of good advice or help For AI practitioners out there in terms of knowing how to tailor our AI programs or models more generally to be efficient for a certain data set or task. Are there any sort of. Of good advice you have or common challenges that, or common pitfalls that you see people falling into that could be mitigated with some best practices?

Yeah, it's a really good question. It slightly depends on whether you're talking about kind of task performance, like how good your model is at a particular task, or whether you're talking about kind of compute efficiency, kind of, you know, how fast does it run in terms of task performance. I think people need to be aware that those two are not kind of islands that are separate the architecture you write, even whether you're aware of it or not, even if you just like take what other people do or whatever has been kind of affected by the underlying compute platform and what's efficient. Because if you were kind of trying to study whether a particular kind of model architecture, for example, got good task performance but it was really, really slow, then you probably wouldn't like pursue that, even though it might be quite a good way of if you're just looking at pure task performance independent of how fast it goes. There've been references to calling this like the hardware lottery, right? Kind of like what hardware you've got kind of affects kind of how lucky you are being able to explore certain things. I think it's a good question on practice there though. So I think one thing is that's useful is just a kind of base understanding of what's going on underneath. And probably the important things there are are if you're interested in the kind of efficiency, the compute efficiency, knowing kind of how things like, for example, batch size or kind of certain sizes, axes of your matrices actually kind of affect the hardware underneath and could be faster or slower. I think the other thing is being very aware of the floating point behavior of various platforms, because that can vary a lot between platform and a good platform provider should document that and you should be aware of kind of the tools they have to show when things are overflowing or kind of underflowing and so on to know when you might be losing time. Task performance, not because of the model structure, but because actually because of the data format they're getting there. And actually we're seeing new techniques coming in to help that actually kind of you'll see things like automatic loss scaling is a really good example. The software stack and the vendors are trying to help there, you know, kind of trying to make things more adaptive so you don't have to think about those things. But I think it kind of is worth kind of just having a kind of good, kind of surface level understanding of kind of what's going on under the board.

Yeah. Chris, you're probably familiar with Bill Kennedy in the GO world, and maybe you are as well, Dave, but we spent some time working together and he always in our conversations was talking about this idea of mechanical sympathy as you're writing code, which I think gets to a lot of what you're saying, Dave, in terms of. Yeah, it's. Maybe not all software engineers don't have to also be hardware engineers, but there is an element of like developing a mechanical sympathy for what you're writing for that helps you write like really robust and good software that I think is really valuable, like understanding, you know, and go. It's, you know, understanding like, oh, if I initialize a variable this way, it actually, you know, does something different than if I initialize a variable this way in terms of the memory that's allocated and copies that are made and all of those things.

Things.

So having that sort of knowledge, I think is really cool and it's interesting to hear you talk about it in the context of AI. I think it's something that I definitely want to develop a little bit better intuition for in my own work.

I think that segues into what I was going to ask too. So we're both thinking along the same lines in terms of questions. And that is you've taken us into this and explained kind of that compiler series and we've talked about kind of the interface being PyTorch or TensorFlow, and I know I am coming at it as a practitioner, and as is Daniel. You have presumably different types of users that need different amounts of that kind of mechanical sympathy, as Daniel described it from Bill Kennedy, or knowledge of what's under the hood at different levels. How far do I need to go? Should I learn Poplar? Am I going that far? Am I going beyond that? What other users. If the answer was no for me, what other users are learning that, how do you break that out? How do you know who should be addressing what? Because that's a question we get all the time from new people in this field. Is there seems like so much stuff I don't actually know what to address first and second and third. So could you break that down a little bit from that end user perspective?

Yeah, it'd be unrealistic to say that, like, everyone needs to become a full stack developer for the full stack being this kind of, there's this incredibly complex.

All the way through, down through the hardware.

Yeah, yeah, yeah, yeah. I think that's a really rare thing. I mean you do get people like that, I've met people like that, they're great and you want to hire them basically when you see them, but you're not going to get that. So I think what we find is it's harder for a sole practitioner. I think kind of what we find in the companies we work for is they do that by teams. Right? So you have kind of very much a kind of implementation focused team and a kind of algorithm ML focused team. And the challenge there is to make them work kind of hand in hand. A bit like the co design thing I was talking about earlier, to work together and kind of just being honest about the fact that you're not going to get this all in one person. I wouldn't kind of suggest that people try and really do that unless they're super, super interested in all the parts. You kind of almost need to pick where you want to go and so on. And I think despite what I said about kind of like these lower parts of the stack, the majority of what I call ML scientists or kind of data scientists and stuff like that don't need to know about those low level details, they need to just have some sympathy with it. And I think that's it. So kind of just a bit, maybe dabble a bit to kind of understand a little bit about machines. Don't try and learn it really well. I mean, and definitely our user base is like that, right? You might say that maybe about 90% of the people are programming at the PyTorch TensorFlow level. Maybe a bit of understanding the machine learning and you've got small kind of group, but those, those grass groups, very important for us because what they can do is they can add new functionality, new capability to, to a stack or an application or a framework or something like that. Actually maybe this is a case for people to specialize a bit and kind of, if you want to be that person that kind of can understand the details, then do that. But kind of find good people to work with that understand the higher level and vice versa. If you kind of really want to understand the machine learning part of it, have some sympathy. But then maybe be find some people to work either in the community or kind of set up your kind of professional kind of team structure to make sure you've got people with those other skills as well. We kind of have a mix at GraphCo. I'd say we have some people who are that kind of full stack, try and kind of go up and down it. Obviously we're in a bit different position because we're kind of an implementation side of things. Right. And then we have kind of people who specialize.

So Dave, I'm interested in, in of course this space is developing rapidly and I know, you know, graph core and IPUs have been gaining traction. There's good use cases out there and we'll definitely link to some of the materials on your website and all of that. I'm kind of wondering, as you look to the future, this next year or two years with Graphcore, where do you see things going and what are you excited about seeing in terms of the development of this technology over the next year? Two years as we look to the future?

Gosh, it's very, very hard to pick. There's an awful lot going on. I think kind of, I think the algorithm space will continue to move very quickly. So I'm very excited to see the fact that we might be doing very different things even two years from now to what we're doing. So people won't be talking about, wow, look at these kind of ginormous transformer NLP models. Wow, look at this other thing done. So I think that's, that's very exciting. I think what's happening in from a kind of more kind of systems point of view, I think kind of what's happening in the data center will be very interesting. The way they're put together, the way to get efficiency, the kind of the software and the processors and the network will all have to work together is a fascinating space that will evolve I think quite quickly in the next couple of years from a software point of view. I'm interested in like how the current frameworks and stacks evolve as we try and go towards getting more efficiency. And I guess I'm talking about kind of parameter efficiency here. I suppose so that's going to kind of give lead to new algorithms. And I think this is just a personal gut feel, I don't know, crystal ball here, but it might be that we kind of are these kind of very linear algebra based frameworks that we have now, the ones that be very popular, the tensiles and titles, so on, are they the right ones? I don't think that's a kind of second thing. And that's very exciting, right, that there's still that kind of uncertainty about that kind of thing. There's all kinds of stuff out there about how it's going to grow and scale and that's not even talking about the actual applications that are going to come out of this space as well. So. So all the way through from top to bottom. There's loads of exciting stuff coming up. I'm absolutely sure of that.

Yeah. Awesome. Well, Dave, we really appreciate you joining us. This is super fascinating. I'm really excited by what Graphcore is doing. We'll make sure and link a bunch of links in our show notes for listeners. Definitely check out what Graphcore is doing. Read their white paper and all the information about the popular software framework. It's really cool. And of course the hardware. And yeah, I was just really enthused by the conversation and have a lot going on in my mind that I want to think about more. So appreciate that. Thank you for joining us.

Dave. Thanks very much. It's been a pleasure. It's been really good fun.

Thank you for listening to Practical AI. We appreciate your time and your attention. Follow the show on Apple Podcasts, Spotify or your favorite podcast app. Your neural networks will thank you. We are also on the web at PracticalAI FM. There you'll find recommended episodes, listener favorites and a free sign up to join the community. Practical AI is hosted by Chris Benson and Daniel Whitenack. It's produced by Jared Santo with music by Breakmaster Cylinder. Thanks again to our sponsors Fastly Linode and launchdarkly. That's our show. We hope you enjoyed it and we'll talk to you again next week.

Sam.