"Foundation" models

But then there's this idea of misuse. You can create a really flexible, extremely powerful model which lets you do a whole variety of downstream things much easier. So in other words, you can do named entity recognition, sentiment analysis, create fake news or target marginalized groups. On the one side, we're creating these large language models that we would like to be more accessible to people because we don't want the concentration of power around these foundation models being with just a few entities. So I don't know. I don't know what the right balance is there.

Hello friends. Jared here to tell you about changelog, our membership program for those of you who want to directly support our work. Your plus plus membership gets you closer to the metal with extended episodes, makes the app ads disappear, and takes our audio to the next level with higher bitrate MP3s. You can join today at changelog.com. Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and the data science happen. Join us at PracticalAI FM community and follow the show on Twitter. We're at PracticalAI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out@fastly.com.

Well, welcome to another fully connected episode of the Practical AI Podcast. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and then we'll dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

Doing great, Daniel. How you doing today?

Doing pretty good. How's the strategy world after that? Our last fully connected when we were talking about innovation teams and stuff, a lot of those ideas have been swirling around in my mind and I just figure that's like what the life of a strategist, like you're, you're thinking about those things all the time. So I've been thinking about you over the past week as you, you've been strategizing.

We pretend like we know what's going to happen next. It's a great gig if you can get it.

Yeah, well, that's the thing, isn't the saying, like if you try to predict the future, that's like the one thing you can be sure you're going to get wrong.

Yeah, yeah, yeah. I'm pretty ok. Okay. Being wrong too.

Yeah. You sort of embrace that.

We did the strategy one and then I know we've talked about data fabric recently and that's also been another interesting thing. I'm still working on that pretty intensely and some stuff that I'm doing and having a good time. Because you know what, this whole AI thing keeps rolling forward.

Yeah, yeah, definitely. And there's something I wanted to bring up in this episode and sort of talk through with you. It's not like an extremely recent thing. It was back in I think August that some of this came out, but we haven't talked about it yet. And I realized we haven't talked about it yet and at the time there was a lot of language floating around that I didn't really know like what people were meaning when they said this stuff. You know, I've kind of circled back to some of that and kind of figured out what, what we're talking about at least a little bit. And I'd love to talk through that with you. And that's around the topic of foundation models. So this is something that was work done at Stanford University actually in a center for research on foundation models, CRFM at Stanford University. And they came out with a fairly lengthy and very interesting report on this topic which they're calling foundation models. And I remember at the time people were sort of like responding to, I think mostly the, the name foundation models. Like oh well, what do you like? Maybe there's some hubris around like saying, hey, we're, we're working on foundation models which are the foundation of, of AI. I think clearly they've, they've also released some articles around this or blog posts and I think that clearly wasn't, you know, at least their intention. But it's a super interesting article. So the, the article that they wrote is on the opportunities and risks of foundation models. First off, I guess, Chris, have you come across this term?

I have come across it and I too have found it confusing in the same way. And so this is an interesting article that you found here. You want to give us a definition?

Well, I figure that being in the good spirit of a programmer, I would just rather copy and paste than create my. So we'll go for the definition from Stanford.

Okay.

And that's actually I think a good thing to start with just sort of parsing through. So they say we define foundation models as models trained on broad data in parentheses, generally using self, self supervision at scale that can be adapted to A wide range of, of downstream tasks. So let's sort of take a step back and parse through that. I think most of our listeners are familiar with some type of models that jargon by this, probably we're meaning mostly like neural network type of models, modern model architecture, but basically some sort of parameterized function in code, right. That is trained based on some data, using training algorithm of some type, an iterative training process. And so that's a model. And they're saying foundation models are trained on broad data, generally using self supervision at scale. So some of those terms, particularly self supervision, might not be the most familiar to people. Have you run across this in your own work, Chris?

Yes, in terms of different levels of supervised learning and the fact that we are now more and more training models that are either unsupervised or at some level between that self supervised. The thing that I am curious about here is how should I think about this in terms of transfer learning? Is this something that I would use as a base for transfer learning to build upon?

Yeah, I think that when they're doing their report, two things are the two kind of pillars on which foundation models that they're calling them are built. If I'm remembering right, they talk about transfer learning and scale. So for those that are maybe familiar to transfer learning, that's where you have some sort of pre training that goes on and that's on a task that's maybe related to the task that you want to eventually do. But then you sort of transfer learn or fine tune that model for the actual downstream task which you're interested in, which is different from the one the pre training task. And so I think this gets, it does get to that, Chris, because they're talking about a variety or a wide range of downstream tasks. So this is the scenario where you had some pre training that happened on one of these foundation models, a large model, and then your transfer learning, or maybe in a zero shot or few shot way, that model is able to perform a variety of downstream tasks. So fit into that zone of like transfer learning, zero shot, few shot type of scenarios. So that's one side of it, that pillar of transfer learning. The other side is scale. Now the interesting, I think connection between scale and self supervision is that in the sort of, I guess more traditional supervised machine learning world that we kind of started with, you always had a data set that was annotated or labeled for the task that you're interested in. Right. And then you trained your model on that data set. The problem with that, right. Is that that's hard to scale, right? It's hard to scale sort of across a wide variety of tasks because you have to have a data set for each of those tasks. And it's very expensive and time consuming to create those labeled data sets. But also, if you think of the size of even one of those data sets, the more data you need or the more complex task it is, you might need to label a whole lot of data. So for example, the question answering task in natural language processing, I've learned because I'm working with some of that data now, it's really time consuming to build those data sets or like conversational AI data sets, sometimes a lot of very hard and difficult work goes into making those. So if you want to scale your data sets for training in that sense, self supervision is actually really kind of another important feature here of these models. Because let's say you had just a bunch of text, just like blah bunch of text, right? Well, you can actually in a self supervised way create your own training data to train this foundational model or this large language model by let's just take some sentences, remove words and say the task is tell me what word went here. And since you are the one that took the word out, right. You know what word should go there. So you're self supervising this pre training task, which is much more scalable in terms of preparing the data for training.

So I misused the term earlier. I was conflating unsupervised with self supervised. Pardon my confusion there.

Yeah, no, it's confusing terminology, right. There's this semi supervised, self supervised, unsupervised supervised. There's a lot of supervision going on of different types of. So yeah, it's worth kind of diving into those. But yeah, I think the idea is we're really talking about scale and you can't build large data sets, at least not in a cost efficient, time efficient way. If you're always thinking just in terms of supervised learning.

Yeah, fair enough. So how might you think about implementing this? If you're out there and you're interested in moving into this as a new skillset that you want to develop for your organization, what would a good first step be?

Yeah, well, I think that that's a really interesting question in this, in this work because actually one of the things that they talk about, because they're talking about opportunities and risks with foundation models, is that accessibility and homogeneity are things that come up with foundation models. Meaning that actually me, like my team working, I might not be able to create One of these foundation models, right? Because I don't have the resources. So just to give a kind of couple examples, we're talking about here, things maybe that certain things that people might be familiar with are things like BERT or GPT3 in the language space. But then there's other computer vision models and other sort of modalities, speech models, like thinking of wave 2 vec or other things. So if you imagine like tasks in computer vision, tasks in natural language processing, tasks in speech technology, and you look at what people are doing now, very often a bulk of the things that people are doing are sort of homogeneous in the sense that they're all built on one of these foundation models. Whether it's BERT or Wave two Vac, or one in the GPT variety, or, you know, all of these are sort of built on one of these foundation models. Like we fine tuned BERT to do this, or we, you know, transfer learned this foundation model to do that. And so this is one of the things that they bring up in the paper as a risk, is that there's sort of two things that come out from this, right? One is not all researchers and practitioners can create their own foundational model because I don't have like racks of GPUs maybe or the computational resources, the manpower to actually create a model on this scale. So that's one thing is like, hey, is it bad? Is that necessarily a bad thing? I think it is concerning that, like this sort of concentration of people that can actually create this state of the art work. But then the other side of that is like, if there's an unexpected behavior or bias in that foundation model, then that actually filters through down to a huge number of downstream tasks and applications because no one's retraining that foundation model, right? It's just there and everybody's using it, right? And so that's another risk. So I think the implications are really what should be on the mind of practitioners in terms of this trend.

Maybe you know what that reminds me of, just as an analogy, as you were saying, that is, if we look at the software world in parallel, there are some software bits out there that you find in open source that are in just about everything and they flow downstream. And if you get a bug thinking about the bias in the model or something, but if you get a bug in the software, something unintended, it can have fairly substantial consequences as it flows downstream. Is that the right way of thinking about this, is that since you're relying on this so much that if you do go awry, it's going to multiply itself many times.

Yeah, definitely. I think that that is exactly the. It's not an exact parallel because we're thinking of like mostly issues maybe stemming from data and bias and data maybe or something like that. But it is parallel in the sense that, like, hey, if everybody imports this one package and it breaks upstream, then there's a huge consequence downstream. Right. And the parallel here would be everybody's using, let's say, BERT to create all of these different NLP applications. Well, what happens if there's this bias in bert? And we've seen examples of failures of these large models. Right. And so, yeah, I think that is a good parallel. So, Chris, I think maybe one of the things that I remember when this came out is that people were kind of thrown off by the terminology. I think even in the blog post that I was reading, which I'll link in our show notes, they say the name foundation model has also drawn significant attention and given that they want to clarify it. So there's a whole clarification in this article. I'll link in the show notes.

They threw me off at the beginning of the show.

Yeah, I mean, naming is difficult, right?

It is.

So like, what is that? Like, naming. Naming variables is the most difficult part of software engineering.

In the software world, there's nothing harder than naming variables.

Well, yeah, so I do feel for them and I think that they were making a good attempt at this. And I've actually felt this tension maybe in our previous conversations because I come from the NLP world and we have this term large language model, which is essentially what they're calling a foundation model for our space. Right. But the thought process around that type of model and the way it's used downstream, the way it's trained, that is pervasive across other modalities. Right. In computer vision, in audio, even in areas where people are doing like biological structures and other things. This idea of having a large base model that's trained in a self supervised way at scale and, and used in a variety of downstream tasks, that idea is kind of pervasive. So I do like the fact that in the past I felt this tension even when I've taught classes like what do I call this thing we're doing, this new trend in how we're operating. And I think you could talk about it under the heading of transfer learning, under the heading of few shot, zero shot or self supervision. But I do appreciate the attempt to create a term that sort of encompasses all of these things that are at play because there's definitely this, this trend. So I don't know if you have any thoughts on that. I'm not opposed to the name actually. I, I kind of like having a name to refer to this as. But I don't know, I don't know if it's catching on. I don't hear like tons of people using it out there. So maybe it's not catching on, but I kind of wish that it or something like it would. Right.

I had heard the term but just like self supervised learning, I don't think I had really gotten the correct meaning right on. It's interesting is in the article here they kind of talk about this trend as you pointed out and where it's going and whether what the implications are and they talk to that a little bit lower in the article and they specifically kind of address the pace of technological process, the entrenchment of the models themselves because of some of the limitations that you pointed out earlier and the demand of kind of the, the human social side of it and technology. And it raises a good point. I mean they empower us to be able to use these things because it gives us a capability we might not otherwise have. But there's definitely some pitfalls there, you know, especially in terms of, of you know, bias in the data that you pointed out. Where do you think, I mean as someone who is using large models models on a regular basis, yourself and the work, do you think that this is probably where things will continue going for some time or do you see any alternative? I think it's kind of a necessary evil to have foundational models by that name or otherwise to build work upon shoulders of giants kind of idea. Any thoughts there or whether that's. Are the risks too great?

Yeah, I think that as you mentioned in my own work, like we don't train for the most part we're training a lot of downstream task models and few maybe what would be considered foundational models or in this sort of self supervised pre training way, although we have done it a bit. So I've benefited a lot in my own work from, from this trend and I think it does come with risks though. I think when I was going through this article and thinking through things through a lot of it centered around sort of concentration of power and other kind of trends in the AI world, especially because I work at a nonprofit that works with minority language communities. For the most part local languages, minority languages are left out of foundational models in the NLP world. Right. So I, I already view this as an issue and they Bring this up in terms of. They talk about, you know, supporting diverse research. And I think regardless of the modality that we're working with, it's important that we consider whether it's geographic diversity, language diversity, but also diversity of those creating the data sets and training the models and having representation. All of those things, I think are both are really critical. If this trend continues and we don't think about that side of things, then those sort of marginalized groups in terms of how they're able to use these foundational models and the implications of the applications that are produced, they're only going to become more marginalized. Because if foundational models are a key piece, or I keep calling them foundational, foundation models are a key piece of this sort of new tech stack that we're building for the digital sphere, and those groups are just left out, or there's bias in the models against those groups in various ways, then that's a real problem in terms of their own sort of flourishing in the digital sphere.

I think that's a fantastic insight that you just made there, and that is that from an economic standpoint, people who make these large models, these foundation models, are incented to solve the problems which are probably being done by kind of what is current, mainstream, kind of current, top of the power stack, if you will, in terms of companies and countries even. And so there is an incentive to unintentionally perpetuate inequality with these. And so it's a real. I think that would be a real dilemma. How might you tackle that going forward to where? If we're looking at foundation models as being somewhat core to most workflows in the AI space, because they're available and they accelerate where you're trying to go, but they are built on imperfect data that has bias that leaves out marginalized groups. Do you just iterate and add those in and redo that? There's a certain investment obviously to be made in that. Do you have any thoughts on least evil path forward to get the best foundation model with the least amount of unintended consequence over time?

Yeah, it's a difficult question. My sense is that that is why Stanford created the Center. Although I wasn't. I haven't. We would love to have someone from the center on the podcast, but I haven't talked in detail about the motivation to them, so I'm not putting words in their mouth. But I think there's multiple sides to this. I think one side of it is having, you know, think tanks or research groups that are really thinking about the problem and how foundation models, large language models, you know, computer vision models and other pre trained models are influencing society more generally. And the risks, the opportunities with that, because there's clear opportunities, but there's also, there's also risks. So having that research side of things, I think is a key piece. But also I think as practitioners, this needs to be part of our thought process when we're using these models. And I mention this fairly frequently on the podcast, but the idea of kind of probing the behavior of the models that you're releasing into production, I think is a key piece of this. So not just trusting that like hey, Google or OpenAI created this great model, but I'm going to fine tune it on this task. Okay. It does great on my test data set. So, you know, push it up to production. I think thinking of behavioral testing, where you actually probe how does my model respond to these changes in my data? What if I perturb my data in this way? What if I switch out this for that? What if I make this change which should be invariant in the output? Is it actually invariant in the output? And creating that sort of test, suite of behavioral tests for your model, I think is a big thing that practitioners can keep in mind in terms of how these foundational models might impact their downstream customers. The report actually does they talk about the social impact, but they have this mindset. There's a figure three in the paper. Their figures are really nice. I wish I could make figures that nice. They talk about these sort of almost like a value chain or something of usage of these foundation models, starting from data creation, through data curation, through training and application and deployment. And they think about the impacts of those models along that chain. So, like there are real people on either side of that chain. Right. What is the diversity that's represented in those groups? And what might be the impact of having biases in this sort of data producers or the consumer side, the makeup of the consumers. So I think that's a really good way of thinking about it. And having that more holistic picture in mind is helpful.

Yeah. You know, as you were going through that, it struck me that in addition to kind of accounting for the unintended consequences behaviorally as you perturb data and stuff, that the flip side of that coin becomes vulnerabilities from a security standpoint in terms of affecting the outcomes of foundation models, or any model for that matter, by doing that. And so it just has a greater multiplier associated with the foundation, since you're building so much on top of that, so as we look at this in the days ahead, it will be interesting to see as well what the security implications are of protecting the end users that are impacted by this. Meaning the model will change the behavior of lots of different executions on that where people are using it for various use cases that affect people. I would conclude by noting that in all of this that we're talking about, it's starting with human beings and it's ending with human beings. And we are a little bit at the mercy of the models as we're trying to make improvements in other ways in our lives to, to do things productive in the world. It's a curious set of problems that we face going forward.

Okay, Chris. So in continuing our way through this material that Stanford has put out, they actually do have, I think, your point made around, like, people on either side of this. They talk about emphasizing the role of people. So I think they're, they're very much with you in that. They also use this terminology in terms of how to mitigate some of this risk and vulnerability stuff. They talk about how foundation models can and increasingly should be grounded. Now, that was also a term that I wanted to make sure that I sort of dug into a little bit because maybe it could mean a variety of things. Is that terminology you've, you've come across?

No, I ran across that in the article a moment ago, just before you said it, and I was wondering. And then you went there, I was wondering, what do they mean by grounded? So what do they mean by grounded, Daniel?

So it is interesting that there's sort of a couple of ways that you could come at this. And I found people like talking about this idea of grounded models coming from the vision perspective, but also coming from the NLP perspective. Both, I think you can kind of get to the same idea from both directions. If you're coming from the computer vision side, you might want to look up, up some, some things about Clip, which is, is a model that's being talked about or a methodology that's being talked about quite a bit. But from the language side, the interesting thing about language is that language is sort of a way to encode knowledge about the real world in some sort of concrete way, right? Like even the words that I'm talking about right now, I have thoughts and ideas in my mind. I have perceptions of the real world and I'm encoding them in a, in a concrete set of symbol representations, right? And so there's a set of symbols that aren't really like, like the set of symbols do g You know, conjures up ideas in my mind about what that those that set of symbols mean. And I know it does for you as well, having 10 or more of them at any given time. But those symbols are representative of things that are grounded in the real world, right? It's just a sort of symbolic representation of things that are grounded in the real world. And so this idea of grounding is thinking about more widely, like, well, instead of, you know, always just having a labeled data set that says like, oh, here's dog, here's dog in my text, and that's an entity, I'm going to label it as an entity of some type and, you know, do named entity recognition or something. Well, thinking about a language model being grounded is a much broader way to look at this where you realize, oh, well, that has other representations that are actually grounded in kind of multiple modalities of how people experience dog. And language is just one of those, right? Or textual language is just one of those. There's also a way to say dog, right? In audio. And there's all sorts of. Of course, the Internet is filled with images and videos of dogs, and there's drawings of dogs. And so there's not a single sort of element that's representative dog in any modality, right? You could. You could do the same from vision, right? If you have a single picture of dog in your data set, your model is going to assume that's the only way you can represent dog, right? And so when you see a different type of dog, it might cause your model to have a vulnerability because it's not used to classifying a pug as a dog when it. All it's seen is a golden retriever, right? And so the idea of grounding. And, you know, if anyone wants to correct me, our listeners can correct me, but it really has to do with pulling together these modalities of data to ground our model in more than just a sort of single symbolic representation of an idea, but building multimodal models that can ground our ideas or our perception of these ideas in these sort of multiple modalities to increase robustness and flexibility, especially if we're using these on a variety of downstream tasks. I feel like I said a lot there. I don't know if any of that came true.

If I'm understanding you correctly. Correct me if I'm not, but you're basically saying having that diversity to achieve what the model is representing. So the representation is built on diversity, gives it that robustness. It gives it that ability to recognize all the things that that representation might take the form of. Is that a good way of summarizing?

Yeah. Yeah. So I looked up a few things here in terms of this idea of grounded. I liked a description from actually from darpa, which, you know, I know you're. You're a fan of, but they talk about the explosive growth of language models and all this stuff. And they say, say, you know, ML suffers from a need to train on large amounts of annotated data. And then they talk about this sort of project that they're talking about, which is related to grounding. Aims to enable computers to acquire language in a manner similar to how children do, because children acquire language based on their perception of oral and visual information, not just textual information. It's not just like, you know, teach a child to read based on just text, apart from audio and visual. So it's a sort of broader way of thinking about how you teach a certain learning. How we teach our models the information that they need to know, which is actually encoded in multiple modalities. And it might not always be, like, audio and video. Maybe it's related to, like, physics, about how the world works or something like that.

Understood. I just want to note, as an aside that I prefer your definition of grounding far more than the one I usually get, which is me on the weekends. It's my wife telling me, no, you're not going to go fly that Cessna. You're going to do yard work instead. You. You, sir, are grounded. That's what I usually hear. So yours is a much better definition in my book.

Yeah, yeah. And you can think of how this would create or could create vulnerabilities. Right. Like, we. Based on our perception of the world, we know that, like, a bird belongs in a tree. Right. But then if I put, like, a car in a tree, we have the sort of common sense to reason that, like, based on all our experience in all of our modalities of existence, we know that that's odd and that's. That's not quite right. But that's. That's not how machine learning and AI systems typically work. They would just know, hey, there's a tree here and there's a car here. And, you know, that can create an issue.

If it's not a whomping willow, it should not have a car in it.

Right. That is definitely true. Okay, well, I'm really interested in this idea of grounding. I think. I don't know if you remember back when we talked with Jeff Adams, who was the. He was on the team or helped lead the team that created a lot of speech technology at Amazon related to Alexa and other things. So he talked about what he was really excited by in the future was thinking more holistically about language. That's particularly what he's interested. But when he said that, what he kind of expressed was that we experience language in all sorts of modalities. So language models and speech technology moving forward is going to be very multimodal, was kind of how he was thinking about that, and that was something really exciting to him. So, yeah, I think this is something that I'm really excited about. And yeah, this idea of grounding is really interesting.

It is. It'll be interesting to see how it develops. I know that in the past, as we've hit in interesting ideas, we tend to have multiple shows then include them in the discussion. So I suspect that grounding is one of the new things and probably foundation is another one. Though I might ask our larger community, please be better at naming, because we've spent a good bit of the show defining some confusing terms as you come into it until you get used to it.

Yeah. So I would say you and I are only just even touching on this idea of foundation models and the other implications of them. Just to give listeners a sense of this, there's other things that are discussed in the work from Stanford, their report, including what I thought was a really interesting one around misuse. So we've talked about kind of the vulnerability of these foundation models, one being that there's sort of a single point of failure. And then we've talked about this idea of grounding, where maybe these foundation models don't always have the common sense that they should or aren't grounded in multiple modalities of how we experience the world. So that's one. But then there's this idea of misuse, which means that. And I thought maybe you would find this one intriguing, Chris, particularly that the idea that you can create a really flexible, extremely powerful model which lets you do downstream things a whole variety of downstream things much easier, means that you can do a whole variety of downstream things much easier. Right. So in other words, you can do named entity recognition, sentiment analysis, other things, but you can also do other things like create fake news or target marginalized groups or, you know, whatever, that you can do those things much easier, too. So, you know, I think that's. That's a difficult one. Like, on the one side, we're creating these large language models that we would like to be more accessible to people because we don't want them sort of the concentration of power around these foundation models being with just a few kind of entities. But the wider you distribute these maybe the more possibility there is for misuse in the community. Which is I think why we saw OpenAI kind of restrict usage of GPT3 for so long. So I don't know, I don't know what the right balance is there or if there is a right balance.

It's interesting, I think that you, you know, when we, a while back, you know, way back early in the show there were really only data scientists and then as, as this industry has matured and we've gotten many different roles with different names, we got deep learning engineers, we got all sorts of different names. Now I think that you just pointed out that there's yet another one there that has come into being and that is you know, like the foundation model security analyst, you know, who is a person charged with evaluating downstream intended consequences by, by a malicious actor. So there's probably a whole cybersecurity industry that you just coined just now. Daniel.

Well, I'll take credit for it.

There you go. It's yours. It's a, it's a resume able thing.

Yeah, it's like coming up with like voice first or. That's right. I don't know yet. Some of those other things.

Yeah, they'll call you Daniel.

Father of, Father of the foundation model security analysis.

That's right, that's right.

I'll take that. So as we finish up here, just to tie in a learning resource to this whole discussion of foundation models, I don't really know a better one than this, but I have mentioned it before and it keeps, you know, growing and I think becoming better is the hugging face course. So if you just look up hugging face course and of course we'll mention it on our show notes. Hugging face. Maybe people think it's just natural language processing, but it has grown far beyond that. And really like if you want to get a sense of like what is it like to use foundation models and import them and fine tune for a downstream task, fine tune a pre trained model or use these big data sets or portions of these data sets that are being used in foundation models. Hugging face and the model hub, the data set hub and their, you know, training API is really the best way to start into that topic, at least in my opinion. And they have their own course, it includes videos and you know, just looking at the thing here, they have already on the left hand side they talk about, you know, give an introduction, they talk about Transformers which is their library, but you can access all sorts of model architectures and then they talk about fine tuning a pre trained model, sharing models and tokenizers. These are all sort of themes that we've talked about in this, in this episode. So indeed, definitely take, take a look.

I feel the need to tell people that you're not paid to do that because you've offered them up as a fantastic learning resource, which they are a number of times. So if our listeners haven't gone there, they really, really should.

They really should. Yeah. And they're, they continue to just boost our own work at sil. So yeah, I don't feel bad at all about, about giving them another plug.

When you're good, you're good, you know, and they're good.

Yeah, I know. Well, Chris, I appreciate you talking through this subject with me. It's been fun.

Thanks for teaching.

Yeah. On to the next. Hope you have a good evening and rest of the week.

Sounds good. See you next time.

Bye.

All right, that is Practical AI for this week. If this is your first time listening, subscribe now at PracticalAI FM or just search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, please do share the show with your friends. It is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world to Breakmaster Cylinder for the Beats and to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time.

Sa.