Blueprint for an AI Bill of Rights

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly. Deploy your app servers and database close to your users. No ops required. Learn more@flyio.

Well, welcome to another Fully Connected episode. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up your machine learning game. I'm Daniel Whitenack, I'm the founder of Prediction Guard and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing today, Chris?

Doing pretty good. How are you today?

Doing really well. I told someone earlier today it was a beautiful day outside and I've got a lot of interesting things to work on. So yeah, I don't know that I can complain.

That's a good way of looking at the world, I gotta say. It's a beautiful day outside here in the metro Atlanta area and I also have some pretty fun stuff to work on. So you know what? We don't have anything to complain about, do we?

Yeah, yeah, I don't think so. And it seems like there's just fun things to talk about these days. I don't know if you and your immediate circles have been talking about this superconductor stuff that's happening. Have you been watching the sort of room temperature superconductor buzz? I guess is the.

I have not told me about it. Let's hear.

Despite having some background in physics, I have not looked at any of this, so I can't really comment too much other than just following it from the sidelines. But apparently there was a research group that claims to have created a superconductor that superconducts at room temperature. So people might have seen these videos in the past of like little things levitating on something that's really cold. That's kind of the typical image of a superconductor.

Like usually isn't it absolute zero kind of temperatures and stuff.

Yeah, it's like measured in kelvin, really low temperature sort of thing. So of course this is very intriguing and I've seen a number of things. There's like one group that's claimed they've reproduced it there's others that are skeptical, but trying to reproduce some of the results. So it's like, I forget the name of it. I'm going to butcher this. I think it's like LK99 or something. Yeah, LK99 superconductor. So if you want to look at some cool stuff that doesn't involve transformers and neural networks, that's cool.

That sounds interesting. So as a non physicist who likes physics, but a non physicist, and since we are the practical superconduct conducting podcast, of course, practical.

If you're going to ask me to explain all this, I'm afraid I'm too rusty to do it. Good job.

Okay, fair enough. I was just going to say, like, what are some of the practical uses of a room temperature? I imagine there's tons.

Well, yeah, I think in general it's like a superconductor, as the name might suggest, conducts, which if you think about that as basically everything used in electronics is some type of conductor.

Good stuff.

Yeah, good stuff. And a superconductor, typically, if it's operating near absolute zero, it's not really that practical to put in your. In your everyday electronic items. So something that is room temperature, I think could open up possibilities, I guess.

I understand.

So I'm not again, the superconductor expert, but people might be familiar with semiconductors as well, which of course are very important to electronics. And the supply of those in recent years has caused a lot of news because, you know, chips for cars and such have had issues in the supply chain and all of this stuff. So you could think about these materials fitting into a similar, similar zone of research.

But it's really interesting and though I don't talk about it much in my day job, there are places where I intersect with microelectronics and there. And that is not my area of expertise. But I do know that there's quite the revolution going on in that space. And so this may be yet another aspect of what may propel the hardware side of things, which I am generally not very knowledgeable about.

Yeah, there's a ton of stuff going on right now in that space. Even in the small town where I'm at where Purdue University is located, they're killing it on a lot of fronts, but they just established a huge partnership to build a bunch of semiconductor research facilities around Purdue because there is a lot of emphasis to kind of decouple chip production from a single location and bring some of that expertise or distribute some of that expertise around. So it's quite interesting to follow some of that, which definitely influences the things that we talk about on this podcast and in a more far reaching or twice removed sort of way. But it's interesting to keep a pulse on for sure.

I have noticed more and more the kind of convergence of microelectronics, really modern software approaches and artificial intelligence all converging into things and intertwining and and so maybe at some point we need to have some dedicated episodes about some of those connection points between them.

For sure. Yeah. And that's not the only news that happened this week. So as I do most days, pulling up hugging face and clicking on models and sorting by trending, which is the default, seeing what new models are out there, a couple things to announce, probably the most interesting would be stable diffusion XL 1.0. People might remember on one of our fully connecteds, not that long ago we talked about stable diffusion XL 0.9 I believe it was, although I get confused with all the acronyms and numbers, but that was essentially a research only kind of beta version of what is now released as the general release of the new Stable Diffusion, which is stable diffusion XL 1.0. And yeah, I mean I've played around a little bit with it through Clipdrop and some other places and pretty stunning output. I've really enjoyed it. I've created some posts on LinkedIn with generated imagery. They release it under an Open Rail license, which we've also talked about on this show. It's more open, although I think as we talked about in that episode about Open Rail licenses, it wouldn't be considered open source, quote unquote, but open access in some way. And yeah, it's pretty cool. So I don't know if you're looking at any of the cool images, Chris, but I am indeed. As we talk they say that of course, best ever. Obviously that's the thing to say when you're releasing something they say world's best.

Open image generation model. But to your point a moment ago, the word open gets parsed in all sorts of different ways, correct?

Yeah, there's all sorts of nuances to that. So the things that they highlight in the post are better artwork for challenging concepts and styles, kind of creating a certain feel imparted by the prompt more intelligent with simpler language. So I guess the thought with this is that the model is able to produce more complicated imagery. Like I'm looking at a panda astronaut in looks to be a coffee shop with a iced coffee.

I see that one on their page.

Yes, yes. Apparently that comes from a simple prompt Although I don't see the prompt in their post. It just says simple prompt.

But they need to have more raccoons on it. You know I'm a raccoon aficionado, so yeah, exactly. Pandas are fine. I like pandas. But they really need a raccoon or two.

Yes, similar to you said. I think they say the best. They also say the largest open image model though. We talked about this on the previous show how it's a two stage model. There's a base model and a refiner model. The base model is actually smaller, it's 3.5 billion parameters. The larger model is the refiner model which is 6.6 billion. So that's a final I think they say denoising but sort of refining make the image better step.

Are those model sizes? I noticed they are both under your magical seven number that you educated us on a while back. Would that be to make this accessible to people so they can get in there and download the model and start or not? Does it just happen?

Chan they say that SDXL 1.0 should work effectively on consumer GPUs with 8 gigabytes of GPU memory, VRAM or readily available cloud instances. So this is definitely a one GPU model. Now might not be work on all GPUs depending on how you implement it and how you call it, but definitely accessible to people which I think is really cool. And also that I think puts it in another realm which is interesting. Another thing they highlight around fine tuning. So if you have a bigger model it's also generally harder to fine tune that model for your own purposes. But along with this they talk about kind of out of the box support with Lora or the low rank adapters type of technique where you can fine tune the model in a very parameter efficient way. And so the thought is hey people, this is open now. Create your own fine tunes off of it as well. And I imagine you know this was just released. Well it's been a few days but the 26th of July as we're recording this 2023, I'm guessing we'll see a whole bunch of fine tunes off of this appear on the model hub and elsewhere in days ahead similar to what we're seeing actually with llama 2. So that's the other thing I was going to note on the model hub is just proliferation of llama2s. So we've got if I'm just looking at it right now, so I see stable Diffusion XL base. That's what's trending at the top. Then we've got the base llama 2 from Meta. Then we have stable Beluga, which is also from stable AI and is a llama 2 fine tune. We've got llama 27 billion 32k looks like 32k context link from together computer the chat llama and then I see 1, 2 a whole bunch of other llamas. So we'll continue to see those proliferate. I think that sounds good.

While you were telling me that I was busy playing with Stable Diffusion here with of course raccoons in space.

Oh nice.

And trying different versions of that.

Nice. Well let me know how they turn out and definitely post them on all the places. We need more raccoons in space.

We need more raccoons in the world. I know. Or out of this world maybe.

Yeah. One one Follow up to on the Llama two front, which is connected to the legal topic that we talked about before with Damien Riehl, which people really loved that episode. I love that episode too. We'll link it in the show Notes about the legal consequences of generated content. But I was chatting back and forth and actually Damien got in the Twitter chat which is there's this interesting thing which we didn't really talk about on the show, but a lot of people are doing. I think it's worth just mentioning it on the show because it's kind of a conundrum to me, to be honest, without actually talking it through with a lawyer, which is technically, I think you're not supposed to use GPT output to train or fine tune another non GPT model. So what comes out of let's say you have an OpenAI account and you generate a whole bunch of output from GPT4, which is going to be really good. And then you fine tune an open smaller model on that GPT output and you make the open model good like GPT4. So that's what a lot of people are doing. It's not really like a hidden thing. People are posting these models on Hugging Face and the question on Twitter which I thought was really interesting and maybe user or listeners can consider is, well first off, it seems to break the license agreement with OpenAI, but also machine generated content isn't copyrightable. But also if they do that and then they post the model, can I use the model that they posted on Hugging Face if it's sort of from the what is the thing Poison well or whatever. And yeah, what all of that would actually hold up in court. It's a whole mix of things. So I just I've been thinking about it a lot and think it's an interesting thing that we'll see play out.

Yeah, I mean, I think the term for kind of that, the sourcing is kind of the provenance of it. Am I thinking correctly? There's a point where it becomes very, very difficult to follow that. And you know, if you have enough rabbit holes that you're going down by using the output, I have no idea how that becomes enforceable. Yeah, down the road I think we're seeing a bunch of these licenses. I think we saw one from Meta, which basically said you can use it for anything so long as you don't compete with us. It's me paraphrasing. I just have no idea how we would possibly have an organization that could follow through on that.

This is a changelog news break. Have you already asked Chat GPT how to design a good UI for your new AI app and gotten back bupkis? Well, check out Lang UI, an open source tailwind library of 60 plus responsive and dark mode enabled components tailored for AI and GPT projects. What exactly does that mean? It means prompt containers, history panels, sidebars, message inputs, and all sorts of stuff that are chatbot related. So you can stop asking ChatGPT and build your own ChatGPT with a sweet UI. You just heard one of our five top stories from Monday's Changelog News. Subscribe to the podcast to get all of the week's top stories and pop your email address in@changelog.com News to also receive our free companion email with even more developer news worth your attention. Once again, that's changelog.com news.

Well, Chris, in addition to seeing interesting things play out with models and licenses and open source or not, and all of these conundrums that we're facing, at the same time, we have policymakers in various places trying to figure out what they should be doing with this. I don't know if you've been seeing that.

I have and it's funny, I. I'm kind of conflicted. There's a part of me that looks at this and says it's important. It's a time, you know, I mean, we have so much change. But there's also a part of me, these, the policymakers are just so far behind this audience, you know, this listening right now and the people that are doing this that there's definitely a part of me that's kind of ready to shoot spit wads at them, you know, while they're doing it and poke fun and Maybe that's okay. I mean, it's. Politicians are there for us to poke fun at, so I'll give them a little credit. They're trying.

Yes, yes, they're trying. And one of the things that just came across my path this week is we've heard of this EU AI act and we've talked about it here where they're trying to restrict certain risky uses of AI and other things that seems fairly, I think, to some fairly restrictive. And one of the things that I saw was that there was an open letter from GitHub, Hugging Face Creative Commons. I don't even know how this works. I guess if you're an organization of that size, you know how to get a letter to the right people. I don't even know if it's open. You just post it on a website and hope they read it and hope that articles get published about it, which I guess is what happened. So I don't know if the EU AI act people are actually reading this. I guess that is where I was going with that. But I assume that they are aware of it. Open letter from GitHub, hugging face and Creative Commons and a number of others calling on the EU to ease some of the rules in the AI act, basically arguing that some of the things in the AI act are kind of regulating the upstream open source projects as if they're more commercial products, which they're kind of this open source ecosystem. And I think the fear is that if the restrictions come like they're planned, then that somehow stifles what we're seeing in the blossoming world of open source AI. So there's probably a counterpoint to that, which would be maybe that sort of blossoming is what's creating rapidly creating issues that are hard for people to deal with. I'm not saying I'm taking either one of those stances. I'm just trying to play devil's advocate. If the open source world is really driving this blossoming of things, and the blossoming of things is what are causing people to really have a lot of these risky type of scenarios pop up. How do you put some regulation around that process when you don't want to stifle the open source thing, which I think we all love, we support that on this show and see how it's benefited things, but also create some problems probably. And how do you deal with that tension?

Yeah, it's a really hard nut to crack, you know, to figure out where the right balance point is on that. Because this open letter that we're talking about, you know, it's really getting it that there are all sorts of negative unintended consequences that can come about by taking an action and making it a regulation or a law in doing that. And yet that's being balanced against really broad fear that the public is expressing. You know, there's a lot in the general media and, you know, not the technical media, not AI media, but the general media, there's a lot of stories being published about concern going forward. But then that gets also balanced against organizations with various motivations that are worried about being left behind. They're worried about their competitors or their adversaries getting ahead of them. Potentially any given human in any one of those organizations has that same fear. So if you're a policymaker, how do you approach that problem? You're kind of lagging behind probably already on the technical side, you know, which kind of, going back to the earlier point, and you're trying to regulate something that's as cutting edge as one can be before it ever happens by looking at what you have today and trying to project into tomorrow. It's a tough position to be in.

It is. And I think there's really a lot of fear on both sides that this is fear on the one side because policy is falling so far behind what is the state of the art, but fear on the other side because there's real consequences, like you say, if something is made into a law, whether it's enforced or not, the reality is that it's there. I'm just looking. Jeremy Howard from Fastai has commented on some of these things and written a blog post, but one of the sort of quotes that I pulled out of or that he looked at with the EU act was the fact that sort of any model made available in the EU without passing certain extensive licensing and approvals could face massive fines. And if you think about where those models are coming from, if those are just some developer somewhere creating a model and posting it on hugging face, certainly that's available in the eu, that puts a liability there. So there's real consequences on both ends. Because also if I'm a policymaker, and we've seen this in the US just this week too, people gathering to figure out, like, where do we. What do we do? Yeah, what do. Where do we go?

There's a dissonance between the way technology develops and evolves, which is not strictly consistent with, you know, nationalities and, you know, and legal barriers and legal lines to some degree. And, you know, you've seen that in many different things outside of the Topic we're talking about where people will move it to a different nationality, be aware that where the laws are different, stuff like that. But there's an added complication here, and that is this is, and we keep talking about it, especially this year, this is moving so bloody fast that the ability to render a law or regulation essentially completely ineffective is quite easy right now by simply moving things around the globe and taking advantage of the different things. So it's going to be interesting to see how different legal entities cope with this. How do you make whatever they end up with, whether it's be the, the EU or I was looking at, you know, member of Congress, you know, commenting, the White House has stuff out on it. But how to make that enforceable in the large. If you're really firmly planted, if, like if you're an American company who does most of your business in America, you're regulated by American entities, that's one thing. But a lot of small businesses don't operate that way and they're not strictly limited to that. So I think going back to our legal episode recently where we talked about the ability to enforce is really going to play out in this.

Yeah, there's the enforcement side. There's also the side of this, which is how are policymakers thinking about this and what conclusions are they coming to and what guidance are they providing, whether that's put into law or not? That's an interesting thing to follow. I think that's one of the interesting things in the US this week, one of the things that you sent me, which I think has been happening for some time. Of course, the White House and others have been talking about AI for some time here in the U.S. but there's interesting things like this blueprint for an AI Bill of Rights, which is published. And it's quite interesting. So just from a practitioner's standpoint, I'm coming to this saying, okay, how are maybe non practitioners, hopefully advised by practitioners, how are they viewing the way in which we should go about doing our jobs? Because probably that's going to affect us at some point and maybe they have some good ideas to influence how we do things practically. So it could just be completely ludicrous and it could provide some really interesting talking points. So I was reading through that. I don't know if you saw the, the Bill of Rights.

I just pulled it up. And you know, it's funny, this is kind of late to the game, in my view, several years ago, you know, without going into specifics that could get, that could get me into trouble I was kind of deeply involved in the early details of AI ethics in several organizations doing some of the work. And here's how I ended up. It's hard to come up with good principles, but even as hard as that is, that's still the easy part of the job because the devil is in the details of how you implement implement and what they mean and how you have the nuance to accommodate all the day to day life. Going back to the discussion we were just having about, you know, open models and unintended consequences and such as that, it's really hard to do they have good verbiage from the White House, but I still at each one of the points that they have, I can't help but wonder which of the many ways might you go about interpreting this and implementing any of those interpretations that you have? It's very nice, lovely, fluffy language and not terribly practical AI yet.

Yeah, I think just to give our listeners an example. So this is an example of what policy makers are I guess, giving us as guidance in developing systems. So for the blueprint for an AI Bill of Rights, they have various parts of this, like the actual Bill of Rights. They even say from principles to practice. So they're from principles to practice. So they break this down into several different points. One is safe and effective system. So their bill is you should be protected from safe or ineffective systems. And then they kind of go into what should be expected of automated systems. Well, it should be expected to protect the public from harm in a proactive and ongoing manner. And then they give some kind of ways to do that, like consultation, testing, risk identification and mitigation, ongoing monitoring, clear organizational oversight. So part of me, when I read this, part of me thinks, well, if I go and I try to make my product SoC2 compliant or something like that, I have to do a lot of those things anyway. So why is this different than some of the sort of compliance things that are already widely accepted as compliance things that matter? And maybe it's where AI comes into the automation, there's some large language model reasoning going on or something like that. But a lot of these things would be things that I. I'm already looking at another one. Algorithmic discrimination protections. You should not face discrimination by algorithms and systems should be used and designed in an equitable way. What should be expected? Proactive assessment, representation and robust data guarding against proxies, ensuring accessibility during design, development and deployment. Disparity assessment, disparity mitigation, ongoing monitoring and mitigation. I think a lot of this is good language Right. Some of it blurs the line a little bit for me to current things that exist in terms of compliance. And then some of it is a nice principle, but what do I do with it?

So there was another point on the White House blueprint. It's a point that I see people grappling with a lot. And I don't think we found the right answer yet. And I don't think what they say in it is necessarily the right answer because I, at the end of the day, I don't think it's practical. And that point is near the bottom. They have human alternatives, consideration and fallback. And they start off with the first line in bold. And the first line says you should be able to opt out where appropriate and have access to a person who can quickly consider and re problems you encounter. And the problem that I have with that particular point is I think that's great for right now, for us, in the moment that we're in at this moment, and the level of AI and the level of automation. But in the years ahead, across all industries, we're going to see dramatically increased automation. We're going to see a number of tasks being automated that are beyond human ability to be able to do. And that will be a natural progression. And that may sound scary today as people listen to this now, but I think that that is the evolution ahead as, as it always has been. Long before I came out. If you, you know, moving from horse to the buggy, you know, to automo of thing, we move into new directions and there are new concerns and dangers and we have to mitigate them. But the distinguishing thing about this particular transition that we're just at the early stages of starting is that we will move into things that we can't do in an automated sense, that there is too much happening, it is happening too fast. When there are millions of considerations in a tiny fraction of a second, there's no human that can handle that. And I think that we will certainly make bloopers. But I think that a statement like this is driven by fear. It's fear of what happens if we lose control. And I'm not saying that's not a legitimate fear. I think it's one of those things that we need to be working through in many different areas. But when the White House starts off by saying, oh, no, no, no, whatever we're going to do, there's going to be a human right there. It's not really considering what we're observing here, the steep increase in AI being applied across many industries. So I'm throwing a stone at that particular item.

Yeah, I do see what they're saying in terms of it could be there could be this vicious cycle that develops. Right. Because AI is getting better at doing customer service and generating responses and AI is getting better at automating systems. Right. So to be hyperbolic, right. If I get on a train car that's automated and there's some problem and I'm stuck on a bridge above a river or something like that, and I call the support number and it's a generated voice helping me through my issue. Right. This whole thing cycles through automated systems, you know, not working properly for me and then trying to help me. And maybe actually the. But to your point, maybe the automated system can help me, guide me through that.

It might be better than the human could have been.

Yeah, I definitely get the concern around this sort of cyclical thing and where does a human actually pop in? Yeah, I think there's a lot of systems that will operate at speeds as well and with complexities that it's going to be hard for a human to debug these things anyway. Right. So it's an interesting point. One of the other things that they link in there is this. It's a NIST AI Risk Management framework. A I R C. Part of me wonders as a practitioner, like if I am developing a new software product or I'm offering software in an enterprise setting, there's probably going to be an expectation on me that I go through some process to maintain GDPR, SOC2, type 2, whatever the specific compliance. HIPAA compliance. Right. You know, monitoring you can put in place, there's third party audits, et cetera, et cetera. So part of me wonders is this sort of AI framework kind of, kind of morph into maybe not this one specifically, but will there be sort of risk management frameworks that filter into not necessarily policy. So I think we've been talking about like the White House and governments, there's certain things that could be put into law. But also I could very well see a scenario where one enterprise says to one of their vendors, oh, are you AIRC compliant? And how do you prove to me that you are? Well, maybe it's a third party audit, maybe it's a monitoring system like it's done with, you know, HIPAA or other things. I wonder if we're going to get into some of that as well, where whether or not the policymakers make laws. I suspect we're going to get into some of these scenarios where we'll have some compliance frameworks put into place that certain enterprises start forcing on other providers. Right. Because they're accepting some level of liability for the type of AI reasoning that they're integrating into their applications. So if I'm an insurance company and I'm hiring X vendor to provide some of my AI logic or something like that, making calls into their system, do they have to be compliant in some way beyond the compliance structures that are already in place, like HIPAA and others?

Yeah, you're right. I wasn't trying to cut you off there, but totally right. I mean, that's a huge business opportunity that is to be realized.

You heard it here, Take it and make the AIRC compliance monitoring framework and you can make some money.

Daniel Whitenack, Father of industry right there.

Father. I'm all the time giving away ideas on this show. I probably need to keep some every once in a while. But you know what?

To stick with the AI theme, we'll call you the Godfather of AI compliance, because the Godfather is a popular thing for at least three luminaries that we know.

There are three luminaries. That's funny. I don't know if I want me associated with the whole compliance field, but maybe. Depends how much you pay me, I guess.

Yeah, I was gonna say it may not be sexy, but it's lucrative. So.

Yeah, exactly. You know, I'm looking at some of this stuff, like in the airc, there's certain things that I could see just knowing myself having gone through some of the compliance things, like when you go through a compliance monitoring thing or an audit, it's like, do you have this policy in place? Are you educating people about it? That sort of stuff. And there are certain things in here, in the govern section, they're like, the characteristics of trustworthy AI are integrated into organizational policies, processes and procedures. So I could just see it now, like, how are you integrating the characteristics of trustworthy AI, blah, blah. And you'll have to show in some policy which may or may not be ever read by certain employees, but hopefully, if you're being truthful, it is. So, yeah, I think we could see that soon.

Not only that, but the ironic thing about this is that you have this framework here, but with this explosion, this proliferation of models that we keep talking about and new techniques that are just happening all the time and being released, as that continues to accelerate for some period of time, being able to apply these to that quickly enough for market forces to work will almost certainly require compliance. AI models that can look at new models, how are they approaching and figure out Whether or not they're doing it.

That'S the real meta thing.

It's meta all the way down. It's meta turtles all the way down on this one.

There's the podcast title, Meta Turtles all the Way Down.

Zuckerberg's all the Way Down.

That's right. Well, on these shows I think it's always good as well to share some practical learning resources with people. And I did find one this week, actually. I monitor hacker news for my good dose of humor and vitriol and superconductors and all the things. But one that was really good to point people to would be this Patterns for Building LLM Based Systems and Products from Eugene Yan. Hopefully I'm saying that correctly, and this was a pretty extensive article. So it's very long article and there's various sections in it, but he walks through a lot of the things that people maybe practically are struggling with in terms of building LLM based applications. So he talks through evaluations, he talks through retrieval, augmented generation, fine tuning, caching, guardrails, defensive UI, and collecting user feedback. All of these things. You know, he talks about being used to measure performance, get better task specific results, reduce latency and cost, et cetera, et cetera. These are all the practical things that people are doing day to day as they're building their applications applications. And a little while ago, Anderson Horowitz put out this evolving ecosystem of the LLM app, and it had a lot of these pieces on there, like caching and guardrails and stuff. And I think this dives into a lot of those pieces in a much greater amount of detail. So if you're wondering, oh, there's this emerging ecosystem of the LLM app, if you want to know about various pieces of that, I think this is a good way to understand a little bit more about how those pieces fit together.

Just for audience. You slacked me over the link and I clicked on it as you started talking about it, and as you've been talking, I was kind of glancing. The very first thing I noticed when it came up was how small the slider on the right side of Chrome was, which expressed how long the It's.

A significant article, it's a significant.

And then the next thing I noticed was it wasn't a three minute read or a five minute or even a seven minute read. It's a 65 minute read.

Yes.

And so I started. You're right. I mean, just like having not had the 65 minutes to go through it, just looking at this, it is incredibly detailed. So I can't. I'm going to dive into this after the show today and that's fantastic. Learning resource. I can tell that. Just all these graphs and everything and it's fantastic.

Yeah, yeah, definitely a lot of graphs. The first one you'll see is and have LLM patterns on a scale from data to user and offensive to defensive. In other words, improving performance or reducing cost and risk and kind of plotting those various strategies. And there's formulas if you want formulas. There's plenty of stuff that you don't need to read formulas to understand. But yeah, great resource. I'm very happy to come across this and point people to it.

Well, that was a good one.

Yeah. Well, Chris, I don't know what AI adventures are ahead of us in the coming week, but I'm certainly looking forward to talking with you about them next time on a on a fully connected or with a guest.

We never have a dull week. It's there. There's so much happening that we're always trying to find. Which thing are we actually going to talk about.

Yes.

So it's a fun time to be in this field.

Yes. Awesome. Well, thanks for chatting, Chris. We'll talk to you soon.

Sounds good. Take care.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.