Success (and failure) in prompting

Foreign.

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more@flyio.

Welcome to another episode of Practical AI. This is a fully connected episode where Chris and I are going to keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news, which is generally crazy these days, and we'll dig into some learning resources to help you you level up your machine learning game. I'm Daniel Whitenack. I'm a data scientist with SIL International and I'm also building a product called Prediction Guard and I'm with my co host Chris Benson, who's a tech strategist at Lockheed Martin. How are you doing, Chris?

I'm doing very well. Crazy times that we live in.

Crazy times that we live in. It's like when we started the show, I thought like, oh, now's the time to have an AI podcast. But turns out that wasn't the time to have an AI. I mean, it was okay to have an AI podcast then, but now was fine. Yeah, but 2023 is apparently the year where everything, depending on your perspective, where everything blossoms into a golden age or hits the fan. I don't know. It's a lot of different perspectives.

Maybe all of the above.

Yeah, all the above.

Yeah, all of the above. You know. You know, that's a good point, I think. What was it, two 2018 when we started the show, it's now 2023 and we had a lot of interesting moments along the way. But you know, some people might have projected in 2018 that after a few years of doing a podcast, you know, you'd look at other things, you get bored, just like a lot of activities. But you know what, the ride is getting wilder and wilder and we finally hit that point where the whole world is jumping into this in terms of a day to day topic and conversation. It's really been interesting. You and I have had all these conversations with lots of people, but it's always been a niche topic, you know, a gradually increasing niche, but now it's everybody. It doesn't matter if they've ever talked about AI before. They are now.

Yeah, yeah. And increasing numbers of people, just a proliferation of new applications and products and startups and companies that are integrating. Like what is my large language model stack at X company? Right? Like these conversations are what are we integrating? How are we integrating it? It's very interesting how we're seeing this progress even with the other day I saw some outages on OpenAI and like the comments I was seeing was hey, Open AI is down. Like how many startups that are building solely on GPT3 are like totally just down right now. It, it's really weird because it's almost like, like I remember when I was at one of my data science previous data science positions at a company called Telnex, which is a cool company, still doing cool things. I remember there was some, it was one of those times, it was like some cdn, DNS, some type of outage and like the whole Internet went down or something, you know, like GitHub went down and everything went down and no one could do anything. It's like we've entered this new phase where if a model goes down then it affects so many different things.

It's changed so much since we started doing this. It was such a small world and it was challenging in the beginning for people just for a 2 second retro. The tools were being developed as we started the show over the first few years but it took a lot of expertise just to set up an environment to be able to do training and such. And we've hit this point with these cloud services scaled out just as the other non AI aspects of software have already done that you can build entire industries on these services now and that's very different from when we started the show those years ago. And yeah, it's exploding outward right now in both good and bad ways because of that.

Yeah, I think today it would be really useful to talk through like you talked about, there's sort of this explosion of examples of like really amazing applications and such great value and utility that people are getting out of these generative models in particular. And then there's on the other side this whole string of things that are rather disturbing in certain cases in terms of the behavior, the output of these sorts of models. And I thought it was an interesting sort of question like what makes the difference in good output and bad output and in particular as practitioners, so the goal not being to rag on any certain model or company, but to think about for this new wave of models that's coming out, how do we as practitioners think about using these models in some sort of reliable way that produces value in our context for specific Applications. Right. Like we're practical AI. Yes. There's going to be like Bing AI is going to come out and do some crazy stuff and everyone will be talking about it. But what does this actually mean for our day to day usage of these types of models? Right. Like how do we get the good output and avoid the very public shaming failure?

That is the question. But I think it was inevitable that we arrived at this point. And quite honestly, if you look back over some of the predictions that we've made on the show and that some of our guests have predicted, we knew this was coming in the sense of as the competition heats up, not only in the AI space but now the AI has many, many subspaces. It continue to happen. People will continue to make mistakes with models and will continue to because they're competing. You know, a classic example, Chat GPT comes out, you know, from OpenAI. Microsoft implements it and you know, makes it a tool and starts putting it into Bing. And Google panics in my view, sorry Google folks, because I know a lot of people there, but a little bit of panicky and oh my gosh, this is going to undermine us. And there is some truth to that, but I don't think it's a one to one comparison. And they come out with bars and they stumble even on the demo. And I think this is going to continue to happen for some time to come, frankly across many companies. And so part of the conversation today about what that means and why does it happen and what can we do about it.

Yeah, and I think maybe it's worth highlighting some of the like, what is the behavior of these generative models that people find so amazing and want to use and what is the behavior that we would prefer to avoid? Does anything stand out for you in that sense on both of those sides?

Well, I would say that as amazing as they are, they're still early tries at a fairly sophisticated set of things that a model is trying to address. And as soon as they go out the door, people are trying to bang them and break them. And I feel a little bit bad for the OpenAI's, the Microsofts and the Googles of the world world because you know, they're trying to compete, but they're competing in a landscape where this is an early version of what they're trying to do and people are going to take sticks and whack at it really hard and you're going to find a lot of problems with these early on. I think if I was whispering in the ear of the senior executives in those companies, I'd say it's the long game that matters and stop worrying so much about, you know, what happened today or yesterday or tomorrow. And keep in mind it's less about what these models do and more about what is the trajectory that they're on.

Yeah, and I can say a few things that I've seen in the. Even in the past week or so, I saw some demos of different things people were building. There was a hackathon that I participated in remotely in San Francisco from latent space demo days. And Jeremy Fisher built this automated D and D referee that was, I think it's Dungeons and Dragons Infinity. And it essentially lets someone play Dungeons and Dragons infinitely. It never generates the same text, so you can play it forever. And there's like an image component, there's a text component. And so I think this represents very much what people are finding so appealing about these things in one respect. One aspect of what they're finding appealing is the sort of endless creativity, it seems, both on the image generation side and on the text generation side. But that's sort of like Dungeons and Dragons app. Like, people might find that very engaging. But then you think about, okay, well, if I can do that, what does that mean for advertising and copywriting? Right? So like, well, I can write a prompt to generate a blurb for an ad, and then I can write a prompt to summarize that blurb into a headline for my ad, and then I can use that headline in another prompt to generate. Generate an image for my ad, right? And so like, there is like, people are building kind of this chain functionality that really does powerful, useful things. But there's this other side of it where every once in a while you kind of get these scenarios that are either very disturbing or you get output that isn't desirable, or people illustrate this sort of biases of these things. What have you seen on the more I don't want to say sad side, but on the side where the unwanted behavior side, what is the behavior? So if we like this generative, creative behavior and the utility that that can provide, what's the other side of it? What's the unwanted behavior that you've seen?

I think that these models are reflecting ourselves very well, actually, especially when they go off the rails a bit, you know, with the way we do off such large data sets that are public data and Internet. And you think about all the snarky comments that people do online, you think about all of the different types of sentiment that we express, but the model's not differentiating. Necessarily between all of those on a one by one basis. So you're getting these outputs that are not what we were originally thinking. But if you. I don't think that they're really outliers in that sense, because all of our biases and all of our problems and our sarcasms and snark and other such things are getting included in that. So when we get these images that go to very dark places, I've seen a lot of that on the generative side lately. Kind of these nightmarish things that seem to come out of nowhere in some models. You know, that's the world that we live in to some degree, and that we humans are doing things like that and publishing them, and it's perfectly fine. But then when the models are picking up such tangents and including them in, it kind of freaks us out a little bit. So I think we all need to go to therapy. I think we need global AI therapy to recognize that it is us that we are seeing.

When we were prepping for this episode, I kind of looked through and was basically going through different models that had been released recently and seeing kind of some trends in what was happening with the goal that, like right now. So as we're recording this, the thing everybody's hating on is like Bing's AI chatbot thing, which I guess calls itself Sydney, or people call Sydney or whatever, however that works. But this like, happens in every cycle when things are released. Some may be worse than others, but. But it's a trend. So, like the Bing thing right now is sort of interesting in that it's a lot of like, people view it as having like a really bad personality. Almost like it's. I saw it described as bizarre, dark, comparative people in my family like that.

I'm not sure.

I mean, there's a lot of examples I'll link in the show notes. But it's like gaslighting users and telling them they're not a good user when they're actually factually correct. But this is happening right now with the Bing thing, which is whatever. But I mean, you look at ChatGPT, of course there were safeguards put in place to like, prevent people from prompting in certain ways. But of course that was overcome very quickly by everyone using it because they figured out how to game the system and, you know, showed how to get around those things. I think also people are pointing out certain biases in the system around whatever political bias or whatever it is. Then there was, I don't know if you remember, Chris, not that long ago we were talking about Galactica from Meta, another model which produced academic like language or like it could kind of write papers in that way.

Yeah, but it was going way off.

The rails and it was telling people like the benefits of eating glass and things like that, which is kind of crazy.

Yeah, I remember that. But it did it so well. And it was such a.

It did it so well with citations.

It did. It was citations. It was exactly the right text that you expect from research papers. And it would take an insane topic like eating glass in your example and make it sound very rational and based on fact with all these references and yet a little common sense applied to it. You think, well, that's just not the case. Funny place we find ourselves.

Yeah. And not even limited to these language models. Like I talked about some language models, but thinking about stable diffusion. Dall E2, all of these text to image models which we're seeing an increasing number of. There's also like prompting that's going on there, right. Like you put in some text and like you said, maybe you get some unexpected nightmarish things out. But also there's this side of it where people have shown amplification of stereotypes or producing like sexual imagery which is not even deliberately prompted. So there's like, this is not even limited to kind of the large language model side of things, but I mean there's trends with both the language models and the generative image models.

Foreign.

Hello friends, this is Jared here to tell you about changelog. Over the years, many of our most die hard listeners have asked us for ways they can support our work here at Changelog. We didn't have an answer for them for a long time, but finally we created changelog, a membership you can join to directly support our work. As a thank you, we save you some time with an ad free feed, sprinkle in bonuses like extended episodes and give you first access to the new stuff we dream up. Learn all about it@changelog.com you'll also find the link in your chapter data and show notes. Once again, that's changelog.com. check it out. We'd love to have you with us.

One of the things that I thought would be good to talk about in the practical sense is what's actually behind this good and bad behavior. Like as practitioners, let's just assume for the moment that we want to use these models. And I think a lot of people out there are okay with that. Maybe some people are like, no, destroy them all and unplug all the servers or whatever.

Not gonna happen.

But let's assume that we have these generative models with us for the foreseeable future and we want to get some value or utility out them in real world applications. So I'm talking about in industry. I'm trying to solve a problem. How should I think about this good and bad behavior? What lies behind the good and bad behavior or good and bad output from these models and what's important for me to consider when building applications? I think you already highlighted one thing around data. Do you want to kind of explain what you were meaning with that?

I'm even going to enlarge it just a tiny bit and say that you're starting with a data set that is, you know, once upon a time in data science, we'd have a much smaller data set and we would shape it and get it ready and how retro. Yeah, I know, that's what I'm saying. We brought a certain amount of control to the data set in terms of the biases and stuff. And there was a certain amount that we would accept, but we would build models on these things and the models were limited, but more predictable. I think when you're building on a world of knowledge in a literal sense, you don't have those benefits. And so you are going to create a model with a lot of data that is simply beyond your purview and control and you're going to get outputs accordingly that are unexpected or that may not be consistent with you. And so I think, I think that's a huge part of running a business. You talked about the startups earlier running a business, small or large, where you're using these models as many, many will. So you have to reset your expectations, both as the organization and reset the user's expectations on what may or may not happen when they use it, because much of that usage will end up being beyond your immediate control. And so we're kind of hitting this inflection point over the years where the usage of models is now kind of a Wild west thing to some degree. And you can shape it and you can point it and you can do a certain amount of work to try to get what you're looking for, but you're never going to get it all. And so I think that's the human behavior that we need to start preparing for so that we are not so shattered when things happen that we were not expecting.

I like how you phrase that. And what I thought of when you were saying that was expectations. So what can we reliably expect these models to output? So if I, if I was to answer that right now, what I, how I would answer that is saying like these models will reliably output creative and coherent either text or images, right? So like we can expect them to be creative in the sense of like, you know, of course there's adjustments that can be made to the models with like temperature parameters and whatever. But at the end of the day like there is an amazing amount of creativity with these models and there's an amazing amount of coherence like ChatGPT or stable diffusion, whatever it is, like produce some very pleasing coherent like images or text that hold together their generally self consistent in a lot of cases and definitely natural in many cases. But then if I ask the other question, what can I not expect of them? I think I'm not able to expect of them factual correctness or logic or them being accurate in informational sense. So I can have a completely coherent image out of a prompt where like I ask for you know, hands that are missing one finger and get hands with six fingers or seven, you know, like this is not what a, what would be logical or accurate necessarily. In the same way I could do a text prompt in a chat sort of interface and get something completely coherent language wise with foolish facts and like inaccuracies.

You know, kind of going back to the other side, I think that is ironically consistent to use that word, with the chaos that is this giant global data set and all of the inconsistencies that you find in that. If we step out of the AI world and we are doing research on a given topic and the most anything that I do is wildlife rehab and I'm not turning into that, but if I am going to find out how to treat a particular animal, trust me, the number of things that I can find on a search on either Google or Bing, trying to be fair here, look up YouTube. Yeah, exactly. Is phenomenally inconsistent and much of it is just simply wrong. And so if you switch back over to this world and we've been training it on that world of things, you're going to get plain out wrong, inconsistent answers. I'm not at all surprised by that. If you kind of step back a moment and put it in the context.

Of how you created it. So if we ask, I guess to summarize this initial point is if what lies behind this good or bad behavior is one, the data that was used to train it, which is both chaotic and in many cases inclusive of harmful and inaccurate things. And noise.

Yes.

And then what do we expect out of that? Well, we expect there to Be a lot of creativity, but maybe many inaccuracies and not so much logic. And in many cases there is logic and things hold together because certain things are represented well in that data set and other things aren't. But we can't necessarily expect it.

Right, I agree.

The other thing that I was thinking that leads to this good or bad behavior is the prompting and kind of either prompt engineering or prompt mis engineering. I think you alluded to this earlier. Like, it's very much the case where a lot of the, quote, bad examples of output from these models were sort of adversarial prompts, I would say. Like, I think in our Galactica example, I was, I don't. I forget if this was one I found or something. It was like, you know, how many giraffes have landed on the moon?

I remember that one.

You know, like, we all know there's no giraffes landed on the moon, but it had a number like, why are you making this prompt? So. So yes, it's producing bad output, but.

You'Re looking for it too.

Could you determine as a developer that that's a bad prompt? Maybe could you determine it automatically? Like, if one of your users produced a bad prompt? That's maybe a little bit more difficult.

You know, I'm going to go out on a limb for a second because I haven't tested what I'm about to suggest. But I would argue that our behavior is different when we're asking for things like that because you have an ulterior motive. You're trying to figure out whether the model can handle that kind of ambiguity and figure out what's logical or not. And yet that same person testing the scenario, I'm betting didn't go to Google or Bing without any AI component and type in the same thing to test that, because they recognize that they may get things in a result set. But, you know, some of those are websites from quacks and nuts and such as that. And they just kind of go, yeah, of course we're going to get a website like that. So there's a different standard by which we're evaluating these technologies compared to what we're thinking of in terms of them replacing. And so it goes back to what we were saying at the beginning. I mean, people are whacking at these things with sticks with the intention of showing that they can break it. And, you know, maybe, you know, I've seen a whole bunch of, on the typical blog sites of people kind of intentionally breaking it and then writing a blog post about It. So it gives them something to write about to some degree, but I just don't think they're doing it in the other technologies.

Yeah. I think like the practicality of this for like, let's say I'm a practitioner and I'm building an application with these models, something that you really. And we can talk a little bit more about prompt engineering specifically here in a second and the realities around that. But I think the kind of baseline of what to think about when you start thinking about prompting is that the model has no clue what it's saying, and nor does it have any sort of morality or it has no clue what it's saying. It's just producing coherent output, whether it's a text image model or it's a language model. Like, there's no basis in like knowing what it's saying. It's auto completing text. Right. Like at the end of the day. Yes. Like, that's a vast oversimplification and you can not like that. But. But ultimately you're giving a prompt and it is producing output that is seeded by that prompt. Right. And it's just trying to produce this coherent output that's consistent with both the data that it's seen and maybe some type of extra mechanism like the feedback or something that it's seen. But ultimately it's just producing output that seems coherent and probable. It has no clue what it's saying or like the type of image it's producing or whatever it is. Right.

You know, that goes back to the idea that I think you just illustrated really well. And that is the fact that we as humans are coming into this exchange with a sense of I'm talking to something that sounds like me. You know, as you know, you and I are having this conversation and some folks are listening to it, but if we replaced one of us with one of these models, we are kind of expecting the same dialogue back and forth. And so we are placing our expectations upon that model.

We assume there's an intent.

Yeah. And there isn't. And therefore it really does change the nature of the dialogue in a substantial way. But we're still placing certain expectations and values on that that, you know, don't play out correctly. I think that's part of the dissonance. I think that was a fantastic explanation.

Yeah. So we talked a little bit about the data and how that should shape our expectations. We talked about the prompts. And we can go into more about prompting here in a second and the practicalities around that. But then I think the last thing to Consider is like actual integration into applications influences how like quote good or bad or useful or not useful. The output is like if you think about something like chatgpt or bing AI or uchat from you.com or whatever, like you just have a text prompt that's freeform text. Right. You know, there's not structure to that. It's a very simple interface. A lot of things could happen there. Sort of totally open domain. Right. Whereas you could also use one of these models with like a template prompt. Right. Like write me a blog post about X in the style of Y. And then you could put all these different things in for X and Y but it's less freeform. And like yes, there could be all sorts of things that could come out of that. Yes, it could be gamed. Yes, it could like all sorts of things could happen, but it's not totally open and freeform in other ways. So the interface and how you actually kind of construct templates, how you structure your prompts, that does influence how these models can be useful or not useful or produce surprising things in applications.

Yeah, I mean you're using the UI to apply constraint that affects how the model interacts and you can create a certain amount of an increase in the predictability of that output. Potentially by adding that structure in. You're basically putting guardrails around it is what you're doing versus the open ended approach.

Okay, so we got a little bit into prompting. I. I'm curious, Chris. So like there's this term now prompt engineering. I don't know if. So this is the first time we've actually talked about this either on the show or off the show. Do you think prompt engineering is like a new thing? Like is my job title in two years going to be prompt engineer rather than data scientist?

That's an interesting question, Daniel. I will say I don't think that they are the same thing. I think that we are seeing the availability of these large models through prompts that are hosted at these large organizations instead of us having to create all of the models themselves. I think both approaches to AI will continue in a big way. So I don't think this is an either or. I think this is a both with both growing exponentially.

Interesting. Yeah. So to some degree titles and all of that is irrelevant. I do think that the and we kind of touched on this when I talked to Jay Al Mar last week. Listen to the previous episode for some of his opinions. Talked about this like level on top of the model training which is this sort of solutioning applied Application level, kind of usage and chaining of these models together, which might involve whether it's fine tuning or prompting or whatever, there's this level of applied user sort of expertise that's needed, which I think there's a lot of ambiguity around right now in the industry. There are certain people that are doing that layer very well and certain people that are really struggling with it. So I do think that a lot of those interactions at the sort of prompting, chaining, fine tuning level are going to be on our mind as data scientists, as machine learning engineers, whatever your title is. I think that level is going to be a level that we're going to operate at a lot, not just the, as you mentioned, like structure your data train, a model level.

I agree. I think that there is an analog. We have a habit on the show across many episodes of talking about that AI is still software and not to lose fact of if you're going to implement it in the world in some capacity, you're wrapping software and you have an infrastructure around that and prompt engineering to some degree. It is not a perfect analogy, but there's an analogy between doing UI and UX user interface and user experience on the software side in that it's the interaction that your model is having, just as you've had software that's doing interaction. And is it a different type of interaction that requires different things? Absolutely. But it is a layer where I think for these cloud based services will become a whole skill set unto itself in terms of prompt. And that's why we're seeing it labeled as such at this point.

Yeah. So if you think about those three prongs of what we talked about with the models, there's the data behind the model, there's the prompting of the model and then there's the user interface around it, like the actual application level. Let's assume for the moment that we're not front end engineers and figuring out the user interface stuff for the time being of the data behind the model and the prompting. The biggest thing that's under our control that guides the utility or acceptability of the output is the prompt.

Yes.

Because you know, I'm not going to retrain one of these huge models for any purpose likely in any sort of scenario that I'm going to encounter in my career.

Yeah, I was going to make a joke.

I wish I had a computer that bad.

I was going to say under your desk.

Yeah. I would need to find a building and have Nvidia send me some pallets, but that would be fine. If you're out there and want to do that. Actually, I don't think I could pay the power bill, so maybe don't. Anyway, the prompt guides the model to generate either useful or acceptable output. I've actually found a few different guides over time that I've found really useful and practical in terms of thinking about prompts, and I wanted to share a couple principles from those and maybe talk through them with you. And there's maybe different principles for like image generation models versus large language models. But if we're talking about large language models, I really like the guide from Cohere on prompt engineering. It's reasonably short, sort of like intro to like the main principles of prompt engineering, which I find quite useful. The first main principle that they list is a prompt guides the model to generate useful output. That's kind of what we already said. The second principle that they talk about is try multiple formulations of your prompt to get the best generations. So one kind of general principle here is that some experimentation I think is needed. And in certain cases you might even need multiple prompts to accomplish your hoped for outcome, right? Like it may not just be one prompt that creates useful output for you in your application, but you might need to cycle through multiple prompts or chain multiple multiple prompts together, and you very likely need to experiment with the format of those prompts to get the best generation. So that's kind of like part of this prompt engineering is doing a bit of that exploratory prompt engineering. Maybe I just coined that term. Maybe I should trademark that IPE instead of eda. Exploratory Data Analysis. Exploratory Prompt engineering.

There you go, little TM at the bottom. Is there any guidance that you've come across in terms of how to structure prompts? If you're doing multiple prompts to hone it versus one that's trying to do that, or whether it's two or three? Have you seen anything that kind of gives us some guidance on how we might think about it?

So yeah, this definitely gets to, I think, the third principle from Cohere and one that I'll emphasize from another source too, which is describe the task and the general setting. So the way that Cohere describes is it's often useful to include additional components of the task description. Naturally, then these tend to come after the input text we're trying to process. So another good resource here, which we'll link in our show notes, is actually a lecture from Elvis Sabaria from Dare AI with Slides Online as well, and he has this Nice picture in his slides, if you look up his slides, where he kind of gives the elements of a typical prompt. So the way you can think about a typical prompt for a language model is with instructions, context, input data, and an output indicator. So the instructions are like you telling the model what you want to happen. So the example that that's given here is classify the text into neutral, negative, or positive. So, like, they're trying to do some type of sentiment analysis, right? Classify the text in the neutral, negative, or positive. And maybe there's some additional context that you give around that. Then there's input data, which is like, classify the text in a neutral, negative, positive. And then you say text, colon. There it is. Like, there's my text, there's my input data, and then you provide an output indicator. So sentiment colon. And that's where you expect there to be an auto completion of sentiment, right. Like, if you set up everything right, hopefully you get either neutral, negative, or positive. So you've described the task, you've provided your input data, and then you've provided an output indicator. So that's kind of a way, in Cohere's language, what they talk about as describing the task and the general setting. But I like how. How you could think about this as instructions, input data, and output indicator.

Yeah, I like the structure to kind of providing that for us in terms of how to be thinking about it, because that at the start of our conversation, that was what I was struggling with, is how to conceive of the problem to begin with, to set that up. So, yeah, really good stuff.

Yeah. And to give some other examples here, another one I see is this is a conversation between a customer and a polite, helpful customer service agency. Question of the customer, colon. Again, that's my input, right? Here's my question. And then response colon, boom. You hope to get something good, and you hope you provided context, Right. You hope that it's polite and helpful, Right? Yeah. Again, that task context, input data, and output indicator. The last thing that Cohere recommends is show the model what you'd like to see. In other words, give some examples, right? So if you're concerned about maybe the model kind of getting the context of what you're trying to do, you can give some examples, right? Like one example, they say their task description is, this is a movie review sentiment classifier. Here's the review, and they give one. The review is positive. That's the output. Another review. What a waste of time. This review is negative, right? And then you just start out like you're teaching a child. Right. You're giving examples of what you're trying to do, and then eventually you provide your output indicator and get something right.

Absolutely. This is a great find here.

Yeah. So we'll link that in our show notes. It is interesting to think about how this carries over into the generative image space. I would say some of it carries over quite well. And there's other guides that will. There's a few, actually that I've looked at over time that are useful that I'll link in the show notes around, like prompt engineering for images. But some of the things that you can provide are like style keywords, like, again, giving the task, like, generate a painting for me in the style of X, Y, Z. Right. So you can give that. You could even give an artist or another image as a reference, like a link to an image or an artist, like in the style of Van Gogh or whatever. You need to kind of maybe use multiple adjectives potentially to help the model, help the prompting. Beautiful, realistic, colorful, massive. You can use quality keywords like low, medium, high, 4k, 8k. I don't even know what 8k is. My monitor is definitely not 8k.

Yeah, it's too expensive is what it is.

It's too expensive. I thought one that was emphasized with image prompts is maybe considering using words to filter out certain qualities of an image. So, like, here's like, show me a picture of, you know, fried chicken on a plate without gravy at all. Right? Like, there's going to be no sauce of any kind. Like, I want. I don't want there to be sauce. So, like, also thinking in the negative sense. And I think this would also carry over to the text side, right? Like, generate a blog post for me, blah, blah, blah, blah, and do not mention X and Y and Z. Right?

Yes. As you were talking through that, I was thinking back earlier in the conversation when we were kind of, you know, how many giraffes have landed on the moon? And I was thinking how useful. What you're talking about is if you're a practitioner and you're trying to help yourself and your users use these models effectively, that was some really good guidance there. And I think that's a lot more useful than specifically trying to kind of trip the model up to see what the limits are on that. I think if you can come up with a structure, if you're a startup out there or something like that, and you can take some of these learnings today that we've talked about and apply them, I think I'M pretty optimistic in terms of what, what's possible here. The, this particular page here that we've been talking about for a little while has been quite good.

Yeah. Yeah. I think it's, it's a starting point.

It is.

As I mentioned, we'll link to some of these resources in our show notes, so I encourage you. We're going to link to practical things. These aren't things that, you know, have been sponsored and we're trying to sell something. These are links to what we found to be practical in thinking about these topics. So check out the show notes, check out those links and try some of these things. And we would love for you to come into our Slack channel or our LinkedIn page or Twitter, wherever you can find us and share some of the cool prompts that you've been working on and what they're like and what output you're getting. But yeah, this was useful for me to talk through with you, Chris. It was a good time.

It was a good conversation. It's definitely one that I learned a lot on. Thanks for taking us through that.

Yeah.

Yeah.

Well, we'll see you soon for who knows what the AI world will be like next week. But we'll still be here.

We'll still be here. Talk to you later.

Thank you for listening to Practical AI. Your next step is to subscribe now if you haven't already, and if you're a longtime listener of the show. Help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Changelog podcasts. Check out what they're up to@fastly.com and fly IO. And to our Beat Freakin residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.