Improve Your Prompting | Using the LARF Framework for AI Outputs

I think one big challenge for me when prompting tools like ChatGPT is I don't know how to evaluate whether a prompt is effective or not. Right. What are ways that you can evaluate the responses of ChatGPT and kind of evaluate the effectiveness as a prompt, even systematically and from a more scientific perspective than just looking at a comparison of how the output is changing?

Yeah, I think that's such a fantastic question, Adil. And I think there's a really simple acronym here that I like to use to effectively evaluate responses from ChatGPT, and that is laugh. Now, it isn't as humorous as it actually sounds. The acronym is larf. And whilst I go into a lot more detail on the course with understanding prompt engineering, I think it would be useful to give a high level overview of how this can make you a more discerning user of ChatGPT to effectively evaluate your responses. So, starting with L, which stands for logical consistency, why I like to start it here is, you know, If I'm asking ChatGPT, look, what are the benefits and drawbacks of the drag reduction system on a Formula one car? And as you can see, we've got Formula one as quite a recurring theme throughout this podcast. Adult, which happens to be one of my, one of my favorite pastimes. Anyway, okay, and it states this list and it goes, oh, it introduces the. Or it makes the car's top speed higher. But then also on the drawbacks it says, oh, it makes the car's top speed higher. All of a sudden you've got this contradicting statement saying that it is also a benefit but also a drawback. And that's. Whilst it's quite a simple example, it highlights that these models are fallible. They do make mistakes. And using the discerning human eye to review the output and check for that coherence, I think is super, super valuable. Moving on, you have accuracy and this tendency for models to hallucinate. And what I mean by hallucination is chatgpt can often state an answer, it can often confidently state an incorrect answer. So if you go, look, who was the first person to walk on the moon? And it goes, oh, it was Buzz Aldrin. Obviously the correct answer is Neil Armstrong, with Buzz Aldrin being the second person. So it's super useful to cross reference these answers with alternate resources. For example, you can add things such as, such as the browsing capability or even use plugins or GPTs that can reference papers and resources that are ultimately infusing the output with factual data that can ultimately lead to a better response. R stands for relevance. So this is essentially meeting the context. And what I mean by that ADOL is you're essentially ensuring that the response aligns with the context and actually what you wanted to get out of the answer when you were writing the prompt. So if you're asking for a list of great restaurant recommendations in London and it says, oh, here's this recommendation, but it's in New York City, all of a sudden you aren't meeting the context of what it is that you wanted to achieve. Excuse me. Now, I think similarly, tools can be a fantastic way of overcoming this and overcoming these limitations, and I'm sure we'll get onto those a little bit later. And then the final part of this acronym, F, factual correctness. So, as we're all aware, these models have a cutoff date. And when you ask a question without the context of online browsing, it's unable to tell you what happened in January of 2024. Now, it might do, but it would be hallucinating, which we talked about earlier, where it's confidently stating this incorrect answer, for example, who won the World cup or other great sporting events or happenings which took place past this cutoff date. Now, why I think it's really important to understand these is it equips you with where ChatGPT strengths, but also limitations lie. And by understanding both, it really allows you to get the most out of the model and enable you to achieve whatever task or action it is that you set out to do.