SmartGPT: Make ChatGPT Smarter

Today we are going to be looking at the concept of Smart GPT and how to use this in your own workflow and your own code. So what exactly is Smart GPT and can it make GPT base models smarter? Well, the idea is to implement self critique or self reflection. So the GPT based model are going to think about the answers that it generates, self critique them and and come up with better answers. This idea was presented by Philip in one of his videos. So Philip has an amazing YouTube channel called AI Explained and I would recommend everybody to check it out. Okay, so the question is how does Smart GPT works? So Smart GPT employs a multi pass process through an LLM instead of a single call. The first step in the process is called Ideation this. The second step is called critique or self reflection and the third one is called resolve. So let's look at each one of these steps in a lot more details. During the ideation steps, multiple calls are made to the LLM using the same prompt provided by the user. So normally when a user asks a question or provides a prompt, the LLM generates just one response and that is shown to the user. However, during the ideation phase, we are going to have multiple passes of the same prompt. So let's say the user can select N as a parameter and we will make N calls to the LLM. And based on the N calls, the LLM will generate n different outputs which are called ideas. Now the theory is that if you set the temperature high enough, there are going to be variation among the responses generated by the LLM. The next step, which we call self critique or self reflection. So basically the LLM evaluates all the ideas that it has generated so far and tries to find possible flaws in each one of them. And based on it its assessment, it tries to pick the best possible answer among the answers that has generated. The last step, which is called Resolve, the LLM tries to improve upon the best idea or best answer that it has generated so far. And that is going to be the final output of the LLM. This approach builds on top of the chain of thought prompting and seems to improve the output from the LLMs, especially for the prompts or the questions that involve logical reasoning. How do you use this in your own workflows? Well, if you're already using LangChain, you are in luck because LangChain recently implemented a new chain called Smart LLM Chain. But there are a few things that you need to remember before using Smart LLM Chain in your own workflows. The first one is that the LLM will use a lot more passes compared to normal implementation. So for example, if you select n equal to 3, it will make three calls to the LLM during the ideation phase, then another call during the self critique phase, and one more call during the resolve phase. But the cost is going to be higher than using normal prompting. The second thing to remember is that it's not going to work with all types of LLMs. The underlying LLM has to have self reflection capability. Smaller models often don't have that ability. And the third thing to keep in mind is that the underlying LLM needs to be able to return just a single output, not multiple outputs. Okay, so with all this in mind, let's look at a code example of how you can use this in your own projects. Okay, so for a code example, we are going to be using the code provided by LangChain and I'm going to walk you through a step by step process. Now, in order to test this, here is the prompt that I want to use. So the initial prompt was if it takes one hour to drive five shirts in the sun, how many hours does it take to dry 30 shirts? And then for added context, I said all shirts are the same type. Assume there is enough space for all shirts and all other factors remain constant. Now the initial answer from GPT4 was that it's going to take around six hours. So basically what it did was it calculated the rate of drying based on five shirts. That will take around 0.2 hours per shirt. And then it simply multiplied that by 30 to get an answer here. But then I said there is a logical flaw here. Do you think you could figure it out? And GPT4 was smart enough that it says that. Okay, yes, you're right. My apologies. The drying time for the shirts should not be directly proportional to the number of shirts. Right? And then it figured out that if it takes five shirts to dry in one hour, so irrespective of the number of shirts, it will take one hour, assuming that everything else remains constant, that it would take one hour to dry 30 shirts, given all the condition remain constant and there is enough space to lay them out for drying. Right. So here we are using GPT4. Okay, so first let's look at what packages do we need. So we need LangChain, OpenAI tick token. And then there's another package called LangChain experimental. So these are different experimental features that LangChain is implementing. Then you need to provide your OpenAI API key. Specifically for this case, we're using OpenAI's models, right? Then you need to import prompt template, chat OpenAI model. And then from the LangChain experimental package, we are importing the Smart LLM chain. After that we are setting up our question. So we're calling it a hard question. And here's the prompt that I showed you that I used with GPT4. And then we are setting up our prompt template. So we provide the hard question as a prompt for the LLM. Again, here we're using the GPT 3.5 Turbo with a temperature of 0.3. You can set it to a very high value if you want more variation in the answers that it produces. Okay, so first I wanted to experiment with the default chain. So I define LLM chain and use this specific LLM that we defined. Then provided the prompt and set the verbose to true when I ran this chain. So here's the answer. It says if it takes one hour to drive five shirts, then it will take 1/5 hour to dry one shirt, right? And then basically it simply multiplies that by 30 and gets to an answer of six hours. If you assume that they are being dried independently, then this answer is correct. But if they're being dried simultaneously and there is enough space, airflow and everything, then it should take around one hour. Okay, now let's see how you can set up your Smart LLM chain. So you need an LLM, then the prompt, then the number of ideas that you want to generate or number of times you want to run the same prompt through the LLM. So in this case I'm setting N IDS to 3, but you can set it to a larger number. Now keep in mind that it's going to be making that many API calls, so if you set it to a high value, it's going to cost you more. Next, we are simply running that chain. So basically, if you see here, it is generating three different ideas. So this is idea one. It comes up with a whole bunch of calculations and the answer is six, right? So idea two, again another computation and it comes up when it answer 6. Then another idea is to find the answer, we can set up the proportion, right? So again, very similar calculations and comes up with again the same answer, which is six hours. Now, after the ideation phase, there is a second phase called critique. So this is basically self reflection, where the model looks at the answers that it has generated and see if it can find any flaws. Right? So for example, let's look at the first idea. So it says that the flaw in this answer option is that it assumes linear relationship between the number of shirts and the drying time. And then it goes on to say it assumes that if it takes one hour to dry five shirts, then it would take six hours to dry 30 shirts. However, this may not be accurate as the drying time may not scale linearly as the number of shirts. Right. Additionally, it doesn't consider any potential limitations or constraints that may affect the drying time. Okay, so basically it tries to critique whatever it came up with. And for the other two ideas, it has very similar critique. And the model is thinking that it was not considering all the different factors that could affect the calculations. Now, during the resolutions phase, it looks at all the three answers and tries to improve upon the best answers. So it says, after analyzing the three options, it is clear that none of them provide a comprehensive and accurate solution to the problem. However, I'll provide an improved answer that takes into account potential limitations and constraints in the drying process when drying a large number of shirts at once. And then it goes on and talks about different factors that can affect the drying time. But here's the interesting part. Actually it says that it is necessary to consider the drying capacity of the drying area. If the drying area can accommodate all 30 shirts without any overlapping or obstruction to airflow, it is reasonable to assume that the drying time would be similar to drying five shirts. So if you think about this answer, it kind of got the answer what we were looking for. And then it says if the drying area is limited in size or airflow, it may take longer to dry all 30 shirts simultaneously. So if you look at this initially, the answers that it has generated, now it's simply critiquing them and trying to come up with a much better answer. Now, I ran the Same prompt through GPT4 using the smart LLM chain. However, in all these cases, it came up with the same answer of six hours. And even the final answer is six hours. But then again it says, however, due to potential variable mentioned earlier, it is important to consider that it's. It's an estimate rather than exact figure. The actual drying time could be influenced by changes in weather condition, how shirts are laid out for drying, drying and material of the shirts, etc. Right. So it does critique some of its answers that it has produced. Now, a couple of other things that I want to show you during this video. So so far what we saw was we were using the same LLM for all three different steps. However, you can use different LLMs for different steps. For example, for the ideation phase, we're using GPT 3.5 in this case and then for the rest of the two phases we're using GPT4. Now ideally you want to use an LLM with higher temperature for the ideation phase so that there is variation among the responses that are generated by the LLM. But for the subsequent critique and resolve phase you probably want to use LLMs with lower temperature. Now here is another example in which I'm defining different LLMs for all three different stages. And if you see for the ideation phase I have a temperature which is really high, probably around something, point 7 is the best value. Then for a critique you want to use a lower temperature and for the third one which is the resolve phase I'm using a temperature of 0.0 right? So this is different setups that you can use with the Smart LLM chain. It's a very interesting technique that forces the LLMs to self reflect on its answers before generating the final answer and it could improve the performance in certain use cases. I hope you found this video useful. Let me know different applications that you can use this in the comment section below. Thanks for watching and see you in the next one.