How Microsoft ML Researcher Actually Uses ChatGPT

All right, so in today's video I'll be talking to a principal researcher at Microsoft. And I thought this conversation was quite interesting because we bring such different viewpoints to it. As you know, I always come from the perspective of power user, a tech enthusiastic consumer that obsesses over finding what's possible with these tools. And she comes from the perspective of a machine learning researcher that works for Microsoft and actually publishes research regularly. And as you know, I usually Talk about using ChatGPT in the web interface and how to find use cases for for your everyday life. And she's particularly interested in how to use it for code generation and debugging. So this is a very interesting conversation. In the beginning we go into the basics of prompt engineering and how to enhance your prompts with techniques like following up or multi shot prompting. And she even talks about chain of thought reasoning. Then I asked her a question that I was super curious to find out the answer to, and that is do you use custom instructions? And if you do, what are they? Then after she reveals that and I give my take on what I personally use, move into the more technical questions of how to use large language models for code generation and debugging. Now, before we hop right into the conversation, I want to point out two more things, and that is that obviously my specialty is not coding and development. As you know, I know my way around Python, but I've never built production ready applications, whereas that is the use case that Besmeer actually deals with regularly. And the last point is that I actually came into this conversation with the expectation of being a guest and being interviewed. And then it turned out that I had to take on the role of actually directing the conversation too. So sometimes I found myself in a spot where I was trying to move the conversation forward while, while providing value, which was a little tricky, but nevertheless, I thought this was a very valuable conversation with a lot of little tips and tricks and a completely different perspective than you usually get from this channel. And I hope you enjoyed this conversation between me and Vezmira facilitated by Microsoft. Let's get into it. So, prompt engineering basics. I think I'm big on frameworks because it's a little abstract to teach people, you know, what to type into that magical little text box, essentially, because the possibilities are endless, right? Like the question of what can I ask is an impossible question because the amount of answers is infinite. So I love to break prompts down into two parts, right? No matter how complex the prompt, no matter how complex the problem you're tackling or Trying to solve it always comes down to two things. The core concept that I always teach at the very beginning is think about prompting in terms of instructions plus context. Instructions are super easy to grasp, right? It's kind of the task that you're tackling. It's the thing that you're trying to solve. You know, write me an email, write me this piece of code, and then the context is the second part to. To the magic, to the formula. And that's where it gets. It can get complicated, right? It can be as simple as write me Python code for Game xyz, right? That's kind of the context. You're narrowing down the possibilities by telling it, hey, I'm not looking for C, I'm looking for Python here. And then you can go super deep with that, right? And that it can also come in like a thousand shades, because if you provided a database, if you provided a code base, if you provided access to a GitHub repo, all of that is going to be taken in as the context. And you're all of a sudden going to be talking to this personal assistant that has read the context that you provided to it. And this is like, essentially, to me, all I like. When we talk teaching prompt engineering, that's what it's all about. It's about defining the context. Because. Because getting the instructions right is super simple. You can get a bunch of templates, ebooks, you can go on Twitter, you'll find a bunch of instructions that are super useful. You know, do this, do that. But the context is where it gets. Well, it can get tricky, but that's also where you unlock all the abilities. So. Yeah. What do you make of that base framework? Vesemira Instructions plus context. Would you like to add anything to that?

I would just like, you know, go one step further and maybe like, think about what one would want this model to sort of help with and what is the situation. There are cases where you'd want the model to do something that you already know how to do very well, but you're kind of just like, you know, lazy to write it down, all that bulky code, and you want to kind of enhance your productivity. And then there's like, situations where maybe you're not proficient with that new problem. It may be a language, it may be like a new library. And then you're just like looking for ideas so that you grasp, like, what the new functionalities are or a new way of playing, solving a problem. And then the third use case can be just like browsing documentation, which we Know, can be very painful oftentimes. So I guess what I'm trying to say is that often, depending on the use case and scenario, things may have to tweak a little bit. And having a more conversational style on things that you don't know is perhaps nicer. And then you get to learn more. And then when it comes to documentation, really giving a prompt that can explore different types of libraries, this is something that I'm using day to day, actually, for many things that maybe I'm not familiar with, just to have a nice comparison between the different libraries over there, rather than reading the documentation of all of them. And then you can dig deeper into things that maybe you're more interested in.

I really like that. I think you touched on one point you mentioned, following up while learning about a new topic. I think that is a core concept that a lot of beginners kind of miss. And it's that you don't have to get it right in your first shot, right? You can. It's a conversational interface, so you can really start out with, you know, what you think you're looking for. And then as the conversation progresses, as long as you respect the token window, the context window, you can add layers on top of that. And what I always advocate is, you know, if you start with, hey, I'm looking for this and that, and then maybe you add in the examples like, hmm, wait a second, I was actually looking for this type of output. Or like, here's a similar app, like, could you make it more like that later on? You can make it more specific. But I think it's just important to kind of like accept the fact that you don't have to get it right with your first attempt. Matter of fact, you can have hundreds of attempts. There's like, the cost to it is virtually zero. So I would encourage people to have an actual conversation. And when you want to, when you know that you're going to be having that conversation multiple times, maybe even daily, as you mentioned in your case, you can start saving some of those conversation snippets and you can start combining like multiple prompts or inputs that you had into like one, right? And the next time you have you face the same problem, you can just like copy paste that one larger prompt that original resulted from you maybe going back and forth 10, 15 times. The way you arrive at that is just by following your intuition, using your social skills. Because we all know how to talk and communicate, right? The fact that can understand each other right now is proof of that. And then just Use the social skills and talk to it. And then you can piece the different inputs together and arrive at something that you will be reusing regularly. And that makes it super easy to begin. Let's get into some advanced techniques here. So if you go beyond the basics of asking it what package you could use, or feeding it some code of yours and asking what you could add, let's talk about. I think zero shot multishot prompting is a great place to start. At the core off all these models, it's just a transformer that like repeats the patterns that has been trained on. So if a pattern is not present in the model itself, you could tell it this is the pattern that, that I'm looking for here. And extending your prompts like that is simpler than most people think. So how could we integrate multi shot prompting into advanced use cases when it comes to code generation?

Yeah, in fact, few shot prompting for code generation is still an open resource problem. In that I don't think we really have gotten it right about how to optimize the examples. And the reason for that is because few shot prompting for previous problems, like for example for prediction problems, has been used as a way to give different instances of exactly the same problem. In that, for example, if you are doing simply just like multiplication, then you'd give some examples of multiplication to really condition the conversation. But in this case, if you say do not know you are in the second scenario when you don't know the solution to the problem. If you knew an example about how that would work, then you wouldn't need to generate new code. So this is where this chicken and egg problem starts with. But there are other ways how you could kind of blend the techniques between chain of thought prompting and few shot prompting to sort of give an idea to the model of like what they could do. So you're kind of like facing a dynamic programming question and you don't know exactly the solution. Right. And there are many dynamic programming approaches out there, but you also don't know which one of them is going to be like most efficient. But you want to tell the model that there are dynamic programming approaches that it could use one idea to give future prompting. There would be to find a similar problem but give maybe even a pseudocode solution that is not in the right language. But just like pseudocode on how to form dynamic programming would make the conditioning work well. Or giving it if you have implemented it in another language or you found a solution in another language and want to give it to the model so that it adopts it in Another language that could be another technique. And on top of all this, like chain of thought reasoning is something that could really help the model to pin down on the parts of the solution that are more important. But it's very important that that chain of thought comes before the solution itself. So this is something that is very inherent in these technologies in that they always feed upon what has come previously in the conversation. And that is super important. And the closer it is to the conversation, the more important it gets. So if we want the model to do chain of thought reasoning, that it has to be prompted so that that reason comes first and then the solution comes next so that the reasoning impacts the solution.

Yeah, just if I may briefly expand on that, I think it beautifully links into what we talked about before, because the conversation that you're having, it's very important to be cognizant of the fact that all that is being now stored and respected. It's like, it's like bringing a new assistant into your office and then work, starting to work with her. So all the work that you've done throughout the day will be stored and will be remembered. And if you open up a new chat, you basically wipe the memory. Right. It's like you give the AI amnesia and you go from the beginning. And you might find that often you get very different results when you Send message number 15 as opposed to sending, sending it as message number one. It's just a whole different conversation. Right. So yeah, just, just realizing how the context works there is important follow up question there. Vezemira, do you use the custom instructions feature when you're using the web interface or how you know how, how much do you use? Let's say that the system prompt, if you're going through the API, how much information do you provide to it in advance before you even start the conversation? Because for anybody who doesn't know you can, you can do that. You can pretty much set it up with a base knowledge. Right. Without even accessing external knowledge bases or anything like that. You can pretty much teach it about yourself, your approach, your preferences. Do you do that in your day to day approach? Because I certainly do, yeah.

Honestly, I only have two of them. One is for just general information. I've kind of customized something that is more basically temperature zero and beyond so that it only gives five facts and doesn't get too creative with a generation. And then just like one for code that just focuses on the main coding problems that I work on. But when it comes to experimentation for research, that is a whole completely different game in the sense that one needs to be very specific in those instructions and even come up with these ideas that encode diversity depending on the problem that we're working on.

Yeah, absolutely. So the temperature obviously is a super useful one if you. I often teach people through the web interface just because it's so user friendly and there you don't really get temperature controls. But that's huge because you can completely eliminate the creativity there and just make it. Except just make it a little more down to earth. But for myself, I include some interesting parts. For example, I like to specify that parts, parts to it, like priming it to only give me, you know, verified data. And it obviously can't do that perfectly, but it attempts to do it. And by you telling it something like that in advance, and by you telling it in advance that hey, every time you reference something, give me the link, it will attempt to do it. So, you know, it's kind of like a coin flip if you do that. But when doing research or when learning something new, I found it super, I found it super helpful to prime it with some of my own preferences. And when I'm doing research concretely for new content, I love for it to be a little more thorough. I love for it to give me a bullet point summarization at the end of every message. I love when it tries to give me the references when it at least attempts to do it. And sometimes I even like a little summary in the voice of a five year old, you know, at the end, just so, just like a super concise summary so you can set it up to really fit your very own needs, providing examples. Let's just, let's just try to, let's just try to dissect that a little more because what I found in my experience is really that that is one of the most powerful concepts you have as a user, as an end user, that is trying to get something done. If you provided multiple examples, specific examples of outcome that you're looking for, it will follow that closely. Like for the learning use case, right. When you're engaging in a new language or library or just trying to explore unexplored territory there, this is not so relevant. Right. But when you're extending your app, which is the other use case that you outlined in that case, where would you go? What are some places to look for examples? Right? What would be some places? Maybe some places across the Internet where you would be looking for such examples if you're developing apps.

Yeah. I also wanted to add a little bit on what you're saying is that you're doing two things at the same time, which is great, you're giving examples, but you're also providing structure, which is so important in that you're saying, this is the structure that I'm looking for in the output and this is how the form should look like, which is super helpful for automating certain things that are maybe repetitive. You wouldn't want to copy paste all those things and modify them slightly like you'd want the tool to do it for you. So structure is the other thing that goes side by side with the examples. Right. It is very impressive how the tools can now add new lines of code without editing the previous lines. Like, it will not break the structure or anything, but it will just add that piece that maybe you were missing or that you wanted to get another perspective.

How much do you go back and forth with ChatGPT, like, just. Just to give our viewers a bit of an. A bit of a feeling for, for what that might look like. Is that like a. Is that a conversation that spans, let's say like two to four messages or, you know, are you like really stretching it and going deep into it? I know it depends on the use case, but generally speaking, when you're like diving into a new topic, new language, new package, whatever it might be. How long are your conversations? Right off the bat?

Yeah, often, you know, it can be a very long conversation. And again, it's impressive that we're able to have these long conversations right now. Often it starts with just picking a library. So basically there is a lot of choice out there. Choosing, it's a commitment every time you pick a library and then you include it in your base code, it's a commitment because you're continue to use it at least for the next few months, and you need to maintain that code and read it over and over again. So maybe some. Often I even ask these tools to basically give me like similar snippets, but using different libraries, and then I kind of compare them side by side and to see, like, which one has the right style for whatever right may mean, you know, based on the context and see whether that is something convenient to use. It's important though, for the audience to kind of like every time there is like a library that is being mentioned in code to check that the library really exists. Because sometimes it can happen that there is like, these models blend many different worlds in a way, like text worlds. And some of them may be pretty close to what you want, but some may not exist. So it's good to do that. First check that the library is there and it's maintained. And once you know that, then you can do a little bit of this experimentation giving like easy tasks to each of the libraries to see which is the right one to pick.

Absolutely. I think that is huge. You know, trust but verify. Like it's a, it's an excellent and very well educated and highly intelligent assistant, but it's still just an assistant so, you know, respect that kind of boundary. And I would also say I remember like recently I had it generate a code snippet for me where it used the pill library inside of Python to interact with images and it used an outdated version and as soon as I fed it the documentation to the updated version, it worked with that and it just updated it. But if I just took it at face value and included it into my app, then I ran into problems pretty quickly. So trust but verify and you can.

Also ask follow up questions again, follow up questions are so important with these things. If there is any aspect of the code that you really care about, then asking again on like, well, you know, can you think about any sort of gaps in this code? Or if we're talking about things like authentication libraries or memory management in maybe like C and C programs, asking follow up questions of like, how do you think this would work in a case where this and this happens? Just to sort of push the boundaries of what the model is, is supposed to do in this case is always good. And you know, it's a very natural question to ask, like, why wouldn't the tool do this on the first place? Like why does it wait for me to ask that follow up question? One needs to understand that, you know, a lot of the knowledge that comes out of these models is always conditioned based on the prompt, based on the input that we are specifying. And if that is not part of the consideration on the first part place, that knowledge will not be surfaced immediately. So you kind of have to have that condition there for that knowledge to appear. And that's what the follow up question is doing. Basically.

This is one of the best parts of the entire model. This is what got me super excited when I started experimenting with especially like GPT4. It's just so good, right? And it's, it's something that we just didn't have a few years ago. I'm just so thankful for it. Especially somebody who hasn't spent his entire life, you know, developing apps. The ability to just take something you built maybe together with GPT and to, to feed it back and to to ask it like, why is this not running? Why is this not working? Why am I getting this error message? Like often with GPT4, the very simplest thing you can do is just copy the error message you get in the terminal or in the console and just throw it back at it. And it will tell you what's going on and it will give you a step by step tutorial on how to fix that. So how often do you use it for debugging in your day to day? And what would maybe be some recommendations for people trying to fix some mistakes which inevitably will occur? Because that's just how it goes.

I mean definitely for checking error messages in that. And it is a good question like why one use web search or just a conversational agent to search for these error messages. I guess the main reason is because conversational agents can be a lot more concise. And then in the web search regime you kind of have to go through different links and see which one answers your question. And that's why that scenario is becoming so popular among developers. But when it comes to sort of like debugging a whole bunch of code where you exactly do not know where the error is coming from and so on, I've been surprised to how the usual practices in software engineering kind of remain the same in that sense, in that you still have to modularize your code and separate it in different functions that are self contained and documented for better debugging. And that is valid also for code that is generated by GPT4 and other tools. And that's just for you as a developer to sort of like know where the error is coming from and really isolate, you know, where the information is coming and what are the possible error messages that are coming from that function. So that you can also ask a more self contained question for debugging later on to these tools. You can just copy paste that function and you can say, well, tell me what can go wrong. This is the error message that I'm getting. This is the function, tell me what can go wrong. And this is much easier than pasting thousands lines of code to the tool and having it try many different things that can happen there. So it will help us as developers, but it will also help the model to focus on the right things.

Okay, so tip of the day, keep your code compartmentalized and if you're using it for debugging, try and tackle it one problem at a time. I think this is one of the most common pitfalls. I think you outlined that perfectly. People just do command or control A, control C, and then they paste the whole thing. And like, if they're writing in one file, if the app might not be that large, with people starting out building wrappers, often it's just like one. It's just one document. They copy, paste the whole thing and then they start asking questions. That's really. That's really confusing to the model a lot of times. So from the ground up, like, from the. From just from the very first step, thinking in terms of, like, code blocks or. I like to always think of it as buckets. You know, just like, start multiple buckets and like, populate them. Step by step will allow you to fix the errors that might occur down the line. All right, so if you enjoyed this conversation and you might want to check out BESMIRA on LinkedIn, Microsoft for Startups on X and LinkedIn. And if you want to learn more about the basics of using the OpenAI API to build your very own chatbots, I have a playlist on all that here where I lead you through it step by step. And that's all I got for today. See you there.