[DataFramed AI Series #1] ChatGPT and the OpenAI Developer Ecosystem

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com I mean, you could pretty much show anything and ask for like what is making this happen. And I think the really interesting thing in some of those examples with multimodal GPT4 is it actually it understands the cause and effect relationships. I think once we start to live with this technology, it'll be much more clear like how valuable and beneficial it actually is to have that level of reasoning and understanding available to you at all times.

Welcome to the Data Framed AI series. This is Richie. I'm extra especially excited today since this is the first of a four part series on generative AI. Generative AI has, as you may have noticed, turned the world upside down in the last few months. Naturally, we're kicking off the series by going right to the heart of the hurricane with a guest from OpenAI. Logan Kilpatrick leads developer relations at OpenAI, supporting developers, building with Dall E, the OpenAI API and ChatGPT. Outside of OpenAI, Logan is the lead developer community advocate for the Julia programming language and is also on the board of Directors@ NumFocus. Today we're going to talk GPT4 plugins and prompt engineering. I'm looking forward to finding out what OpenAI are building and how to use it. Hi there Logan. Thank you for joining us on the show today.

Thanks for having me. I'm super excited about this.

We're going to dive right in And I think ChatGPT is maybe the most famous AI product that you have at OpenAI, but I'd like to get an overview of what all the other AIs that are available.

Yeah, that's a great question. So I think two and a half years ago OpenAI released the API that we still have available today, which is essentially our giving people access to these models and for a lot of people, giving people access to the model that powers ChatGPT, which is our consumer facing first party application, which essentially just in very simple terms puts a nice UI on top of what was already available through our API for the last two and a half years. So it's democratizing the access to this technology through our API. And if you want to just play around with it as an end user, we have ChatGPT available to the world as well.

And just on that terminology, I know a lot of people get confused like what's the difference between ChatGPT and what's the difference between like GPT 3.5 and GPT 4? How do they all relate to each other?

This is a great question and I try. I know most people are not as close to this as me, but I still, it still slightly irks me when I see like the wrong term. Like I see people say chatgpt4 and stuff like that. It just like, it's like nails on a chalkboard for me because I'm so close to all this. But this sort of the really simple like understanding of this is ChatGPT is the consumer facing web application that people use to interact with our models. GPT3, GPT 3.5, GPT4 is the actual machine learning model that underlies the web UI interface. So that's the biggest distinction is GPT 3.5, 3, 4. Those are the actual models. ChatGPT is like the application that sits on top of it. So there's too many GPTs everywhere and now people like build their own projects and companies and applications and they call it like something GPT, which I think adds to this confusion. But that's the basic breakdown.

Brilliant. I'm definitely making sure all our marketing people listen to this because I know a lot of people are getting confused. So yeah, ChatGPT, web application, other GPT plus a number is a model. Okay, so I think beyond talking about the web application, the API is really interesting, particularly for data scientists. So when should you use this API rather than using the web interface?

Yeah, that's a great question. I think the web interface is great for experimentation. I think it's really opened people's eyes to this technology. Like prior to the API you basically had to either know how to code to experience like this large language model technology or go through our platform.OpenAI.com website. We actually have a bunch of examples. We have a playground. It was like the beta version of the ChatGPT UI experience, but not focus on like mass adoption. More focused on, here are people who are trying to use this technology through the API. Here's like a simple way for them to start playing around. So I think if you actually want to build something, if there's some product or service that you're trying to integrate large language model, conversational AI technology into. That's where you would want to use the API. If you're looking to just sort of play around and have a solution to like some simple questions and answers. Probably just want to use the ChatGPT UI itself. That way you don't have to set up your API key and get Python working with all the right versions and all that kind of stuff, I guess. Yeah.

If you want to send like a few messages, then the web interface is pretty easy. But by the time you're sending like a million messages, probably a bit more effort and you want to use the API, right?

Yeah. It's also like, cheaper too. You can actually locally host your own version of ChatGPT. And one of the many challenges of scaling ChatGPT to the level that it's at today is there's a lot of people who are trying to use it. But you can actually again, locally host your own version of ChatGPT using our API. And it'll be available all the time as long as our API is available, essentially access it at your convenience, which is super nice. And like the cost can sometimes be less than if you were to Pay for a ChatGPT plus subscription or something like that.

But it seems like the API is maybe slightly more developer focused. Is it also relevant for data scientists and data analysts as well?

Yeah, definitely. It's something that's very interesting, is a lot of our initial use cases, or maybe not a lot of them, but definitely some of the initial use cases. Use cases were around like, you know, you can do sentiment analysis and clustering and stuff like that, like actual, like typical data science tasks without actually having to write code to do this, which I think is super interesting. Like you could just pass in some. A table of data that you might be looking at as a data scientist and say, figure out these things or like explore these different ideas that I have and it will look at this data and do that work for you. Obviously it's not, it's not doing that in like the statistical sense. Like you wouldn't want to rely on it to put in the data and be like, what's the average of these numbers? It will probably give you something wrong for that. But it can do more like understanding problems probably better than you would be able to like program them and set up the experiment yourself, if that sort of makes sense.

Okay, so these are sort of natural language processing type tasks. Is it going to be text classification and things like that or what?

Yeah, text classification or like clustering things into different, different clusters of whatever the categories are that you would be interested in. Those types of things are all possible using the models. There's also a lot more that's probably similar to what data scientists would want to do with our embeddings. So embeddings is you take a bunch of. A bunch of text and you turn it into a numerical format. It ends up being like a 1536 length list of floating point numbers. So you can take like a arbitrary input size text, turns it into that numerical format and then you can do a bunch of like cool data science operations. The cosine similarity between 2, two of those lists of numbers to tell how related two strings of text are. That type of stuff. All of that is probably more up the alley of data science folks.

That's interesting. So looking at how related two bits of text are, I guess that could be useful for maybe finding duplicate data or near duplicate data, or maybe trying to think of examples off the top of my head.

That's a great one though, because it's actually like really just something simple as finding duplicates. You change the period at the end or like all these weird edge cases that are really difficult if you were to explicitly programmatically try to do these things and with embeddings, with the context similarity search, you could actually do a much better job of figuring it. So it's actually a perfect example in my mind of one of the use cases that makes sense.

Nice. Yeah, I mean it's a really important thing for data quality. And I guess the other really hot thing at the moment is your plugins. So can you tell me what's going on with those?

Yeah. Plugins is the ability for people to essentially build apps with inside of ChatGPT. And I can imagine again, in the context of people doing data science and stuff like that, you can imagine like an Excel plugin. So now you have all the functionality of ChatGPT, but you want to integrate it with Excel. All that type of stuff is possible with this new developer ecosystem that we're building. And it's literally as simple as you create an API, which is essentially some arbitrary endpoints on an Internet server somewhere. People send a request in a certain way and then you send a response back. That's the basic gist. If folks aren't familiar with APIs. And what ends up happening is you can define this API in a way that the language models, specifically ChatGPT can understand that API and then it can make those API calls itself when it thinks that the question or what you're asking it is related to that plugin. So if I install the Excel plugin and I say, what's my favorite color? ChatGPT is smart enough to know that's not related to the plugin. It's just going to give you. It's probably going to say, hey, I'm a large language model, I don't know what your favorite color is, or something like that. But then if you ask it here, I've uploaded this Excel document. Give me the average of this column, and you can write your plugin in a way that it actually sends the Excel spreadsheet to your server. You do whatever the computation is, you return the results, and then you're able to actually see that answer right there inside of the ChatGPT interface. So I think this will open up just so many cool new opportunities that, like, I think even this Excel example that I'm giving is probably a bad example. But, like, it's just like a scratching the surface of what's actually going to be possible.

It sounds a little bit like search engine plugins, where you've got the sort of standard search behavior. And then you can have plugins that say, okay, well, if you want to search a particular website for something interesting, then you can do that. So in terms of who's going to make use of a plugin, it sounds like you can have plugins for other software. So you mentioned the Excel example. Do you envisage the idea of things like corporate plugins as well, where you have one, you can do something for one particular company?

Yeah, and we already have those today. Like, for example, part of our launch of plugins features a bunch of people who have already built a plugin. And some examples of these folks are like Expedia and Airbnb. So you can install the Airbnb plugin, then you can say, I really have this dream vacation, and here are all these things that I would want to do, and you can send that information off and it'll literally just like, return, like, all the information that you would need to make that vacation happen through Airbnb, for example. And it's actually, it's so fascinating to see. Like, I think the probably the coolest example of one of these corporate plugins that I've seen people do is chatgpt to do meal prep for them. So you can say, like, here are my dietary restrictions. Here is the number of, like, you know, calories or whatever that I want to consume per meal. These are the things that I like eating. ChatGPT will just use its base understanding of language to create these meals for you. And then you can install the Instacart plugin and be like, now order all of these items that I need to make these meals on Instacart and it'll actually like do all that for you. And there'll be a one click button. You can go to Instacart and actually make that order. So something that would be like actually a decent amount of work for us as humans to figure out the meal, figure out the recipes for all this, then go and figure out like how many of all these different things do I need to buy? Like probably hours worth of work you can now do using ChatGPT and plugins in five minutes or less, which I think is so cool. And again reduces the friction for something that like my opinion is like, in general would be very positive for your life to do meal prep and you save money and you can get a bunch of healthy food that you're excited about and reduces that friction for people, which I think is really cool.

It's amazing how much effort, just simple like daily life tasks. So yeah, having that sort of reduction of friction does seem pretty useful. It's interesting that it somehow becomes easier to type sentences about what you want and describe what you want compared to actually like figuring out the precise details. Do you see any other areas in sort of day to day life where this sort of things can become useful?

I think one of the things that's always top of mind for me as I'm thinking about this technology is the fact that like we have so much like on my computer right now. I have so much context about the work that I do and the things that I'm involved with. And it is actually almost 100% inaccessible to use my own data. Like I have all this data that's on my computer, I have it in my email, I have it in all these different sources. Again, it's very difficult to like actually take action on that data in any way. And I think that ChatGPT is going to add this layer of you taking your own data, self hosting it with something like our retrieval plugin, where you can run a server locally on your computer and drag and drop a bunch of files into this web UI that's running locally on your computer. And then now ChatGPT can access that data when you want it to and be like, hey, I know Richie sent me an email about this podcast that's coming up. Find me the email that has the Google Doc for this podcast recording that we're doing. And like, it has that context because it's connected to your Gmail, because it's doing all these things when me actually having to find that myself. And maybe this is a simple example because we have search and email and stuff like that, but you could imagine like a much more complicated version of this where I have some files that are sort of nested away and maybe I want to know something more abstract. I don't know who I emailed about this topic, but I was talking to somebody about waterfalls a couple of months ago and I don't remember at all who it was. And like, it can pull that context of using embeddings, using your own locally hosted data through that natural language interface. And I think that that's going to be so powerful for people to like, actually be able to use the data that they own in a way that's productive and, like, makes their life, hopefully, like, all the information in life more accessible. So I'm super stoked about that.

It's exciting stuff. And from what you're describing, it sounds like ChatGPT or GPT becomes more powerful when you're integrating it with other pieces of software as well. So can you talk about how you envisage, like, that ecosystem forming and coming together? Like, how does everything end up linking up together?

Yeah. So I think the idea is that plugins will do this for us, like, at the start of any ecosystem. There's a lot of question marks today of, like, things that we're still figuring out. Yeah, a lot of the question in my mind is around what ends up being part of, like, the core ChatGPT product itself versus what ends up being, like, a great plugin. And I think you see similar tension with this for Apple's App Store, for example, around what stuff should Apple build and Give to all iOS iPhone users versus what do they want to rely on the ecosystem to build and like than just make available via the App Store? So I think there's a lot of those questions for us. But I do think that plugins is going to be that interface, or at least the first version of this interface that we see between language models and actually connecting them to the world. And I really do genuinely think that it makes the technology so, like, I almost can't imagine now a world where ChatGPT doesn't have plugins. Like, it just seems so much less useful to me than like, actually putting in the specific context and taking the specific actions that are going to be super important to do.

I really love that app store analogy and maybe we'll see like the OpenAI app store at some point. But yeah, I don't want to push you on that, but we'll see what happens. So I think another thing that'd be cool to talk about is the image input feature of GPT4. So that was maybe one of the more exciting announcements about GPT4.4. Can you tell us, like, how you expect people to use this feature?

This is another thing that I don't think that people have actually seen a great example of how powerful this technology is going to be in everyday life. So the basic idea is that, yeah, GPT4 is multimodal. Today. If you were to use GPT4, whether it's in ChatGPT or the API, it doesn't accept image inputs. It's in the very limited beta right now, or alpha or whatever the correct terminology is, with one partner, be my eyes, who provide services for low and no vision people. So if you're blind or partially blind, they provide the service where you can essentially have an iOS app and you can take pictures of things in real time that are around you. And then what they've done in the past is there's actually a volunteer, a real human at the other end of that phone who like describes to you what is happening in some particular image. And now with GPT4 and this multimodal capability, we're actually able to just take that picture and in real time tell people what's available. And I think, obviously I'm not impacted by being low vision, but I can imagine for somebody who is having ubiquitous access to this technology is going to be so groundbreaking and so freeing for them to be able to do all the things that we, with people who have perfect vision, are able to take for granted in everyday life. So I'm super excited about that. I think it'll be awesome to see what that unlocks in that particular use case. I also think that there's just so many more like incredibly cool things that are going to be possible that integrate the image functionality with text. Like even like in the demo for GPT4, Greg did that amazing example of sort of drawing out the visual UI that he was interested in, taking a picture of it and then saying, like saying to GPT4, hey, build me this website and make it look visually like this example that I've drawn out. And I think that is going to unlock so much creativity for people who, even for myself, like, I have a really hard time doing like front End visual interface design, like, it just doesn't. There's something about it that just doesn't really click well for me. But I do think that I can concisely draw something out by hand and if I'm able to do that, or even take a picture of another website and be like, hey, build me something that's similar to this. I think that's just going to be so, so transformative to open up access to this technology. It's going to be awesome.

That's a pretty cool example with. Well, first of all, the accessibility stuff, I think it's often very easy to forget that not everyone has a full set of working senses and people have different needs and being able to help those people through whatever medium is most comfortable for them is going to be really powerful for that sort of accessibility. The other idea about designing websites is also interesting. I'm wondering, are there any like, data science specific use cases? I think, like, there's a lot of database going on and how might GPT be useful there?

Yeah, this is, this is a great example. I think one, perhaps not that useful example is. And there's a bunch of iterations of this in a very similar context, but you could take a picture of some graph, like some different graph and be like, write me the code to actually. And we had this working as a demo and ended up going for something else instead. And it just takes whatever the way that the graph is moving, whatever kind of graph it is, it takes that and then it'll produce for you, like the Python code that will generate the graph that looks exactly like that. How useful that is for people doing data science is a little bit questionable to me. Like, I'm not sure what the actual use case for that is. It's cool to see. I think another version of this is, you know, you could take a picture of a bunch of columns or rows in an Excel spreadsheet and be like, write me the Python code to do X, Y and Z. I think that maybe will be where it's more useful and maybe like just a simple screenshot of a couple rows and you put them in ChatGPT and say again, find me the mean or the average or whatever it is. Build me a regression model. I think that will be cool. Again, I'm not sure that that will end up being the killer use case for data science folks, but I think that's like, that's where we'll be able to start, which I think is exciting.

Actually, that first case is something I have stumbled over where someone's shown me some like, paper journal, like from 1970 or something, and be like, okay, can you just reproduce this plot here? And you're like, okay, how do I do that? Having that done automatically would be really fantastic. Although that's a niche academic use, the idea of being able to show a plot and then have some text explanation of it does seem really useful. Is that something that's possible?

Yeah, most definitely. I think all of those types of explanations will be super powerful, like super powerful and actually very feasible to make happen. So I think you could pretty much show anything and ask for like, what is making this happen. And I think the really interesting thing in some of those examples with multimodal GPT4 is it actually it understands the cause and effect relationship. So there's a bunch of pictures of something being held by a bunch of balloons. And you can ask through GPT4 what happens if I, you know, cut or remove the connection between the object and the balloons. And GPT4 will say, well, you know, the object will fall down and the balloons will go up. And it has that, that understanding of these images and the text that's associated with them. Again, I, I think once we start to live with this technology, it'll be much more clear, like how valuable and beneficial it actually is to have that level of reasoning and understanding available to you at all times. But even for me right now, like, I just haven't had a ton of time to play around with it and like, see how it's going to be useful in my life. But yeah, I'll be interested to see. We'll have to do another time, less than a year and a half and see how this has changed significantly over.

Absolutely. Actually, I have a really stupid question about this, but it's been bugging me. So there's a sort of running joke about how with all these sort of generative AI like image things where, like, you've got dall E and the stable diffusion, all these where if you try and get them to draw hands, like they can't count the number of fingers correctly. I'm wondering, does it work in reverse? Can GPT understand hands properly as an input?

Yeah, that's an interesting question.

I know this is a very specific question.

Yeah, I think it'll get better at doing this. I think a lot of those I don't. I need to read a research paper or two and try to understand better, like, what's the actual reason why the model has a hard time with that? Because, like, it's something for humans that is like, very intuitive and very simple. My guess is that this, like, multimodal understanding will help solve part of. Help solve part of the problem, at least.

All right, so the idea that you have, like, text and images together, that somehow improves the model performance, is that correct?

Yeah, I think it's the understanding of the visual piece. Like, right now you can generate an image and do it, like, pretty well, but it still has these problems with hands, for example, and faces is another thing. I think the other piece of it is that. I guess this is actually the real challenge is Dall? E does a really good job, for example, of, like, generating abstract art, because there's no. There's no constraints on what it is. Like, there's no. In some image of, like, the future of, like, buildings and stuff like that. There doesn't need to be doors. Like, it's essentially something that's completely made up. It's totally abstract. And the problem that we have is when you take something that is like, we would expect as humans to be like, there is one right answer for this. What do hands look like? You know, with a bunch of fingers? Like, we have a preconceived notion of what that should look like. It's something that we're very familiar with. And the model then, like, has a hard time of understanding that. It's very. Like, there's all these weird, nuanced things that these language models and image models, in the case of Dall? E have a hard time with. Like, something that's really simple for humans to do and not difficult, like, even just reversing a string. If you take a string and you ask GPT3 or GPT4 to reverse it, it actually won't do a great job of this. And you might sit there and question, like, it's actually, like, a very deterministic, easy thing to do. There's no uncertainty in this. There is one correct answer forward and backwards. And the model just has a hard time parsing things in that way because of the technical limitations, because it's designed to do things at the token level. And then when you reverse it, like, the token indexes, like, change. So it has to do all this, like, crazy mapping behind the scenes to try to make this work. And, like, it just doesn't end up working very well. And I think there's something like that happening with the fingers in the images. I. That's at least my understanding of it.

That's a really interesting explanation, and it reminds me a bit of the idea of trying to do calculations using GPT where it's got a lot better over the last few months, but it's not a perfect calculator.

And yeah, it can actually do calculations, which is the, it's like informed guessing, which is the fascinating thing. Like it doesn't actually have, have the calculation capability built into it to make this stuff work. It's literally just guessing, which is interesting that it's able to get right answers in some cases.

All right, so I'd like to talk a bit about fine tuning models and. Well, actually, do you want to explain to everyone what fine tuning a model is and why you might want to do it?

Yeah. So essentially today we have a bunch of models that are available to be fine tuned. The original GPT3 models through our API are able to be fine tuned. And the basic intuition is you take a model, it has a bunch of weights, it's been trained for the entire thing. What you can end up doing is like making small customizations to the final output layers of the model in order to influence it to more correctly understand whatever data is relevant for you. So a lot of people, I think one of the biggest use cases that people try with fine tuning is, hey, I have some corpus of data for my company or my project, whatever it is, I want this language model to be an expert at this information and really have like ready access to this information. The problem ends up being is most people are not using, at least, you know, people who we work with who are trying to fine tune things are not using fine tuning in the, in the correct case, like it's actually only, at least in the current form, is only really useful for classification tasks like giving it a bunch of examples of the different classes that you're interested in and having the model learn that is actually quite successful. But making the model very specifically follow like certain formats with our current fine tuning infrastructure doesn't work well. It doesn't like it can't like retrieve information really well most of the time. What people are actually looking for when they're fine tuning with our models at least is they're looking for embedding. They're looking for like direct retrieval of some source of truth. They're like a customer support company, for example, or they are trying to use this model for customer support and they wanted to be able to reference different solution guides. And here's the answer to X, Y and Z question. And embeddings are just a much better solution to that problem than trying to fine tune the model. In general, outside of the context of OpenAI, fine tuning is a viable solution for some of these problems. But it's just like, you know, our flavor of fine tuning that makes this a little bit different than like if you were just fine tuning a model in Pytorch, for example.

Okay, interesting. So fine tuning is purely for classification at the moment, but anything else, look for embeddings. Okay, can you give some examples of maybe. You mentioned a lot of companies are doing this wrong. So let's start with like the lessons learned. Then what are some other examples of like where you might go wrong with trying to use this fine tuning?

I think that's probably the biggest thing with fine tuning specifically. I do think that there's. Most of the companies don't end up going totally wrong because they get in contact with us and our team is able to help guide them through that process. The other piece of this as well is that the models that are available to fine tune today are only the original GPT3 base model. So like, all of the innovation that's happened, all the progress that's been made in the last couple of years, those models aren't available today to fine tune. So it just makes it like a much less powerful solution. If you could fine tune GPT4, for example, maybe it'd be better at solving some of these problems than some of those original base models. So it's just, again, it's just a little bit less useful than a lot of other approaches that you could take today.

I mean, it certainly just seemed quite appealing to be able to just throw in a load of your own company data to GPT4 and then have it give better results somehow.

But I think with GPT4, specifically embeddings, plus the ability to have a system message that like, is truly deeply steering the conversation. A great example of this is Khan Academy. So Khan Academy was one of the launch partners for GPT4. If people aren't familiar with Khan Academy, it's a software platform that allows free education for the world. And they've done a great job of making that technology specifically suited for their use. Case to the point where one of the biggest challenges in the educational space is you don't actually want, if you're trying to help someone learn something, you don't want the model to get give an answer and like, by default, the model really wants to give an answer. That's what it's there to do. It's there to answer your question. So if you ask it, you know, how many stars are in the galaxy or whatever, like it's going to try to answer that question and what Khan Academy was able to do just through system messages and some other stuff, is actually make it so that the model was steered away from answering people's questions and acting more as a tutor to help guide them towards the answer, instead of just, here's the answer, and that's. That capability is much more powerful in GPT4 than it is for some of our previous models to actually follow those system messages and set the tone of what you want this AI system to do and customize it for your use case. Like, you could make it say you can put in the system message, you are a customer support chatbot who works for X, Y&Z company. You shouldn't respond in, like, a way that's aggressive towards users, even if they're mad at you. You should always be polite. You should apologize if you say something wrong. All those kinds of things you can put in the system message and the model will actually understand and stick to those as it processes stuff, which is super interesting to see.

That is actually absolutely amazing. The use case you described, because something datacamp is building at the moment, is trying to be like, okay, well, we'll use GPT to give students some help, but not give them the answer. And it's just getting that balance right of being supportive but not actually giving things away. Very delicate.

Yeah, it's a hard. It's a hard line to walk, especially when you can imagine for younger learners who are maybe like, less intrinsically motivated to be going through this learning process, they're really going to be like, continually prompting, just give me the answer. I don't want to do the work, just. Which I totally understand. And I think, yeah, the folks at Khan Academy had to do a lot of work to sort of make this work for them. And I think, at least from the demos that I've seen that Cellcom put on YouTube, it seemed like it was extremely successful for them, which I'm really happy about.

I'm hoping it's going to be successful for datacamp as well. He's looking promising anyway.

Fingers crossed.

Magnificent. All right, so I'd also like to talk about Copilot. So while we're on the subject of using GPT for learning things. So Copilot is a great solution for generating code for you, so it's more developer focused. But can you tell me what are the data science use cases here?

Yeah, I think a lot of, like, if you're doing work in a Jupyter notebook, for example, and you're running your data science experiments, you could imagine that like in the future you're able to. And maybe not even in the future, probably today, with Copilot X, which is what's on the horizon of being available to developers through GitHub, you could describe like the experiments that you want to run. Like I, I imagine remember back to when I was at Apple and I was a machine learning engineer trying all of these different model configurations, we were trying all these different starter pre trained models to then do transfer learning on for computer vision. And I was like by hand like setting up all these different manual experiments and you could imagine that you just say here are the list of the experiments that I want to try in natural language, write me the Python code, write me the Pytorch code to actually run all these different experiments. And with GPT4 powering all of these features, new features on GitHub, you're actually able to make that happen, which I think is so exciting and will hopefully put data science folks back in the position of being really close to solving problems and like thinking critically about how to solve the problems versus just typing keys to set up experiments. So I think for myself like that would be thinking back to doing that work of actually training models and doing data science workflows. Like it would be so convenient to have this tool. And also another layer of this is being able to explain what's actually happening to me as the developer, as the data scientist who's running these experiments. My background was like pure computer science. Like I didn't have a lot of the like statistical background and for some of the models that we were doing at Apple it was more like statistical models. I just didn't have a good grasp of the stats background that was required to make some of that work. And I could just imagine how helpful it would be to like while the code is running and being generated. Here's GPT4 and ChatGPT just like giving me a explanation in simple terms of like how this statistical method or property actually works. That would be so helpful. And now you're like an actual more informed at your job and you could be more successful at it, which I think is. It sounds incredible.

Absolutely. And there's always been the problem with data science is that you need both the statistics skills and the coding skills, not to mention all the other stuff like communication. And if you can get support in all of those areas then that's going to make life easier I think for data scientists. So there was a recent announcement related to Copilot saying the underlying technology has changed and it's all going to be based on GPT4 now. So how is that going to change its capabilities?

I think what the folks, you know, and again, I don't have any inside knowledge on this. I think my guess is what the folks at GitHub realized was that with how powerful some of the models have become to just do like these types of like learning tasks, you can actually get like a great balance of the ability to just generate code, which I think was originally like Copilot in its current form or like previous form to this new announcement was just able to like generate like a, a single completion line of code or like a little section of code. You couldn't like talk to it, you couldn't like iterate with it. It did have the ability to show you multiple generated options and you could choose one if you wanted to, but it didn't have that conversational layer. And I think what GPT4 is going to allow it to do is still be extremely exc. Excellent at generating code for problems, but also have like a deeper reasoning understanding of what's happening in that code that Copilot wasn't able to do before. You couldn't copy and paste a, you know, file of Python and put it into Copilot and say like, hey, explain to me what's happening here. And that's what's going to be possible today with GPT4. Integrating into Copilot just makes the technology much more usable, in my opinion.

And I guess related to this, you talked about deeper reasoning about what's happening. So one of the big objections to using things like GPT4 is occasionally it's going to make mistakes. I mean, the calculator example is maybe one of those. So I guess when is it going to be a good idea to have this sort of automated, like generation of code and things like that, and automated explanations compared to just saying, okay, well I'm going to have to do this manually.

Yeah, this is a really good question. I think it's a deep question. Maybe we won't be able to cover all the nuance in our conversation. I think my gut feel is that a lot of like the first pass of solving some of these problems can be done by models like GPT4, but it still requires somebody who has the deep expertise and understanding of. You know, I did a live stream earlier today and we were trying to essentially build like the. A UI clone of ChatGPT and we were asking ChatGPT to actually generate that for us based on a bunch of context that we gave it and it didn't go perfectly, probably because I didn't do enough prep for the, for actually going and solving that problem live. And the, the sort of takeaway for me for this whole thing was, yeah, you know, ChatGPT was able to generate a bunch of code that looks really great, but when just one little thing doesn't hook up correctly, all of a sudden the whole thing blows up and then you're sort of back to that mode. As a programmer of, okay, now I have to solve this problem. It was great that all this stuff was given to me and maybe it's even a little, I'm a little worse off than I had wrote this myself because now I don't have the sort of intellectual background, context of what this code is actually doing. Why did I do it this way? So in some sense it can actually make that problem a little bit more abstract and harder to solve. But still, I think it requires the use of somebody who like, has the programming background. So this goes back to the broader conversation around will these new productivity gains mean that there'll be less software engineers or less data scientists in the world? Unlikely, because you still need to have somebody who's very capable with these tools in their hands. But I, I do think that the likelihood that you will start off with some AI generated template or first pass at whatever the problem you're trying to solve, I think that goes up significantly in the next six months. Like even for me today, when I'm doing software engineering development or data science workflows, like I'm starting with whatever GPT4 generates for me just because I don't want to have to like, do the extra few hours of work that it would probably take for me to set all that up and get it running myself.

That's interesting. It sounds like maybe the skill set for developers is going to change a bit. So rather than being able to write code, it's more about can you review and understand and tweak code? Is that, does that sound correct? And if so, I guess how do you think that's going to change? Like how we train people to do data science and create software?

I would posit that it's probably still going to be very similar for developers. And I think the context, or at least my thinking on this right now, is that we're essentially all doing this today. Like, I don't have the knowledge base of like how Pytorch works to train a machine learning. Like, I intuitively understand how Pytorch works. I intuitively understand how to train A computer vision model. But if you put an IDE in front of me with no resources, I'm not going to be able to just like type out the code to train a machine learning model right now. Like, I don't have that in my memory to just do off offhand. So what I will do is as a developer, go to the Pytorch website, look at the templates that they have, look at the examples, probably start with one of the examples that they give and then modify that for my specific use case. So it's a very similar process to like what we do today. We just go off to all these different websites and we have to like hunt around for the right resource or try to make some resource like fit into our use case. Like I remember doing this a lot with the Pytorch documentation. We're like, yeah, I'm kind of solving this problem, but I'm kind of also solving this like image segmentation problem. And you know, they don't have the tutorial that like falls in between the two. So I'm like trying to pull things from both and like, you know, it's kind of a hairy situation. And you can imagine that process becomes actually a little bit easier to do when you have GPT 4. Do the first pass on these things for you instead of having to go and look through 12 Stack Overflow posts to like find the thing that you're looking for.

That's a very funny story, looking through 12 different stack overflow posts. So I like that idea that it's basically it's creating templates for you, but it's maybe just a little bit more personalized to your problem. So I guess just going back to your previous example about trying to build a. It was like the OpenAI website using GPT and you said you were struggling with it slightly like getting the right prompts and things. And that does seem to be the other objection I've heard from a lot of people is that they'll go, they'll try and they'll write a prompt and then they'll get a bad response and they're oh, well, this is a stupid model, I'm going to use it again. So it just seemed like you have to provide a lot more detail to the prompts than at least I first thought. So do you have any tips on like, well, how do you go about writing good prompts to get the response you want?

Yeah, I think the mental model for people that puts them in the position where they're not able to get the most of what they want out of this technology is the mental model that you're talking to another human who has expertise in this topic. And the reality is you and I have some shared understanding of a bunch of stuff that like, we haven't agreed on but is probably there. And that if I told you to do something, just the model might actually, you might infer a bunch of things that I didn't actually explicitly say. And this is the exact same thing that happens with the model where I'm going to ask it to do something. It has its own knowledge base that it's been trained on just like any human would, and it's going to go and do things in the way that it thinks makes the most sense given the experience that it has, given the knowledge that it has. So not unlike humans. And I actually think this goes back to great managers or people who are able to really effectively delegate things to people do this part well, where they're able to like explicitly say, here are the things that I'm looking for in excruciating detail. This is what I want to see happen. I think that we don't have to do a lot of that all the time just because we're usually like, okay with the variants in some sense. But like, if you have like a. You might actually do this in your daily job. If you're some software engineer and there's like a list of like requirements that you're. You've gathered from some clients, those things are often like, very explicit, like they want X, Y and Z&GPT 4. ChatGPT can do a great job of understanding those things because it's very intentional. I think when you give the model room to interpret, it's unlikely that it's going to interpret them or infer things in the same way that you would. So you have to be explicit. And again, with GPT4, it's much better at like actually following those specific things that you laid out that you were looking for. If you try, like GPT 3.5, for example, it might do some of them, but like, it's not going to like, listen to all of your instructions, which is what makes GPT4 so powerful.

That's interesting. So the difference is just about its ability to follow instructions, like the difference between the two levels of model. Are there any particular, like, specific bits of advice, like for data scientists in terms of, well, okay, if you're going to like, tell it to do a data science task, like what are the kind of. I mean, you said be specific, but are there Any particular sort of things that you need to know about when you're writing prompts?

One of the things that the model is limited by is like how much context you can give it. For example, we in this demo that I was doing earlier where I was trying to make my own version of the ChatGPT UI that's on the website today, there's a bunch of context that the model doesn't have because it's training cutoff was sometime in 2021. So like all new knowledge that's been created since the end of 2021, the model has no idea about it doesn't whatever breakthroughs in science or technology or new versions of some Python library that came out in 2022, the model isn't going to know about. And what you could do is actually give it example. So it's like you shot learning where you can like take an example of here's how I want this problem to be solved. Here's like the response that I want to get from you. The model can do an incredibly good job at actually solving it and giving you back the data in the way that you've formatted it if you provide that single shot or a couple of shot example. And I think that's probably the best, you know, for a. Today it's not possible with a multimodal input. You can imagine the future where as a data scientist you're like, here's the dashboard that I want. Here's the picture of like how I want the different things set up. Here's an example of like my data input sources and I want to use this version of this library, you know, maybe it's been updated since. So here's a few lines of the code that it might look like. All of that context can be bundled together to create some actual data science dashboard that it just generates for you automatically based on those inputs. So again, I don't know if this is specific enough, but not making assumptions about what the model knows and very clearly giving these examples I think makes it so much more effective at actually solving whatever the problem is that you have.

So it's not just about, well, this single prompt is about give it a bit of training data, say, okay, I want you to do something like this and then ask it the question. That seems like really cool advice. So I'd love to talk about the different interfaces to particularly to Python SQL R Julia, because I think if you're a data scientist or a data analyst, that's what you're going to be using and I know you're one of the developers of the OpenAI Python package. So can you tell me a bit about what's happening with development of that?

Yeah, just nothing I would say groundbreaking. Just an interface to our API through Python. There's a couple of like extra nice helper functionality that comes in the Python package as well. Super widely used. I'm not one of the core folks, I help out a little bit and help triage a lot of the craziness that happens on our open source repositories, but a bunch of other folks doing like the difficult engineering work to make that package work. We also have, I think that library and our Node JS1 are the only ones that are like officially hosted by OpenAI. There's also this huge swath of community libraries across all those languages. They said Julia are probably some of the like other more niche data science languages as well. If you're like MATLAB, maybe there's an integration. I'm not 100% sure, but maybe there is. So yeah, essentially language agnostic, whatever language you're using today, you could use with OpenAI's API, which is really nice and it's like a, at the end of the day it's like a simple curl request. So any language that you could make some arbitrary call to a server endpoint, you could do that with whatever language you're excited about.

Okay, cool. And is the plan to make the R and Julia packages official at all or is these going to be purely community efforts?

Yeah, it's a difficult trade off for us. I think in a perfect world with unlimited resources, we'd probably want to do some of the more popular ones ourselves. I would guess Julia is maybe not one of those, but definitely maybe R as well would be one of those that you might want to consider again. The reality is we don't have the resources to do that, so relying on the community is a nice sort of happy medium. But it does mean that like when we release stuff, all the community libraries, they don't necessarily get a heads up, they have to see it when everyone else does. And therefore things are like a little bit out of date. You have to rely on your maintainer for your community library to actually update it when we release something. So it does introduce that burden for users. But it's untenable for us to do more in that space than we're doing today. It's already a tremendous amount of work just for us to do our Python library. So yeah, it's a challenge.

So it sounds like if you need to Be up to date at all.

Times then Python library at the moment. So I would highly suggest folks to make use of it if possible. We do have an open API spec available, so if you like, there's 1001 different tools that like auto generate SDKs based on open API specs. So that's probably the other way to not building a library by hand using the open API spec auto generate it. And that's, that's probably the most effective way.

All right, so I think a lot of organizations are trying to just figure out, well, how do I get started with this? And I guess they're not sure where to start. So if you want to include OpenAI stuff within your products, what's step one?

I think step one is taking a step back and actually this is probably the same for most things like understanding what is the actual problem that you're trying to solve. Do large language models, does something like chatgpt or embeddings or image capabilities, whisper or audio, is that actually the solution to the problem? And I think there's, I think this question is more important now because of how much excitement there is around this technology. And I think there's a lot of people who are perhaps integrating this technology for the sake of integrating it without taking a step back and saying, does it actually make sense for us to just plop this in as a feature in our product? Or Maybe we, and OpenAI is seeing a lot of, and I'm seeing this as well, a lot of companies that are actually rewriting the whole roadmap and saying, okay, instead of adding this language model based feature into our product, let's actually rethink our product entirely with this understanding of the future of language models being important. And part of that is reshaping our platform to use that technology from the ground up. So that's the conversation that I would sort of push folks towards initially. And obviously it's a much bigger thing to chew off to say, okay, yeah, we're going to completely redo our entire product or platform using this technology. But I think that's the given the moment that we're in, like that's actually the thing that's going to differentiate, like people want that different experience. They're not looking for your product plus a chatbot on top of it. They're looking for like a completely different interface to access the information that or the service that your product is, your company is offering. Yeah. So I think starting with that is the right place and then the actual tech. The cool thing to me is the Technology is actually really easy to integrate. It's literally just making an API call. The hard part is like figuring out the problem, doing all the prompting correctly, figuring out like a data flywheel of like improving the model's performance or improving the prompts that you have based on the data that you're collecting. Those are the more nuanced pieces. But actually, yeah, it's a simple few lines of code to integrate the technology into your application.

Okay, so it does sound like really the hard part is just making sure that you are adding value for your customers and not just building stuff because it's kind of cool.

Yeah, I totally. I mean, I think that there's a lot of companies that are likely building it because they think it's cool and haven't thought deeply about like, how does this actually help people? And is this the right experience that I want to provide to people? And I think the added layer of complexity for like smaller companies and individual developers is there's actually a cost associated with every API call. So every API call to the OpenAI API costs money. It's like relatively cheap to make these API calls, so you can do a lot for a little, but it still costs money. So like, thinking about this is another like forcing mechanism to think about. Like, how do I actually build a scalable business? Like, how do I provide enough value to somebody that they're interested in like giving me money back for the service that I'm providing. And I think you could circumvent a lot of that with a $5 a month Heroku server somewhere and do a lot of things with just that without having to think about how you actually build a business around this. And because this technology costs a little bit more money and is powerful, having that sort of front of mind earlier on in the conversation I think is also important for people and something that I think folks are missing who are at like a much smaller, like obviously big companies are thinking about how to make money, but smaller individual developers perhaps aren't thinking about that. And I think it's important so that you actually are running a solvent product or service, whatever it is, and you don't need to make gobs of money, but you need to make enough money to pay your bills. Yeah.

Absolutely. And I have to say it's like, it does seem pretty cheap to run the API, but I was trying like generating data sets. You generate a five row data set, it's like super cheap, less than a cent. Try and generate like a million row data set. That's going to be a Little bit more expensive because you're paying for the output. So yeah, I understand how thinking about your API costs is going to be a good idea. So I'd like to think a little bit about what's happening in the future. I don't know whether you can tell me what's coming soon from OpenAI or what's exciting in the pipeline.

I think this fusion of multimodal models, which is a tongue twister, I'll say you have to say it three times quick if you're listening to this multimodal models, I think that will end up being more impactful than people realize. I think that will again continue. Today we have audio models. I have no idea whether we'll have the fusion of more of these different modalities together, but like my, my brain says, potentially, yes, it makes sense. I'm not sure if that's technologically possible or not. So the fusion of more of these things together. I would hope that there's a bunch of advancements in the size of these models so that we could run things locally on your computer, on your phone. And we've seen this with some of the open source models that have come out where you're able to like actually run them on a few thousand dollars computer versus having to have like this massive GPU infrastructure and a server somewhere to actually run them. So I think that will be super important because, yeah, there's just a ton of use cases where it makes sense to keep the data that you're processing locally on the device in which it's coming from and not have to send it to a server. So that will be really cool to see. And again, I think the ChatGPT as a product will just continue to evolve and get better over time. Like it's something that the team and all those folks care deeply about and I would expect it to continue to be, get more valuable, more useful, more integrated with now all these plugins that are coming in the future. So it's going to be super exciting. I think that's the general takeaway is that the future is bright. I'm optimistic that this technology is going to help so many people become better at doing the things that they really enjoy and sort of automate away some of the work that they're not interested in doing. And hopefully we can get the balance of all these things right where it's, it's a really positive outcome.

All right, fantastic. Do you have any final advice for anyone who's wanting to develop products around generative AI?

I would get started now, like. Like, I've seen a bunch of quotes that, like, AI isn't going to take your job, but somebody who is using AI is the person who's going to disrupt you. And I think for folks who are writing off the technology or aren't becoming in tune with its capabilities, even if they're not using it on a daily basis, I think that's super important to understand how you can be more effective in what you do using these tools so that you don't end up getting disrupted by the potential change of this technology that's just accelerating so quickly. So definitely check it out, play around with these things, understanding of what the problems they actually solve are.

All right, brilliant. Thank you so much for your time, Logan. That was. That was really informative. Amazing.

Yeah, this was super fun, Richie, thank you for having me. And I'm super excited. I'm hopeful that data camps, tutoring efforts go well. And again, if there's anything that I can do to help, feel free to let me know.

All right, thanks.