[DataFramed AI Series #1] ChatGPT and the OpenAI Developer Ecosystem (with Logan Kilpatrick)

You could pretty much show anything and like ask for like, how is this happening? Or like what is making this happen. And I think the really interesting thing in some of those examples with multimodal GPT4 is it actually it understands the cause and effect relationship. I think once we start to live with this technology, it'll be much more clear like how valuable and beneficial it actually is to have that level of reasoning and understanding available to you at all times.

Hi there Logan. Thank you for joining us on the show today.

Thanks for, thanks for having me. I'm super excited about this.

We're going to dive right in And I think ChatGPT is maybe the most famous AI product that you have at OpenAI, but I'd like to get an overview of what all the other AIs that are available are.

So I think two and a half years ago OpenAI released the API that we still have available today, which is essentially our giving people access to these models and for a lot of people giving people access to the model that powers ChatGPT, which is our consumer facing first party application, which essentially just in very simple terms puts a nice UI on top of what was already available through our API for the last two and a half years. So it's sort of democratizing the access to this technology through our API. And if you want to just play around with it as a, as an end user, we have ChatGPT available to the world as well.

And just on that terminology, I know a lot of people get confused like what's the difference between ChatGPT and what's the difference between like GPT 3.5 and GPT 4? How do they all relate to each other?

This is a great question and I try, I know most people are not as close to this as me, but I still, it still slightly irks me when I see like the wrong term. Like I see people say like 2, ChatGPT 4 and stuff like that. It just like, it's like nails on a chalkboard for me because I'm so close to all this. But the sort of, the really simple like understanding of this is ChatGPT is the consumer facing web application that people use to interact with our models. GPT3, GPT 3.5, GPT4 is the actual machine learning model that underlies the web UI interface. So that's the biggest distinction is GPT, GPT 3.5, 3, 4. Those are the actual models. ChatGPT is like the application that sits on top of it. So there's too many GPTs everywhere and now people, like, build their own projects and companies and applications and they call it, like, something GPT, which I think adds to this confusion. But that's the basic breakdown.

All right, brilliant. I'm definitely making sure all our marketing people listen to this because I know a lot of people are getting confused. So, yeah, ChatGPT web application or the GPT plus a number is a model. Okay. All right. So I think beyond talking about the web application, the API is really, really interesting, particularly for data scientists. So when should you use this API rather than using the web interface?

Yeah, that's a great question. I think, you know, the web interface is great for experimentation. I think it's really opened people's eyes to this technology. Like, previous, prior to the API, you basically had to either know how to code to experience, like, this large language model technology or go through our platform.OpenAI.com website. We actually have a bunch of examples. We have a playground. So you can sort of. It was like the beta version of the ChatGPT UI experience, but not focused on mass adoption, more focused on, like, here are people who are trying to use this technology through the API. Here's like, a simple way for them to start playing around. Um, so I think if you actually want to build something, if there's some product or service that you're trying to integrate large language model conversational AI technology into, that's where you would want to use the API. If you're looking to just sort of play around and have a solution to, like, some simple questions and answers, probably just want to use the ChatGPT UI itself. That way you don't have to, you know, set up your API key and get Python working with all the right versions and all that kind of stuff.

I guess if you want to send like a few messages, then the web interface is pretty easy. But by the time you're sending like a million messages, probably a bit more effort and you want to use the API, right?

Yeah, it's also, like, cheaper too. Like, if you. You can actually locally host your own version of ChatGPT, and like, you know, one of the many challenges of scaling ChatGPT to the level that it's at today is there's a lot of people who are trying to use it, but you can actually, again, locally host your own version of ChatGPT using our API, and it'll be available all the time. As long as our API is available, you can essentially access it at your convenience, which is super nice. And the cost can sometimes be less than if you were to Pay for a ChatGPT plus subscription or something like that.

But it seems like the API is maybe slightly more developer focused. Is it also relevant for data scientists and data analysts as well?

Yeah, definitely. It's something that's very interesting. Is a lot of our initial use cases, or maybe not a lot of them, but definitely some of the initial use cases were around like, you know, you can do sentiment analysis and clustering and stuff like that, like actual like typical data science tasks without actually having to write like code to do this, which I think is super interesting. Like you could just pass in some like, you know, a table of data that you might be looking at as a data scientist and say like, you know, figure out these things or like explore these different ideas that I have and it will look at this data and do that work for you. Obviously it's not doing that in like the statistical sense. Like you wouldn't want to rely on it to like put in the data and be like, what's the, you know, average of these numbers? Like it will probably give you something wrong for that, but it can sort of do more like understanding problems probably better than you would be able to like program them and like set up the experiment yourself, if that sort of makes sense.

Okay, so these are sort of natural language processing type tasks. Like is it going to be text classification and things like that or what?

Yeah, text classification or like clustering things into different, into different like clusters of like whatever the categories are that you would be interested in. Those types of things are all possible using the models. There's also a lot more that's probably like similar to what data scientists would want to do with our embeddings. So embeddings is, you know, you take a bunch of a bunch of text and you turn it into a numerical format. It ends up being like a 1536 length list of floating point numbers. So you can take like a arbitrary input size text, turns it into that numerical format and then you can do a bunch of like cool data science operations. Like you know, the cosine similarity between to two of those lists of numbers to like tell how related two strings of text are. That type of stuff. All of that is probably more up the alley of data science folks.

That's interesting. So looking at how related two bits of text are, I guess that could be useful for maybe like finding duplicate data or near duplicate data or maybe.

Trying to think of examples off the.

Top of my head.

That's a great one though because it's actually like really like just something simple as finding duplicates, like, you know, you change the period at the end, or like, all these, like, weird edge cases that, like, are really difficult if you were to, like, explicitly programmatically try to do these things, and with embeddings with, like, the context similarity search, you could actually do a much better job of figuring it. So it's actually a perfect example in my mind of one of the use cases that makes sense.

Nice. Yeah, I mean, it's a really important thing for data, for data quality.

Okay.

And I guess the other really hot thing at the moment is your plugins. So can you tell me what's going on with those?

Yeah. Plugins is the ability for people to essentially build apps with inside of ChatGPT. And I can imagine, again, in the context of people doing data science and stuff like that, you can imagine like an Excel plugin. So now you sort of have all the functionality of ChatGPT, but you want to integrate it with Excel. All that type of stuff is possible with this new developer ecosystem that we're building. And it's literally as simple as you create an API which is just essentially some arbitrary endpoints on an Internet server somewhere. People send a request in a certain way and then you send a response back. That's the basic gist. If folks aren't familiar with APIs, and what ends up happening is you can define this API in a way that the language models, specifically ChatGPT, can understand that API, and then it can make those API calls itself when it thinks that the question or what you're asking it is related to that plugin. So if I install the Excel plugin and I say, you know, what's my favorite color? You know, ChatGPT is smart enough to know that that's not related to the plugin. It's just going to give you. It's probably going to say, hey, I'm a large language model. I don't know what your favorite color is or something like that, but if you ask it, you know, here, I've uploaded this Excel document, you know, give me the average of this column, and you can basically, you can write your plugin in a way that it actually sends the Excel spreadsheet to your server. You do whatever the computation is, you return the results, and then you're able to actually see that answer right there inside of the ChatGPT interface. So I think this will open up just so many cool new opportunities that, like, people like. I think even this Excel example that I'm giving is probably a bad example, but, like it's just like a scratching the surface of what's actually going to be possible.

That's interesting. It sounds a little bit like search engine plugins, where you've got the sort of standard search behavior. And then you can have plugins that say, okay, well, if you want to search a particular website for something interesting, then you can do that. So in terms of who's going to make use of a plugin, it sounds like you can have plugins for other software. So you mentioned the Excel example. Do you envisage the idea of things like corporate plugins as well, where you have one, you can do something for one particular company?

Yeah, and we already have those today. Like, for example, part of our launch of plugins features a bunch of people who have already built a plugin. And some examples of these folks are like Expedia and Airbnb. So you can install the Airbnb plugin, then you can say, hey, you know, I really have this, you know, dream vacation. And here are all these things that I would want to do. And you can send that information off and it'll literally just like return, like, all the information that you would need to make that vacation happen through Airbnb, for example. And it's actually, it's so fascinating to see. Like, I think the probably the coolest example of one of these corporate plugins that I've seen people do is ask ChatGPT to, like, do meal prep for them. So you can say, like, here are my dietary restrictions. Here is the number of like, you know, calories or whatever that I want to consume, consume per meal. These are the things that I like eating. ChatGPT will just use its base understanding of language to create these meals for you. And then you can install the Instacart plugin and be like, now order all of these items that I need to make these meals on Instacart, and it'll actually do all that for you. And there'll be a one click button. You can go to Instacart and actually make that order. So something that would be like, actually a decent amount of work for us as humans to figure out the meal, figure out the recipes for all this, then go and figure out, like, how many of all these different things do I need to buy? Like, probably hours worth of work you can now do using ChatGPT and plugins in like five minutes or less, which I think is so cool. And like, again, reduces the friction for something that, like, my opinion is, like, in general would be very positive for your life to do meal prep and you save money and you can get a bunch of healthy food that you're excited about and reduces that friction for people, which I think is really cool.

It's amazing how much effort, just simple, like daily life tasks. So, yeah, having that sort of reduction of friction does seem pretty useful. It's interesting that it somehow becomes easier to type sentences about what you want and describe what you want compared to actually like figuring out the precise details. So do you see any other areas in sort of day to day life where this sort of thing's going to become useful?

Yeah, that's a, that's a great question. I think a lot like one of the things that's always top of mind for me as I'm, as I'm thinking about this technology is the fact that like we have so much like on my computer right now, I have so much context about the work that I do and the things that I'm involved with. And it is actually almost 100% inaccessible to use my own data. I have all this data that's on my computer. I have it in my email. I have it in all these different sources. Again, it's very difficult to actually take action on that data in any way. I think that ChatGPT is going to add this layer of you taking your own data, self hosting it with something like our retrieval plugin, where you can run a server locally on your computer and drag and drop a bunch of files into this web UI that's running locally on your computer. And then now ChatGPT can access that data when you want it to and be like, hey, you know, I know Richie sent me an email about this podcast that's coming up like, find me the email that has the Google Doc for this podcast recording that we're doing. And like it has that context because it's connected to your Gmail, because it's doing all these things when like me actually having to find that myself. And maybe this is a simple example because we have search and email and stuff like that, but you could imagine like a much more complicated version of this where I have some files that are sort of nested away and you know, maybe I want to know something more abstract, like, you know, who did I, you know, I don't know who I emailed about this topic, but I was talking to somebody about, you know, waterfalls a couple of months ago and I don't remember at all who it was. And like it can pull that context of using embeddings, using your own locally hosted data through that Natural language interface. And I think that that's going to be so powerful for people to like, actually be able to use the data that they own in a way that's productive and like, makes their life, hopefully, like, all the information in life more accessible. So I'm super stoked about that.

It's exciting stuff. And from what you're describing, it sounds like ChatGPT or GPT becomes more powerful when you're integrating it with other pieces of software as well. So can you talk about how you envisage that ecosystem sort of forming and coming together? How does everything end up linking up together?

Yeah, so I think the idea is that plugins will do this for us. There's, you know, like, at the start of any ecosystem, there's a lot of question marks today of like, things that we're still figuring out, I think. Yeah, a lot of the question in my mind is around, like, what ends up being part of like, the core ChatGPT product itself versus what ends up being like, a great plugin. And I think you see similar tension with this for, you know, Apple's App Store, for example, around like, you know, what should, what stuff should Apple build and like, give to all iOS iPhone users versus, like, what. What do they want to rely on the ecosystem to build and like, then just make available via the App Store? So I think there's a lot of those questions for, for us, but I do think that plugins is going to be that interface, or at least the first version of this interface that we see between language models and actually connecting them to, to the world. And I really do genuinely think that it makes the technology so, like, I almost can't imagine now a world where ChatGPT doesn't have plugins. Like, it just seems so much less useful to me than like, actually putting in the specific context and just taking the specific actions that are going to be super important to do.

I really love that App Store analogy. And maybe we'll see like the OpenAI app store at some point, but yeah, I don't want to push you on that, but we'll see what happens. So I think another thing that'd be cool to talk about is the image input feature of GPT4. So that was maybe one of the more exciting announcements about GPT4. Can you tell us, like, how you expect people to use this feature?

Yeah, this will be. This is another thing that I don't think that people have actually seen a great example of how powerful this technology is going to be in everyday life. So the basic idea is that, yeah, GPT4 is multimodal. Today, if you were to use GPT4, whether it's in ChatGPT or the API, it doesn't accept image inputs. It's in a very limited beta right now, or alpha or whatever the correct terminology is, with one partner, Be My Eyes, who provide services for low and no vision people. So if you're blind or partially blind, they provide the service where you can essentially have an iOS app and you can take pictures of things in real time that are around you. And then what they've done in the past is there's actually a volunteer, a real human at the other end of that phone who describes to you what is happening in some particular image. And now with GPT4 and this multimodal capability, we're actually able to just take that picture and in real time tell people what's available. And I think obviously I'm not impacted by being low vision, but I can imagine for somebody who is like having ubiquitous access to this technology is going to be so groundbreaking and so freeing for them to be able to do all the things that we, with people who have perfect vision are able to sort of take for granted in everyday life. So I'm super excited about that. I think it'll be awesome to see what that unlocks in that particular use case. I also think that there's just so many more like incredibly cool things that are going to be possible that integrate the image functionality with text. Like even like in the demo for GPT4, Greg did that amazing example of sort of drawing out the visual UI that he was interested in, taking a picture of it and then saying like saying to GPT4, hey, like build me this, build me this website and make it look visually like this example that I've drawn out. And I think that is going to unlock so much creativity for people who, like, even for myself, like I have a really hard time doing like front end visual interface design. Like it just doesn't, there's something about it that just doesn't really click well for me. But I do think that I can concisely like draw something out by hand and if I'm able to do that or even take a picture of like another website and be like, hey, there'll be something that's similar to this, I think that's just going to be so, so transformative to open up access to this technology. It's going to be awesome.

That's a pretty cool example with. Well, first of all, the accessibility stuff, I think it's often very easy to forget that not everyone has a full set of working senses and people have different needs. And being able to help those people through whatever medium is most comfortable for them is going to be really powerful for that sort of accessibility. The other idea about like designing websites is also interesting. I'm wondering, are there any, like, data science specific use cases? I think, like, there's a lot of database going on and how might GPT be useful there?

Yeah, this is a great example. I think one, perhaps not that useful example is like, you know, and there's a bunch of iterations of this in a very similar context, but you could take a picture of like some graph, like some different graph and be like, write me the code to actually. And we had this working as a demo and ended up going for something else instead. And it just like takes, you know, whatever the way that the graph is moving, whatever kind of graph it is, it takes that and then it'll produce for you, like the Python code that will generate the graph that looks exactly like that. How useful that is for people doing data science is a little bit questionable to me. Like, I'm not sure what the actual use case for that is. It's cool to see. I think another version of this is like, you know, you could take a picture of a bunch of columns or rows in an Excel spreadsheet and be like, you know, write me the Python code to do X, Y and Z. I think that maybe will be where it's more useful and maybe like just a simple screenshot of a couple rows and you put them in ChatGPT and say again, you know, find me the mean or the average or whatever it is. Build me a regression model. I think that will be cool. Again, I'm not sure that that will end up being the killer use case for data science folks, but I think that's where we'll be able to start, which I think is exciting.

Yeah, actually that first case is something I have stumbled over where it's like someone's shown me some paper journal from 1970 or something and be like, okay, can you just reproduce this plot here? And you're like, okay, how do I do that? Having that done automatically would be really fantastic. Although that's kind of a niche academic use. The idea of being able to show a plot and then have some text explanation of it does seem really useful. Is that something that's possible?

Yeah, most definitely. I think all of those types of explanations will be super powerful, super powerful and actually very, very feasible to make happen. So I think you could pretty much show anything and ask for, like, how is, how is this happening? Or, like, what is making this happen? And I think the really interesting thing in some of those examples with multimodal GPT4 is it actually, it understands the cause and effect relationship. So, like, there's a bunch of pictures of like, you know, something being held by a bunch of balloons. And you can ask through GPT4, like, what happens when I, if I, like, you know, cut or remove the connection between the object and the balloons and GPT4 will say, well, you know, the object will fall down and the balloons will go up. And it has that, that understanding of these images and the text that's associated with them. And I, again, I, I think once we start to live with this technology, it'll be much more clear, like, how valuable and beneficial it actually is to have that level of reasoning and understanding available to you at all times. But even for me right now, like, I just haven't had a ton of time to play around with it and like, see how it's going to be useful in my life. But yeah, I'll be interested to see.

Absolutely. Actually, I have a really stupid question about this, but it's been bugging me. So there's a sort of running joke about how with all these sort of generative AI, like image things where, like, you've got dall e and the stable diffusion, all these where if you try and get them to draw hands, like, they can't count the number of fingers correctly. I'm wondering, does it work in reverse? Like, can GPT understand hands properly as an input?

Yeah, that's a good question.

This is a very specific question.

I think it'll get better at doing this. I think a lot of those. I don't. I need to read a research paper or two and try to understand better, like, what's the actual reason why the model has a hard time with that? Because, like, it's something for humans that is, like, very intuitive and very simple. My guess is that this, like, multimodal understanding will help solve problems. Help solve part of the problem, at least.

All right, so the idea that you have like text and images together that somehow improves the model performance, is that correct?

Yeah, I think it's the understanding of the visual piece. Like right now you can generate an image and do it like, pretty well, but it still has these problems with hands, for example, and faces is another thing. I think the other piece of it is that the, the real. I guess this is actually the real challenge is there's not a lot of. Dall? E does a really good job, for example, of, like, generating abstract art because there's no, like, there's no constraints on what it is. Like, there's no, you know, in. In some image of, like, the future of, like, buildings and stuff like that. Like, there doesn't need to be doors. Like, it's essentially something that's completely made up. It's. It's totally abstract. And the problem that we have is when you take something that is, like, we would expect as humans to be, like, there is one right answer for this. Like, what do hands look like, you know, with a bunch of, you know, fingers? Like, we have a preconceived notion of what that should look like because it's something that we're very familiar with. And the model then, like, has a hard time of understanding. It's very. Like, there's all these weird nuanced things that these language models and image models in the case of Dall? E have a hard time with. Like, something that's really simple for humans to do and, like, not difficult, like, even just reversing a string. If you take a string and you ask GPT3 or GPT4 to reverse it, it actually won't do a great job of this. And you might sit there and question. Like, it's actually, like, a very deterministic, easy thing to do. There's no uncertainty in this. Like, there is one correct answer forward and backwards. And the model just has a hard time parsing things in that way because of the technical limitations. Because it's designed to. Because it's designed to do things at the token level. And then when you reverse it, like, the token indexes, like, change. So it has to do all this, like, crazy mapping behind the scenes to try to make this work. And, like, it just doesn't end up working very well. And I think there's something like that happening with the. With the fingers and the images. That's at least my understanding of it.

Okay, that's a really interesting explanation. And it reminds me a bit of the idea of trying to do calculations using GPT, where it. Somehow it's got a lot better over the last few months. But it's not a perfect calculator.

Yeah, it can actually do calculations, which is, like, informed guessing, which is the fascinating thing. Like, it doesn't actually have the. The calculation capability built into it to make this stuff work. It's literally just guessing, which is interesting that it's able to get right answers in some cases.

All Right. So I'd like to talk a bit about fine tuning models and this is. Well, actually, do you want to explain to everyone what fine tuning a model is and why you might want to do it?

Yeah. So essentially today we have a bunch of models that are available to be fine tuned. The original GPT3 models through our API are able to be fine tuned. And the basic intuition is you take a model, it has a bunch of weights, it's been trained for the entire thing. What you can end up doing is making small customizations to the final output layers of the model in order to influence it to more correctly understand whatever data is relevant for you. So a lot of people, I think one of the biggest use cases that people try with fine tuning is, hey, I have some, you know, corpus of data for my company or my project, whatever it is, I want, you know, this language model to be an expert at this information and really like have, have like ready, ready access to this information. The problem ends up being is most people are not using, at least, you know, people who we work with who are trying to fine tune things are not using fine tuning in the, in the correct case. Like it's actually only, at least in the current form, is only really useful for classification tasks like giving it a bunch of examples of like the different classes that you're interested in and having the model learn that is actually quite successful. But making the model have like very specifically follow like certain formats with our current fine tuning infrastructure doesn't work well. It doesn't like, it can't like retrieve information really well most of the time. What people are actually looking for when they're fine tuning with our models at least is they're looking for embeddings. They're looking for like direct retrieval of some source of truth. They're like a customer support company, for example, or they are trying to use this model for customer support and they wanted to be able to reference like different solution guides and like, you know, here's the answer to X, Y and Z question. And embeddings are just a much better solution to that problem than trying to fine tune the model in general. Outside of the context of OpenAI, fine tuning is a viable solution for some of these problems. But it's just like our flavor of fine tuning that makes this a little bit different than if you were just fine tuning a model in Pytorch, for example.

Okay, interesting. So fine tuning is purely for classification at the moment, but anything else, look for embeddings. Okay, all right, can you give Some examples of maybe you mentioned a lot of companies are doing this wrong. So let's start with the lessons learned then. What are some other examples of where you might go wrong with trying to use this fine tuning?

Yeah, I think that's probably the biggest thing with fine tuning specifically. I do think that there's, most of the companies don't end up going totally wrong because they get in contact with us and our team is able to help them, guide them through that process. The other piece of this as well is that the models that are available to fine tune today are only, are only the original GPT3 base models. So like, all of the innovation that's happened, all the progress that's been made in the last couple of years, those models aren't available today to fine tune. So it just makes it like a much less powerful solution. Like if you could fine tune GPT4, for example, maybe it'd be better at solving some of these problems than some of those original base models. So again, it's just a little bit less useful than a lot of other approaches that you could take today.

Okay. I mean, it certainly just seemed quite appealing to be able to just throw in a load of your own company data to GPT4 and then have it give better results somehow. Okay.

But I think with GPT4 specifically embeddings, plus the ability to have a system message that like, is truly deeply steering the conversation. A great example of this is Khan Academy. So Khan Academy was one of the launch partners for GPT4. If people aren't familiar with Khan Academy, it's a software platform that allows free education for the world and they've done a great job of making that technology specifically suited for their use case to the point where one of the biggest challenges in the educational space is you don't actually want, if you're trying to help someone learn something, you don't want the model to give an answer. And like by default the model really wants to give an answer. Like that's what it's there to do, it's there to answer your question. So if you ask it, you know, what's, you know, how many stars are in the galaxy or whatever, like it's going to try to answer that question. And, and what Khan Academy was able to do just through system messages and some other stuff is actually make it so that the model was steered away from answering people's questions and acting more as a tutor to help guide them towards the answer instead of just here's the answer. And that capability is much more powerful in GPT4 than it is for some of our previous models to actually follow those system messages and, like, set the tone of what you want this AI system to do and customize it for your use case. Like, you could make it say, like, you know, you can put in the system message, you are a customer support chatbot who works for X, Y and Z company. You know, you shouldn't respond in, like, a way that's aggressive towards users, even if they're mad at you. You should always be polite. You should apologize if you say something wrong. All those kinds of things you can put in the system message and the model will actually understand and stick to those as it processes stuff, which is super interesting to see.

That is actually absolutely amazing. The use case you described, because something datacamp is building at the moment is trying to be like, okay, well, we use it. We'll use GPT to give students some help, but not give them the answer. And it's just getting that balance right of being supportive but not actually giving things away. Very delicate.

Yeah. It's a hard line to walk, especially when, like, you can imagine, for younger learners who are maybe, like, less intrinsically motivated to be. To be going through this learning process, they're really going to be like, continually prompting, like, just give me the answer, like, I don't want to do the work. Like, just. Which I totally understand. And I think, yeah, the folks at Khan Academy had to do a lot of work to sort of make this work for them. And I think, at least from the demos that I've seen that salcom put on YouTube, it seemed like it was extremely successful for them, which I'm really happy about.

Okay, well, I'm hoping it's going to be successful for datacamp as well. It's looking promising anyway.

Fingers crossed.

Magnificent. All right, so I'd also like to talk about Copilot. So while we're on the subject of, like, using GPT for learning things. So Copilot is a great solution for generating code for you, so it's perhaps more developer focused. But can you tell me what are the data science use cases here?

Yeah, I think a lot of, like, if you're doing work in a jupyter notebook, for example, and you're running your data science experiments, you could imagine that, like, in the future you're able to. And maybe not even in the future, probably today, with Copilot X, which is what's on the horizon of being available to developers through GitHub, you could describe the experiments that you want to Run, I imagine remember back to when I was at Apple and I was a machine learning engineer, we were trying all of these different model configurations, we were trying all these different, you know, starter pre trained models to then do transfer learning on for computer vision. And I was like by hand, like you know, doing all these, setting up all these different manual experiments and you could imagine that you just say, you know, here are the list of the experiments that I want to try in natural language. Write me the Python code, write me the Pytorch code to actually run all these different experiments. And with GPT4 powering all of these features, new features like GitHub, you're actually able to make that happen, which I think is so exciting and will hopefully put data science folks back in the position of being really close to solving problems and thinking critically about how to solve the problems versus just, you know, typing keys to like set up experiments. So I think for myself, like that would be, yeah, thinking back to, you know, doing that work of actually training models and doing data science workflows, like it would be so convenient to have this tool. And also like another layer of this is being able to explain what's actually happening to me as the developer, as the data scientist who's running these experiments. Like I didn't like my background was like pure computer science. Like I didn't have like a lot of the like statistical background and for some of the models that we were doing at Apple it was more like statistical models that like I just didn't have a good grasp of like the stats background that was required to make some of that work. And I could just imagine how helpful it would be to like, you know, while the code is running and being generated, you know, here's GPT4 and ChatGPT just like giving me a explanation in simple terms of like how this statistical method or property actually works. That would be so helpful. And now you're like an actual more informed at your job and you could be more at it, which I think is. It sounds incredible.

Absolutely. And there's always been the problem of data science is that you need both the statistics skills and the coding skills, not to mention all the other stuff like communication. And if you can get support in all of those areas, then that's going to make life easier I think for data scientists. All right, so there was a recent announcement related to Copilot saying the underlying technology has changed and it's all going to be based on GPT4 now. So how is that going to change its capabilities?

Yeah, I think it'll become, I think what the folks, you know, and again, I don't have any inside knowledge on this. I think my guess is what the folks at GitHub realized was that with how powerful some of the models have become to just do like these types of like learning tasks, you can actually get like a great balance of the ability to just generate code, which I think was originally like Copilot in its current form or like previous form to this new announcement was just able to like generate like a, a single completion line of code or like a little section of code. It didn't have any, like you couldn't like talk to it, you couldn't like iterate with it. It did have the ability to like show you multiple generated options and you could like choose one if you wanted to, but like it, it didn't have that conversational layer. And I think what GPT4 is going to allow it to do is still be extremely excellent at generating code for problems, but also have like a deeper reasoning understanding of what's happening in that code that copilot wasn't able to do before. You couldn't copy and paste a, you know, file of Python and put it into Copilot and say like, hey, explain to me what's happening here. And that's what's going to be possible today with GPT4 integrating into copilot. So it just makes the technology much more usable, in my opinion.

Yeah, fantastic. And I guess related to this you talked about deeper reasoning about like what's happening. So one of the sort of the big objections to using things like GPT4 is occasionally it's going to make mistakes. I mean, the calculator example is maybe one of those. So I guess when is it going to be a good idea to have this sort of automated like generation of code and things like that, and automated explanations compared to just saying, okay, well we have to do this manually.

Yeah, this is a really good question. I think it's a deep question. Maybe we, we won't be able to cover all the, all the nuance in our, in our conversation. I think my gut feel is that a lot of like the first pass of solving some of these problems can be done by models like GPT4, but it still requires somebody who has the deep expertise and understanding of, you know, I did a live stream earlier today and we were trying to essentially build like the, a UI clone of ChatGPT and we were asking ChatGPT to actually generate that for us based on a bunch of context that we gave it and it didn't go perfectly, probably because I didn't do enough prep for the, for actually going and solving that problem live. And the sort of takeaway for me for this whole thing was, yeah, you know, ChatGPT was able to generate a bunch of code that looks really great, but when, you know, just one little thing doesn't hook up correctly, all of a sudden the whole thing blows up and then you're, you're sort of back to that mode. As a programmer of okay, now I have to solve this problem. It was great that all this stuff was given to me and maybe it's even, you know, a little, I'm a little worse off than I had wrote this myself because now I don't have the sort of intellectual background context of like what this code is actually doing. Why did I do it this way? So in some sense it can actually make that problem a little bit more abstract and harder to solve. But still, I think, yeah, so it requires the use of somebody who has the programming background. So this goes back to the broader conversation around will these new productivity gains mean that there'll be less software engineers or less data scientists in the world? Unlikely, because you still need to have somebody who's very capable with these tools in their hands. But I, I do think that the likelihood that you will start off with some AI generated template or first pass at whatever the problem you're trying to solve, I think that goes up significantly in the next six months. Like I wouldn't like even for me today when I'm doing software engineering development or data science workflows, like I'm starting with whatever GPT4 generates for me just because I don't want to have to like do the, you know, extra few hours of work that it would probably take for me to like set all that up and get it running myself.

That's interesting. It sounds like maybe the skill set for developers is going to change a bit. So rather than being able to write code, it's more about can you review and understand and tweak code? Is that, does that sound correct? And if so, I guess how do you think that's going to change? Like how we train people to do data science and great software.

I would posit that it's probably still going to be very similar for developers. And I think the context, or at least my thinking on this right now is that we're essentially all doing this today. I don't have the knowledge base of how Pytorch works to train a machine learning. I intuitively understand how Pytorch works. I intuitively understand how to train a computer vision model. But if you put an IDE in front of me with no resources, I'm not going to be able to just like type out the code to train a machine learning model right now. Like I don't have that in my memory to just do off offhand. So what I will do is as a developer, go to the Pytorch website, look at the templates that they have, look at the examples, probably start with one of the examples that they give and then modify that for my specific use case. So it's a very similar process to like what we do today. We just go off to all these different websites and we have to like hunt around for the right resource and, or try to make some resource like fit into our use case. Like I remember doing this a lot with the Pytorch documentation. We're like, yeah, I'm kind of solving this problem, but I'm kind of also solving this like image segmentation problem. And you know, they don't have the tutorial that like falls in between the two. So I'm like trying to pull things from both and like, you know, it's kind of a hairy situation. And you can imagine that process becomes actually a little bit easier to do when you have GPT4 do the first pass on these things for you instead of having to go and look through 12 Stack Overflow posts to find the thing that you're looking for.

That's a very funny story looking through 12 different stack overflow posts. So I like the idea that it's creating templates for you, but it's maybe just a little bit more personalized to your problem. So I guess just going back to your previous example about like trying to build a. It was like the OpenAI website using GPT. So and you said you were, you were struggling with it slightly like getting the right prompts and things. And that does seem to be the other objection I've heard from a lot of people is that they'll go, they'll try and they'll write a prompt and then they'll get a bad response and they're oh, well, this is a stupid model, I'm going to use it again. So it just seemed like you have to provide a lot more detail to the prompts than at least I first thought. So do you have any tips on, well, how do you go about writing good prompts to get the response you want?

Yeah, I think the mental model for people that puts Them in the position where they're not able to get the most of what they want out of this technology is the mental model that you're talking to another human who has expertise in this topic. And the reality is, you know, you and I have some shared understanding of a bunch of stuff that like, we haven't agreed on but is probably there. And that, you know, if I told you to do something, just like the model might actually, you might, you know, infer a bunch of things that I didn't actually explicitly say. And this is the exact same thing that happens with the model where, you know, I'm going to ask it to do something. It has its own sort of knowledge base that it's been trained on just like any human would, and it's going to go and do things in the way that it thinks makes the most sense given the experience that it has, given the knowledge that it has. So not unlike humans. And I actually think this goes back to great managers or people who are able to really effectively delegate things to people do this part well, where they're able to explicitly say, here are the things that I'm looking for in excruciating detail. This is what I want to see happen. And I think that we don't have to do a lot of that all the time just because we're usually like, okay with the variants in some sense. But like, if you have like a, yeah, you, you might actually do this in your, in your daily job. Like if you're a, some software engineer and there's like a list of like requirements that you're, you've gathered from some clients, those things are often like very explicit, like they want X, Y and Z&GPT 4. ChatGPT can do a great job of understanding those things because it's very intentional. I think when you leave, when you give the model ro room to interpret, it's unlikely that it's going to interpret them or like infer things in the same way that you would. So you have to be explicit. And again, with GPT4, it's much better at like actually following those specific things that you laid out that you were looking for. If you try, like GPT 3.5, for example, it might do some of them, but like, it's not going to like listen to all of your instructions, which is what makes GPT4 so powerful.

That's interesting. So the difference is just about its ability to follow instructions, like the difference between the two levels of model. Okay, are there any particular specific bits of advice for data scientists in terms of, well, okay, if you're going to tell it to do a data science task, what are the kind of. I mean, you said be specific, but are there any particular sort of things that you need to know about when you're writing prompts?

Yeah, that's a good question. I think a lot of it can. One of the things that the model is limited by is like how much context you can give it. So for example, in this demo that I was doing earlier where I was trying to make my own version of the ChatGPT UI that's on the website today, there's a bunch of context that the model doesn't have because its training cutoff was sometime in 2021. So like all new knowledge that's been created since the end of 2021, the model has no idea about it doesn't, you know, whatever breakthroughs in science or technology or new versions of some Python library that came out in 2022, the model isn't going to know about. And what you could do is actually give it example. So it's like a few shots, few shot learning where you can like take an example of, you know, here's how I want this solution, this problem to be solved. Here's like the response that I want to get from you. And the model can do an incredibly good job at actually solving it and giving you back the data in the way that you've formatted it if you provide that single shot or a couple of shot example. And I think that's probably the best, you know, for a. Today it doesn't actually, Today it's not possible with the multimodal input. You can imagine the future where you have like, you know, as a data scientist, you're like, here's the dashboard that I want. Here's the picture of like how I want the different things set up. Here's an example of like my data input sources and I want to use this version of this library and, you know, maybe it's been updated since. So like, here's the code that you, you know, here's a few lines of the code that it might look like. All of that context can be bundled together to create some actual data science dashboard that it just generates for you automatically based on those inputs. Again, I don't know if this is specific enough, but not making assumptions about what the model knows and very clearly giving these examples I think makes it so much more effective at actually solving whatever the problem is that you have.

Right? So it's not just about, well, write this single Prompt is about like, give it a bit of training data. Say, okay, I want, I want, I want you to do something like this and then ask it the question. That seems like really cool advice. All right, so I'd love to talk about the different interfaces to, particularly to Python SQL R, Julia, because I think if you're, if you're a data scientist or a data analyst, that's what you're going to be using. And I know you're one of the developers of the OpenAI Python package package. So can you tell me a bit about what's happening with development of that?

Yeah, just nothing, I would say groundbreaking. Just an interface to our, to our API through Python. There's a couple of like extra nice helper functionality that comes in the Python package as well. Super widely used. The, I'm not one of the core folks. I help out a little bit and help sort of triage a lot of the craziness that happens on our open source repositories, but a bunch of other folks doing like the, the difficult engineering work to make that package work. We also have, I think that Library and our Node JS1 are the only ones that are like officially hosted by OpenAI. There's also this huge swath of community libraries across all those languages. They said Julia are probably some of the other more niche data science languages as well. If you're MATLAB, maybe there's an integration. I'm not 100% sure, but maybe there is. So yeah, essentially language agnostic, whatever language you're using today, you could use with OpenAI's API, which is really nice. And it's like, at the end of the day, it's like a simple curl request. So any language that you could make some arbitrary call to a server endpoint, you could do that with whatever language you're excited about.

Is the plan to make like the R and Julia packages official at all or is these going to be purely community efforts?

Yeah, it's a difficult trade off for us. I think, you know, in a perfect world with unlimited resources, you know, we'd probably want to do some of the more popular ones ourselves. I would guess Julia is maybe not one of those, but definitely, you know, maybe R as well would be one of those that you might want to consider. Again, the reality is we don't have the resources to do that. So relying on the community is a nice sort of happy medium. But it does mean that when we release stuff, all the community libraries, they don't necessarily get a heads up. They have to see it when everyone else does, and therefore things are a little bit out of date, you have to rely on your maintainer for your community library to actually update it when we release something. So it does introduce that burden for users.

But, but.

It'S untenable for us to do more in that space than we're doing today. It's already a tremendous amount of work just for us to do our Python library. So yeah, it's a challenge.

So it sounds like if you need to be up to date at all times, then Python is the interface that you need.

Yeah, Python is definitely our most supported library at the moment, so I would highly suggest folks to make use of it if, if possible. We do have like an open API spec available. So if you like, there's 1001 different tools that like auto generate SDKs based on open API specs. So that's probably the other way to like not building a library by hand using the open API spec auto generate it. And that's, that's probably the most effective way.

All right, so I think a lot of organizations are trying to just figure out, well, how do I get started with this? And I guess they're not sure where to start. So if you want to include Open Air stuff within your products, what's step one?

I think step one is taking a step back. And like, actually this is probably the same for most things, like understanding what is the actual problem that you're trying to solve. Do large language models? Does something like chatgpt or embeddings or image capabilities whisper for audio? Is that actually the solution to the problem? And I think there's, I think this question is more important now because of how much excitement there is around this technology. And I think there's a lot of people who are perhaps integrating this technology for the sake of integrating it without taking a step back and saying like, you know, does, does it actually make sense for us to just like plop this in as a feature in our product? Or Maybe we, and OpenAI is seeing a lot of, and I'm seeing this as well a lot of companies that are actually rewriting the whole roadmap and saying, okay, instead of adding this language model based feature into our product, let's actually rethink our product entirely with this understanding of the future of language models being important. And part of that is reshaping our platform to use that technology from the ground up. So that's the conversation that I would push folks towards initially. And obviously it's a much bigger thing to chew off to say, okay, yeah, we're going to completely redo our entire product or platform using this technology. But I think that's the. Given the moment that we're in, like, that's actually the thing that's going to differentiate. Like, people want that different experience. They're not looking for, like, you know, your product plus a chatbot on top of it. They're looking for, like, a completely different interface to access the information that or the service that your product is your company is offering. Yeah. So I think starting with that is the right place. And then the actual tech, the cool thing to me is the technology is actually really easy to integrate. It's literally just making an API call. The hard part is, like, figuring out the problem, doing all the prompting correctly, figuring out a data flywheel of improving the model's performance, or improving the prompts that you have based on the data that you're collecting. Those are the more nuanced pieces. But, like, actually, yeah, it's a simple few lines of code to integrate the technology into your application.

Okay, so it does sound like really the hard part is just making sure that you are adding value for your customers and not just building stuff because it's kind of cool.

Yeah, I mean, I think that there's a lot of companies that are likely building it because they think it's cool and haven't thought deeply about, like, how does this actually help people? And is this the right experience that I want to provide to people? And I think the added layer of complexity for, like, smaller companies and individual developers is there's actually a cost associated with, like, every API call. So every API call to the OpenAI API costs money. It's like, relatively cheap to make these API calls, so you can do a lot for a little, but it still costs money. So, like, thinking about this is another, like, forcing mechanism to think about, like, how do I actually build a scalable business? Like, how do I provide enough value to somebody that they're interested in, like, giving me money back for the service that I'm providing. And I think, you know, you can sort of circumvent a lot of that with like, you know, a $5 a month, you know, Heroku server somewhere. And, like, do a lot of things with just that without having to, like, think about how you actually build a business around this. And because this technology costs a little bit more money and is powerful, having that sort of front of mind earlier on in the conversation I think is also important for people and something that I think folks are missing who are at, like, a much Smaller, like obviously big companies are thinking about how to make money, but smaller individual developers perhaps aren't thinking about that. And I think it's important so that you actually are running a solvent product or service, whatever it is, and you don't need to make gobs of money, but you need to make enough money to pay your bills.

Absolutely. And I have to say, it's like it does seem pretty cheap to run the API, but I was trying like generating data sets. You generate a five row data set, it's like super cheap, less than a cent. Try and generate like a million row data set that's going to be a little bit more expensive because you're sort of paying for the output. So, yeah, I understand how thinking about your API costs is going to be a good idea. All right, so I'd like to think a little bit about what's happening in the future. I don't know whether you can tell me sort of what's coming soon from OpenAI or what's exciting in the pipeline.

I think this fusion of multimodal models, which is a tongue twister, I'll say it three times quick, if you're listening to this multimodal models, I think that will end up being more impactful than people realize. I think that will again continue. Today we have audio models. I have no idea whether we'll have the fusion of more of these different modalities together, but my brain says potentially, yes, it makes sense. I'm not sure if that's technologically possible or not. So the fusion of more of these things together, I would hope that there's a bunch of advancements in the size of these models so that we could run things locally on your computer computer or on your phone. And we've seen this with some of the open source models that have come out where you're able to like actually run them on a, you know, few thousand dollars computer versus having to have like this massive GPU infrastructure and a server somewhere to actually run them. So I think that will be super important because, yeah, there's just a ton of use cases where it makes sense to like keep the data that you're processing locally on the device in which it's coming from and not have to send it to a server. So that will be really cool to see. And again, I think the ChatGPT as a product will just continue to evolve and get better over time. Like it's something that the team and all those folks care deeply about and I would expect it to continue to get more valuable more useful, more integrated with now all these plugins that are coming in the future. So it's going to be super exciting. I think that's the general takeaway, is that the future is bright. I'm optimistic that this technology is going to help so many people become better at doing the things that they really enjoy and sort of automate away some of the work that they're not interested in doing. And hopefully we can get the balance of all these things right where it's a really positive outcome.

All right, fantastic. Do you have any final advice for anyone who's wanting to develop products around generative AI?

I would get started now. I've seen a bunch of quotes that like, AI isn't going to take your job, but somebody who is using AI is the person who's going to disrupt you. And I think for folks who are sort of writing off the technology or aren't becoming in tune with its capabilities, even if you're not using it on a daily basis, I think that's super important to understand how you can be more effective in what you do using these tools so that you don't end up getting disrupted by the potential change of this technology that's just accelerating so quickly. So definitely check it out, play around with these things and get an understanding of what the problems they actually solve are.

All right, brilliant. Thank you so much for your time, Logan. That was really informative. Amazing.

Yeah, this was super fun, Richie, thank you for having me and I'm super excited. I'm hopeful that Data Camp's tutoring efforts go well. And again, if there's anything that I can do to help, feel free to let me know.

All right, thanks.