ChatGPT goes prime time!

It's a three step process. So you pre train a language model, then you gather this sort of human preference data and train a reward model. Now the second reward model is trained to take in a prompt and a response and score it like a human would score it according to preference. It's actually trained on the human preference data and it outputs a prediction of what a human preference might be on this output. The third and final step is that you fine tune a copy of your original language model using this trained reward model and a reinforcement learning loop.

Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive and accessible to everyone. Subscribe now if you haven't already. Head to PracticalAI FM for all the ways. Special thanks to our partners at Fastly for delivering our shows super fast to wherever you listen. Check them out@fastly.com and to our friends at Fly IO. We deploy our app servers close to our users and you can too learn more@fly IO.

Welcome to welcome to another fully connected episode of the Practical AI Podcast. These episodes are where Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and then dig into some learning resources to help you level up your machine learning game. I'm Daniel Whitenack, I'm a data scientist with SIL International and I'm joined as always by my co host Chris Benson, who is a tech strategist at Lockheed Martin. How you doing Chris?

Doing very well. Happy New Year 2023. This is our first conversation.

Yeah, Happy New Year. This is the first one we're recording in the year 2023 looking already to be an exciting year for AI things. I hope you got a bit of a refreshing break over winter because there's a lot of I'm guessing it's going to be a whirlwind of AI stuff this year.

I think it is going to be a whirlwind. I didn't get a rest over the break because having nothing to do with AI, our animal nonprofit, we had all the winter weather that most people in the US were aware of and we were doing animal emergencies. So we saved a whole bunch of lives which made the lack of rest worthwhile.

But there was a lack of rest.

There was a lack of rest, but we did a lot of good. But interestingly, the conversation we're going to have today will play into that very non AI side of my life because we're starting to see see some crossovers. We'll see in a few Minutes here.

Yeah, it's interesting, you know. So today, spoiler alert, we're going to be talking about ChatGPT. You've probably been expecting us to talk about ChatGPT for some time. One of the things we wanted to do is really dig into the internals of ChatGPT, how it works and its implications. And so we wanted to do it justice, which is partially why we wanted to take some time and prep for that. But it is interesting also to get a little bit of perspective now that ChatGPT has been out for not that long, but a little while over Christmas, you know, I was at Christmas with my family and even at our family Christmas dinner, my dad was asking me about ChatGPT and, you know, I, at my church, I've had people come up to me and ask about ChatGPT who aren't, don't work in tech or anything like that. And my barber, you know, whoever is in my life, it seems like they're at least aware of ChatGPT. They might not know exactly what it is, but they know that it's a big deal. Are you having a similar experience?

Very, very similar to that. And for folks, Daniel and I haven't talked through the holidays, so, you know, this is the first time I'm hearing it just as you are, and I'm having the same experience. And it's been really notable that we've, you know, each new large language model comes out and you know, the various GPT series and we talk about it, this is the one that's crossed over into mainstream awareness and broad use. And I mentioned, as we were getting into the conversation, that it's now crossing over from the technical and AI side of my life into the non technical and animal side. As we do things like narratives, both written in video and educational material, this is an amazing tool that completely non AI focused people can use productively to really do good in the world and get things done that they want. So it's been really interesting to see how this one has been different from the GPTs before.

Yeah. So in your case, it's something that, like as you're creating content, you see it as potentially playing a role in whatever scripts or articles or whatever that might be. Is that right?

Absolutely. It's been quite humbling in that way, experimenting what was possible because the, the quality of the outputs are typically much better than I can do by myself. I've done that both in terms of I'm writing a children's story to teach children about animals and I've been Experimenting with it. And every time I write something and then I seed it into ChatGPT, it does a better job than me. So it's been very humbling in that way. I think of myself as a decent writer as well. And then the quality of video output is. Has been quite good and there's a little workflow, but it means that we can do more good in the world faster. It accelerates the ability to put out great content. And so I think that this is one of those inflection points that we've seen, not just on a technical merit, but in the world at large.

Well, your usage seems to be much more useful and valuable than my usage, which has mostly been things like writing. I remember I had ChatGPT write a new Christmas Carol for me about the three wise men in the style of a rap song by Eminem. I have to say, it was. It was a great rap song. I didn't record it because I'm not Eminem, but I sent it to his people and we're having discussions.

Okay, Yeah. I can't believe you're not sharing that with us.

Well, maybe before we jump in, I think some of what we wanted to do today was just like, describe a bit of, like, what ChatGPT is, what the interface looks like, what you can do, but then really do a deep dive on what are the guts of the system. Why is it different than what's come before? In what ways is it similar to things that have come before? Both of those things are true. And so we want to do a deep dive and then think about some of the implications. So buckle up. Hopefully this will be fun. First off, it is called ChatGPT, which is interesting. So the interface that they've chosen for this and the sort of design of the system is a chat interface. So if you go to chatgpt.OpenAI.com, you need to create an account, and we can talk about some of the implications around that in a second. When you log in, it gives you some examples of what you can do, some example capabilities, and some limitations. I found this interesting, and we can talk about it later. Some of how they describe the limitations and they released the model. But the basic idea is there's a chat interface. You can type a prompt and it will respond, and then you can actually continue to have dialogue with the system. So you can say, you know, tell me more about that, or I don't understand this part. You know, explain that bit more. So some of the examples that they give are, you know, explain quantum Computing in simple terms, prompt or how do I make an HTTP request in JavaScript. So there's even a, you know, it can output code, it can help you debug code. Like I mentioned, it can provide lyrics or scripts or structured types of things like the Eminem song. So yeah, that's the basic input output. How did you find this sort of interface, Chris, in terms of your own usage as related to like building scripts and other things?

It's been interesting in that it will take it a direction. Like as I've been trying out, the children's story thing is something I've been playing with and where, where ChatGPT chooses to take the beginning of a seat of a narrative. Like I would start off with, you know, once upon a time there was a precocious raccoon named Pandora because that's the hero in the story. And it's been interesting to see how it's taken it, but it's also, it will go off in directions I don't want. So then I'll ask questions to kind of steer it a little bit and it will come back. So it doesn't. It's not final output, but it's producing a body of narrative that's better than I could have done by far. And so I find myself, instead of being the creator of the story, I'm kind of editing it to make it work. But it's a collaboration in a sense between. This is one of those first points where we've talked in an aspiring way about collaboration with AI for a long time. But I now am doing that and steering it in different ways. With the entering in the chat and seeing where it went and asking specific questions about the story, it's been quite remarkable. For the first time, it's like having a partner in the process. It just happens to be that the partner is not human. In this case. A friend of mine, Brent Siegel, has been also playing with it a lot, doing some stuff on him and that's how he described it as. Well, he was looking at some different topics and he said, you know, it's like having a dozen world class scientists for the things that he was focusing on right there available to you. And they're not wrong, you know, they never get it wrong. You know, he had some pretty cool stuff that he was working on in a very different thing. But it's that sense of collaboration with the technology in a real life sense that's really different now from the way it was before this.

Well, as you were chatting, which though some great Context I asked ChatGPT the following. What state of the art AI topics does the Practical AI Podcast need to cover in 2023 and who should they reach out to for interviews? And the response so you're ready for what we'll cover in 2023?

Let's hear it.

ChatGPT said it's difficult to predict exactly what the state of the art in AI will be in 2023. Definitely an interesting start.

Yeah.

As the field is evolving rapidly. However, some topics that might be worth covering include machine learning, interpretability and explainability. And then it actually gives a blurb about what that is, AI safety and ethics. So that's right in your wheelhouse and it gives an explanation of that. Natural language processing. So we can continue to talk about my favorite topic of NLP in 2023, and computer vision was their other one. And they said to find guests to interview on these topics, you might consider reaching out to researchers and practitioners working in these areas.

Very sensible.

Some suggestions include Rachel Thomas, co founder of FAST AI Timnit Gebru, co lead of the Ethical Artificial Intelligence team at Google, which is interesting that it gave that response because that is not factually correct anymore, as she is not with Google. And actually that was in the news quite a bit.

It was, that was. That was a significant story in the AI world a few months ago.

And then it gives a few others, including Yann Lecun, who, you know, of course, we would love to have him on the show. We'd love to have Rachel and Timnet as well on the show. But yeah, interesting. So a few things, I guess, that strike me as an example with this certain case is the output is definitely natural and coherent. Right. So that is thing one, that's striking. Thing two for me is there's actually a good bit of structuring that goes on here. So they actually give 1, 2, 3, 4 the topics that we need to cover and then a bulleted list of the people that we need to have on the show.

Yeah.

Thing three is, despite it being coherent and natural, it is not fully correct factually. So that's maybe another element of this.

You know, it's funny because we've seen a fair amount of criticism about, you know, chatgpt getting things wrong and stuff. I find it curious that as we talk to humans about human things, we get things wrong constantly and fact checking and, you know, was that misinformation or was it just unintentional? And. And yet we hold these technologies to such a perfect standard that we are ourselves completely unable to hold up. You know, I wouldn't want to ask one question and assume that it was 100% right. But it makes it a little bit more interesting to me that collaboration, I dare say, takes on a human element by having error in it.

Yeah. And we'll talk a little bit later about the interaction between this and humans and where the burden lies. I do think that the interface that they've provided and being explicit about limitations, that's a good thing. Now, certain people might kind of go back and forth on this model is not open access. Right. Like you can sign up and create an account and a lot of people have done that and you can interact with it, but like the model waits itself and you know, it's not released publicly in that sense, even if a lot of people can use it for free at the moment. There's pros and cons there. But I think it's interesting that this model, as opposed to GPT3 earlier, it was, I think, easier for the general population to interact with this model right away in comparison with GPT3, which had a very prolonged kind of wait list and timing and all of that and lots of explanation. So it seems like that they've kind of shifted the scales a little bit in terms of making access to, to run the model more open while still maintaining it as a closed model and providing limitations. So it's interesting to see also that kind of shift in dynamics, which I think probably was influenced by the fact that, you know, actual open access models like stable diffusion and others have taken off so widely so quickly because they are more open access wise. And so I felt like we saw OpenAI shift a little bit in how they released this while still kind of maintaining some of the elements of how they released GPT3 and others.

I agree with that. Yeah. I mean, we've seen that kind of evolution as they've explored release approaches over time, you know, with iterations and such. I think one of the things that we've seen, you know, across this is the fact that every time a breakthrough comes on, we're starting to have fairly quick follow up. But once people know that something is possible, they manage to kind of reverse engineer it. So I suspect that aside from strictly chatgpt, that we will see some fast followers pretty soon.

All right, Chris, let's get into the technical details of this, which is, I know I'm excited to chat through. No, I guess, pun intended in that case.

Oh boy.

There's kind of two elements of this that I think are important to talk about before we Talk about what actually was done with ChatGPT specifically. And these two things are more general than ChatGPT. One is sort of the GPT family of language models and those types of language models, and then also a technology or approach called reinforcement learning from human feedback. Those two things kind of combined here to create the ChatGPT system. And these two types of models and approach have been applied more widely in other cases and by other people. But here they were applied by OpenAI. So, starting to talk about this sort of language family model of GPTs, we had GPT and GPT 2 and GPT 3 and GPT 3.5. And I don't know what, to be honest, I don't know what number we're on now. But these GPT language models are just that. They're a language model and they're a specific type of language model called a causal language model. People might be familiar or at least have heard the words causal language model CLM or mass language model mlm. So mass language model kind of takes a sentence, and what it's trained to do is kind of, for one word that's masked in the sentence or taken out or given a special token, it's trained to predict that based on everything else in the sentence. So it sort of looks both ways at the sentence and tries to predict the mask. GPT is not a mass language model, it's a causal language model, which means that it's trained to predict the next word in a sequence of words or in a sequence of tokens, whatever those tokens might be. It does that and it predicts the next word in the sequence, but it does it based on all of the previous words, and it does that sequentially. So as you go through the sentence, the training methodology is what they call autoregressive, means that it predicts the next thing from all of the previous things. And then once it's predicted that next thing, then it predicts the next next thing based on all the previous things, and then the next, next, next thing, and et cetera. And that's the autoregressive part of it.

I suppose we're kind of seeing that in action because when you're using the interface, it doesn't just give you the entire output all at one time. It comes back with two text. You see the text developing much as if you were typing it on the screen yourself. So I guess you're gradually seeing each of those iterations coming back.

Yeah. And I think in the original GPT3 interface or the playground that we Both played with, you kind of see this as well. You kind of give a prompt and then it, then it generates this text out and that allows it also to be very flexible. Right. And produce these structures and also allows it to be flexible between different tasks. Like if you start prompting it with question answers, it sort of learns that pattern and in a sort of few shot way and then starts predicting next question and answers or something like that. Or if you want a script or if you want a narrative or if you want something else, it kind of adapts in that few shot learning sort of way, which is a key element of this GPT or causal language model structure. And GPT is not the only one, there's other ones, but this is the family which GPT sits in.

And you mentioned just as a two second sideline, you mentioned few shot. Do you want to real quick, just for those who may not be familiar.

Yeah. So kind of some jargon, few shot, zero shot is thrown around. A zero shot prediction or usage of a model means that maybe you're using a model on inputs or a type of input that it's never seen before, even though it's seen maybe similar things. So this happens with machine translation models that are multilingual maybe because you might have in your training data like English to French and Arabic to Spanish, but you don't have examples English to Spanish, but you have English and Spanish data in the data set. And so you could still ask that model to try to output an English to Spanish translation. And actually that can kind of work in certain scenarios. Few shot means that you're not quite doing it that way, but you're providing a small number of prompts that kind of guide the language model into the type of thing that you're wanting to do. So in the GPT3 interface or playground, if you remember, you can kind of start with a question answer template and provide some examples and then you can provide the next one and it'll answer it for you. And so you provide that set of templates or prompts. And this kind of gets into this idea of prompt engineering and that sort of thing because these models are so flexible. So that was the original paper from GPT3 was titled Something like, you know, language models or few shot learners or something like that. That was one of the big ideas there that kind of gets us to GPT and language models. But ChatGPT, well, I guess it is a model like that. So it is a GPT based model. But the reason why the system is so powerful is because it's a language model that has been trained in a very unique way that has proved to be actually quite valuable. And that's that it is a GPT based model that was trained using Reinforcement learning from human feedback or RLHF. Reinforcement Learning from Human Feedback. We'll link to this in the show notes. But there is a really great article on the Hugging Face blog from Nathan Lambert, Luis Castrocato, Leonardo Von Vera and Alex Havel Ria called Reinforcement learning from Human Feedback. And they talk about ChatGPT and other like models. So we're going to pull a lot of our insights from this article. So thank you to all of you for writing this article because it was really helpful, much more helpful than maybe the OpenAI blog by itself. The major idea here with reinforcement learning from human feedback is trying to answer the question can we use human feedback on generated text as a measure of performance that goes beyond just automated measures of performance? How do we integrate human feedback into the loop of training a model as a performance metric? And in that way we're sort of training a language model, but we're also training it in ways that match human preference for answers. So human preference is a key piece of this. And I think that's why people like ChatGPT is we prefer the things that it outputs, right? I don't know if that was the case for you. With just a raw language model like GPT3, you can get some cool stuff output, but it might not fit your preferences of like how a human would actually respond to something.

Going back to the example I mentioned in the beginning, that was the trick for me was, you know, like using the children's story as an example. I had a specific rough narrative in mind because I'm trying to teach and there are certain points that I'm trying to illustrate. And obviously it doesn't know that the model, but the model, if you work with the model, being able to kind of, you know, continue to point it the right way. That was, that was very interesting. I'm curious going back to what you were talking about a moment ago with, you know, the reinforcement learning with the human feedback, how does that scale, you know, because if we were to compare this for a moment, I know this is very much a kind of a newbie question, but for those of us who are not, you know, deeply into language models, you know, when we were looking at other types of models a couple of years, two, three, four or five years ago, there was always a challenge about getting human feedback to scale with the amount of training data. How Is that tackled in this approach so that you can do reinforcement learning that way, but it scales to, you know, what we're doing at GPT.

There's actually like a whole loop of models involved here and different training sets that are of different scales and different models that are of different scales. So let me talk through a little bit of that and I hopefully that will become more clear because yeah, obviously human feedback is expensive in terms of gathering it. Right. So how much of this do you need? So there's three steps. The process with which ChatGPT was trained and other models using this reinforcement learning from human feedback approach. It's a three step process. So you pre train a language model, which is not new. We've been doing that for quite some time.

Right?

You pre train a language model, then you gather this sort of human preference data and train a reward model. Now the second reward model is trained to take in a prompt and a response and score it like a human would score it according to preference. It's actually trained on the human preference data and it outputs a prediction of what a human preference might be on this output. And then the third and final step is that you fine tune a copy of your original language model using this trained reward model and a reinforcement learning loop.

Is it just kind of the discriminator? You're using the reward model as the.

Discriminator in that it's like in a reinforcement learning loop you would have a kind of policy which outputs like what you should do next sort of thing. And then you have some type of reward system that rewards the agent for acting according to the policy or not. So in this case the reward model is outputting that reward or that preference and the language model is actually acting as the policy here. So you have an original language model that is kind of your original policy and isn't fine tuned yet according to to human feedback. Then you gather some human feedback, like actual human feedback, train a reward model to simulate that human feedback, and then you fine tune a copy of your original language model or a copy of your policy with this reward model. And so the pre trained language model, it could be any language model, it doesn't have to be a GPT3, but in the case of OpenAI it was GPT3. But you have an original language model and that language model could just be a general pre trained language model or you could additionally fine tune that model and maybe for a domain or a specific type of output you want. So that's your pre trained language model. And the step two to get the Reward model. What you do is you start outputting data from your original policy, from your original language model, and you have humans rate it. Maybe you combine that with certain human output or certain other outputs and you have human rating. So that way you're kind of, you're creating a training set for your reward model which includes human labels of their preference. Then also in the step two, you then train a reward model using that data that you've gathered from humans to output the preference. Now, to your point of like, how does this scale? Well, the fine tuning of the policy is done kind of with this automated reinforcement learning loop. But you do need humans to generate enough data to train your reward model that's used in that loop. And what's interesting is, and the Hugging Face blog makes this point, is that different people or different groups that have applied this reinforcement learning from human feedback have used different sized reward models. And obviously as the size of your reward model increases, you need more data to train it. That would be a general rule. In the case of OpenAI, their main language model was like 175 billion parameters and the reward model was much, much smaller, 6 billion parameters. In other cases, people have done similarly sized models. And so I think that is an open question, like, how should these models size wise be related to one another? What types of models should you use for your reward model and how much human feedback do you need? To be honest, I think those are open research questions.

Let me ask you another question on that. With us getting high quality output that is comparable to human output very closely, and where, if you were to get that output, you would find a very difficult time knowing whether it was the model or human that did that? Does that potentially go back in to train further reward models where you're using essentially synthetic data as the output of a previously trained and so you can build on it. And essentially there's a point where you have enough data where you're largely able to take humans back out, recognizing it's the tool of the day. But in the future you can take humans back out of that loop of providing the reward model to do that. Do you anticipate that that would be a reasonable expectation?

I think in this methodology, the reinforcement learning from human feedback, one of the goals in that middle step is to get enough human feedback that you kind of reduce the harm and the helpfulness of the output model. So this is really addressing, I think, some of those kind of problems with large language models of hallucination and harmful effects, general output. And you can address those. What I Think is the finding here is you can address those with humans in the loop rather than humans totally out of the loop. Now here in the next step that we'll describe in the process, humans are taken back out of the loop to fine tune the model. But that central piece. So this three step process of starting with a language model on one end, ending with a reinforcement learning trained model on the other end, has this middle step that I think is a really key piece of it that actually helps the utility of the output and potentially reducing harm of the output, which is that human feedback piece. All right, Chris, we're about to the end of this reinforcement learning from human feedback loop. Just in summary, the loop is we have a pre trained language model, then we gather this human feedback or rating of the output to train a reward model. Now we're actually going to use that reward model. So in the final step of the process, we make a copy of the original language model or the policy. So you have an original policy and you have a copy of the policy or original language model and a copy of the language model. You put in a prompt to each of those models and then you get an output from each of those models. Then you use a sort of constrained reward function where you actually penalize if the updated model is straying too far away from the original model. Because I think what they have found is if you allow it to just take any direction in the output you want, it can have computationally some optimization problems. So you gradually change this language model from the original and you have a penalty for how far that output strays from the original output. And then you score that output with this reward model that you've created. And the way that they're doing the updates for ChatGPT and some of these others, with a reinforcement algorithm called proximal policy optimization, which you have sort of two levels of what in physics I would think of as adiabatic change. Meaning like things don't change too quickly. One is you don't stray from the original policy output too much or you're penalized from that. And secondly, this reinforcement learning algorithm called PPO prevents you from making too big of updates to your model weights in each step. That way you don't have again this kind of hard optimization to do. But in summary, you kind of have these two models, the original one, the updated one, you output a prompt from both of them. Those go into your reward function, which includes a penalty element for straying too far from the same output. It also includes the actual estimated reward or estimated preference from your reward model and then that reward is then used to update the weights of your copy model or your new policy using this PPO reinforcement learning algorithm. So hopefully there's some diagrams in the post that I think are quite helpful. It's a bit hard on a podcast, but hopefully that loop makes some sense in terms of how you're updating this. And this updated policy or this updated language model is the model that is used. So this is like the ChatGPT model that comes out of the end.

I think given the limitations of our medium here, I think that was a very lucid explanation of translating it. So appreciate that.

I've definitely learned some good I know formulas. You can ask ChatGPT to output all of the right formulas and I'm sure it would do a fine job.

Where do you think we're going from here? Like as you have looked at this progression of these models over that we've covered on the show over time with ChatGPT in particular, I've been kind of amazed at what it could do and using it. But I'm really, really curious about where this is going and I think it's capturing a lot of people's imagination in that way that are outside the field. Like what's next?

I think there's still open research questions here that are worth exploring and then there's like workflow and practical implications. I think on the first side, as was mentioned already and we were discussing this reward model, as far as I can tell, it's not totally determined what the architecture of this reward model should look like, how big it should be in relation to the model that you're fine tuning. How much human feedback should you use? How does the amount of human feedback that you get influence the harmfulness or the utility of the output and that sort of thing. So I think there's a lot to explore around that dynamic between the reward model and the language model. In addition, I mean, language models are still being developed, right? So ChatGPT use the GPT 3.5 language model as this original policy, right? And they actually used a fine tuned version of that using supervised methods and human chat conversations. So they started with a fine tuned version of chat or of GPT 3.5. So obviously we're going to have a GPT 4, GPT 5. We're going to have other language models from other providers, right? From other research groups, you know, Big Science or Google or Microsoft or whoever's developing these other language models. We're going to have updated versions of those. So I think we can see a research direction with this where people are trying different pre trained models as their original policy, where people are trying different reward models, where they're mixing them up in interesting ways, where they're maybe using slightly modified versions of the PPO algorithm or other reinforcement learning algorithms to do the updates. So there's a research direction where I think we'll just see a lot of exploration with this kind of template as the structure that they're exploring. The second piece, which is maybe more interesting to some of our audience, is like, what are the implications of this in terms of people's workflow?

I was about to ask you that if you hadn't gone there.

So, yeah, I don't know, what are your initial thoughts there, Chris?

It's less about the technical aspects of the model and more about going back to the user interface considerations that we talked about earlier in the conversation. I would be amazed if the community at large, not just OpenAI, hasn't understood the impact of making choices like that. It may not be specific to the model development, but how you're putting it out there. And they're seeing widespread adoption. You know, when you go into their interface, you get a warning right off the bat. We're experiencing exceptionally high demand. Please hang in tight as we scale our systems. And I think that's indicative of the fact that people who are not normally listeners of this podcast are starting to find a lot of utility for the first time ever. It'll be interesting. You know, we keep talking about exponential growth in this field and these amazing, you know, kind of mini revolutions along the way, but this is that first point where it's probably going orders of magnitude broader in terms of applicability to different workflows and audiences. So, and as we're looking at, you know, you're combining, just for a moment, going back and combining natural language with the large language models with generative capabilities with reinforcement learning. And we're kind of seeing, we saw slices of each of these fields over the last few years developing, and we've been talking about this fusion of the fields. And so how soon before we start seeing entertainment? You know, that is, that is being heavily, heavily based on these technologies. I'm seeing it in my little tiny nonprofit because we can suddenly leverage this to put out content to help folks in a charitable fashion that we can do at least 10 times as much as we would have been able to before by taking advantage of these. And so I think we're at that inflection point now where this will be the first and as we have continuing episodes through the course of this year, and some new things come out, whether it's from OpenAI or similar things from other organizations, I think we're getting to that point where it's really hitting broadly in real life. So I'm really fascinated. I would love to hear from our listeners on ways that they're using this technology, what they think might come next, and how they are envisioning using it within their own organizational missions to accomplish what they want. It's a fascinating moment in history of AI that we're in right this second.

And one thing which I can't claim as my own insight that I stole from Twitter, but I think has really shifted my thinking a little bit on this subject is, so this is actually a tweet from Chris Albon, who is the director of machine learning at Wikimedia. And the statement he made, which I think was really insightful, and maybe other people are having similar observations, but he said Sci Fi got it wrong. We assumed AI would be super logical and humans would provide creativity, but in reality it's the opposite. Generative AI is good at getting an approximately correct output, but if you need precision and accuracy, you need a human. So I think the observation here is like, and we've talked about this on the show with language models also. Language models are really good actually at naturalness, creativity, apparent coherence. That actually is what they're good at. But they get the facts and the precision and the accuracy wrong many times. Whereas I think in the past people have thought the unique thing about what humans can provide in an AI driven system is creativity, not logic and that sort of thing. Actually, the opposite is really the case. Right? Like the AI bits are really driving the creativity and the humans are enforcing the logic, the facts, the accuracy and the precision. That has really shifted. Like, I think I've been realizing that over time. But that statement really put some words to I think what I was think thinking.

It's comforting in a way. And the reason I say that is we talked in times past about creativity coming from the humans rather than the machines. And yet the evidence that we've been looking at over these last couple of years has been not that. And so I have actually been wondering what role is there for the humans in that equation. So the fact that it's flip flop back, it's the inverse of what our expectation was, it still means there's room for a human in the picture. And that's a little bit of a comforting moment. It may not be what we thought it would be, but there's still a place and I think that's probably a good high note to leave people with.

On the note of things being useful to humans and humans getting involved, we did want to leave you with a few learning resources to explore things related to ChatGPT. Of course, play around with ChatGPT. You can go on the website and interact with it. We'll provide the link. But also, I would really highly recommend that you look at this hugging face blog about reinforcement learning from human feedback. There's actually a bunch of links in there as well to other things that you can kind of spin off and look at, like the PPO algorithm and other things in there. Also, there's a good reference. I always love looking back at Jay Al Mar's descriptions of how certain language models work. He has one on GPT3 and other GPT. Actually a number on GPT different, different from different perspectives. And then there's an interesting article on GPT3 architecture on a napkin from a blog, Dugas Ch. I found it quite interesting how they describe some of the things there.

I like that one as well.

Yeah, yeah. So go ahead and check those out. Those are great learning resources. They're all free. You can take a look at them and learn in more detail some of the things that we only had, you know, 45 minutes to talk about here.

On the podcast in our social media channels. I'm encouraging our listeners to share with us some of the ways they're using the technology. I'm really waiting to hear that. And, you know, the more unique, the better.

Yeah, sounds great. Let us know what you're creating with ChatGPT. It's been a fun one. Chris, good to chat with you.

Absolutely. Thank you very much for the incredibly lucid explanation. It certainly helped me understand and I appreciate it as always, Daniel. Talk to you next week.

Bye bye.

All right, that is our show for this week, if you dig it. Don't forget to subscribe. Head to PracticalAI FM for all the ways and if Practical AI has benefited your life, pay it forward by sharing the show with a friend or a colleague. Word of mouth is the number one way people find shows like ours. Thanks again to Fastly for fronting our static assets to fly IO, for backing our dynamic requests, to Breakmaster Cylinder for the Beats, and to you for listening. We appreciate you. That's all for now. We'll talk to you again on the next one.