the day ai went everywhere (GPT-4 & Google PaLM API release)

5, 4, 3, 2, 1.

Good morning, good morning, good morning!

Oh my goodness, it was a very exciting day. As you can see, GPT-4 has been released.

Now I have no idea how this video is going to go.

I kind of just wanted to grab my live reaction and feeling because I'm quite excited, as you can tell, because I think this is probably a momentous day in history. Like, it's, it's Wednesday, 15th of March, 8 AM here in Sunny Brisbane, Australia.

It is quite warm outside, so I might move inside shortly, but I think this is a momentous day, a big, big day.

But anyway, let's, uh, I've already read the blog post. GPT-4 is amazing.

So I've got a few things.

Uh, there's also Google releases new AI APIs. Google are joining the bandwagon. Anyway, let's jump into GPT-4. It's OpenAI's most advanced system, producing safer and more useful responses. I'm not going to go through the video, all of it. I'm kind of just going to explore. But one of my favorite new things is that it's multimodal. That means that it can take images and text. So this is one of my favorite things, is can accept images as inputs and generate captions, classifications, and analyses.

So you can put in an image ask it a question, what can I make with these ingredients? And have an output.

This is super exciting because my brother and I are currently working on an app called Nutrify, which is take a photo of food and learn about it. So we're already starting to think, how can we integrate this into our app? Now, as far as I know, only the text-based model is available for now. The images are going to come later on, but Nutrify.app, very primitive. But you can take a photo, upload it there. The iOS app is coming soon. That's enough about that. This is more about GPT-4. The main part that you can find out more is by going through the blog post here. Of course, I'm not going to go through it, but we might as well say what GPT actually stands for. Generative Pre-trained Transformer. So let me make this a little bit bigger. Again, this is just completely live. Well, not live, this is recorded, but so generative means GPT models are trained to predict the next word in a sequence. So the dog jumped over the blank. So what's the most likely word there? Fence, or you could say chair. Might be another likely word to come in there. And so if you do that on an internet worth of data, you get pretty good at text. Then we have pre-trained, which is where the, the internet's worth of data comes from. Is so do the above at scale, an internet worth of text. So imagine all of Wikipedia, all of the books that are available publicly online, all of the, the different machine learning datasets, all the— as much text as you can fit into a large amount of computing, which OpenAI have partnered with Microsoft, so they have Azure, which is a lot of compute. And then Transformer is the Transformer architecture, which is from a paper in 2017 called Attention Is all you need. There we go. So it's the same architecture there. So GPT, Generative Pre-trained Transformer. If you scale up the data, scale up the architecture, you get some incredible results. But that is GPT-4 in a nutshell. So some of my favorite things, of course, we already knew that the capabilities were going to be phenomenal. So GPT-3 and GPT-3.5 are already fantastic. But GPT-4 just accelerates that, builds on top of that. But not only in text, it, it does it with now a visual aspect. And again, the visual API isn't live yet, but I believe you can use the new— in ChatGPT, you can use GPT-4. And so there's all these tests here that GPT-4 does incredibly well at. So the bar exam, which is like a a law test. I don't know what all of these tests are, but it's doing pretty damn well. Like 90th percentile, SAT, LSAT. What is LSAT test? Law school admission test. Oh my goodness. Okay. So is it getting into law school? I don't know what a requirement. SAT math is 700 out of 800, 89th percentile. It's doing incredibly well. It can solve calculus problems. So that's an interesting one to note, is that it's a little bit worse, GPT-4, than GPT-3.5 at calculus problems. But that will require a little bit more diving in. And here's some more benchmarks on— these are machine learning, uh, language model benchmarks. So MMLU, can we link on that? Can we click on that?

Here we go.

Test measuring massive multi-language, massive multitask language understanding. So leaderboard. I think this might have to be updated with GPT-4, but essentially GPT-4 beats all previous models and even does it get the best? I think it even beats the SOTA models that are using specific datasets. So GPT-4, the important thing to note about GPT models is that they aren't generally specifically trained for a certain task. And so the process in machine learning of training a model for a certain task is called fine-tuning. So say, for example, you had a really large language model, GPT-4, and you wanted to make it really good at just classifying whether the emails in your inbox are spam or not. You would maybe pass it a dataset of a million emails that are spam and not spam and then go, can you classify this? So that's fine-tuning, taking a big model and specifying it for a smaller task. However, what we're starting to see is emergent capabilities that these larger models don't actually need to be fine-tuned. They're so good at general tasks that you can just, if you put in the prompt, like, say what you want, please classify this email as spam. Spam or not spam, it can do that really good without any fine-tuning. And so what this is important to note is that the GPT-4 benchmarks are better than some of the SOTA models here— state of the art, that's what SOTA stands for— that are specifically trained for certain tasks. So that, that is outstanding. So that is like having a general knowledge system or artificial general intelligence. Can you call it that? I don't know. I don't really have a definition of artificial general intelligence. I like the coffee definition. Uh, Steve Wozniak's definition is artificial general intelligence— can you take— can a random robot walk into a random house and make coffee in the kitchen with— with— it's just a general kitchen, there's no specific things, it just figures out how to make the coffee and it actually makes it for you. I like that. But that is incredible. So it outperforms models that are specifically trained for certain things. And then we have here some great results across different languages. So not only— this is another thing, is that English is the language— I don't think it's most spoken, but it's definitely most studied in terms of language models. And language models often perform really well on English. However, GPT-4 not only performs really well on English, better than any previous model before, but really well on a lot of different languages. And that's so important. Now, of course, visual inputs.

This is where it gets exciting.

So it was really good at text and now it seems to be really good at visual inputs. So what is the funny thing about this image? Describe it panel by panel. So this is a 3-panel image. Of a charger. If I was to go through it, it's a VGA cable that goes into typically into a monitor, but it's got a Lightning cable. And so panel 1, a smartphone with a VGA connector, it gets that. And then it's got the package for the Lightning cable, for Lightning cable. And then down bottom, a close-up of the VGA connector.

This is the output from GPT-4 with a small Lightning connection.

So it inferred all of that from just this image. And then we have some more of my favorite examples.

It can look at infographics. So world daily meat consumption per person, 1997 of the—

sorry, this is in the EU. Finland, right up there, big meat eaters. But to find the sum of the daily average meat consumption for Georgia and Western Asia, we will follow these steps. Identified the average daily meat consumption for Georgia. Identify the average daily meat consumption for Western Asia. Add the values from steps 1 and 2.

Oh my goodness, this is going to be revolutionary for businesses.

You can just input infographics or pictures from presentations and have them read. Like if you have a meeting, the presentation can be summarized by GPT-4 and you can get all the notes. Oh, I don't know what language this is. Principia de— is this Latin? I don't know what language this is. You might be able to tell me, but it can read the paper. This is a technical paper in a different— in a foreign language to me. It can read it and then do math given the equations in the paper. What's funny about this? The unusual thing about this image is that a man is ironing clothes on an ironing board attached to the roof of a moving taxi.

This is— you put in the text and the image and you get this as a response. This is incredible.

Now, this is a paper that is about GPT. So this is a little bit meta. InstructGPT, which is a machine learning paper which was improving GPT results through instructions. So fine-tuning it on different instructions. I believe it's from OpenAI as well.

And so this is—

below is the part of the InstructGPT 4 or GPT paper. Can you read it and summarize it? I would love to know how fast this is, how fast this comes back. If it's as fast as the current APIs, that is incredible. This is a technical paper and it is just outputting. The InstructGPT paper focuses on training large language models to follow instructions with human feedback. So if you, if you ever use the ChatGPT interface and you say, oh, you give it a thumb up for a good response, that's, that's a form of human feedback. Or if it's a thumb down, that's another form of human feedback. I like this result. I don't like this result. You do that at scale, you get pretty good improvements. The authors note that, uh, making language models larger doesn't inherently make them better at following a user's intent. So that's important. If you want a large language model, bigger doesn't always mean it's going to follow your direction. Large models can generate outputs that are untruthful toxic, or simply unhelpful.

Oh my gosh.

It's reading machine learning papers and summarizing them for you. Can you explain this meme? The meme is a joke that combines two unrelated things, pictures of Earth from space and chicken nuggets.

Oh my gosh.

I love how OpenAI have just released possibly the most advanced AI model ever, and it's deciphering memes.

This is beautiful.

So if you want to improve, can you explain this, why this is funny? Think about it step by step. So this is another trick. It's called prompt engineering. Step-by-step prompt engineering. If you, if you say please—

yeah, there we go.

In plain English, let's think step by step. It's an instruction to GPT which is going to explain it step by step. I'm going too long on here. Uh, so visual question and answering benchmarks. So these are for vision capabilities. Uh, text VQA, vision question and answering. So GPT-4 zero-shot means it's given no examples to, to practice on. It's just, it's just, hey, you've got this big amount of knowledge, here's an image, answer this question about it. So it's getting 77.2% with no, never previously seeing similar examples. This is just the baseline model. That's what few-shot evaluation means. And then here's the few-shot state-of-the-art is Flamingo, which is a big model from Google and DeepMind. And then, uh, SOTA state-of-the-art, which is, uh, a best external model which includes specifically training for that task. So VQA, visual question answering, the best model which has been trained specifically for that task is 84%. Whereas GPT-4 is not trained for that task, but it's getting 77%. It's even beating it on textVQA.

This is incredible.

Uh, and then another important point is the steerability, so the way you can navigate the model to produce the outputs you would like. So this is the chat interface system, which is very, very powerful, and, and kind of, I I'm imagining a future where every app or every operating system, macOS, if we just built it into Spotlight, I, uh, Apple are definitely working on something like this, or I hope they are, where you could chat to any app from editing a video in Final Cut Pro or, uh, whatever. Every app is going to have some sort of chat AI capability. So the JSON assistant. So you tell the system that you are an AI assistant and always write the output of your response in JSON. This would be very helpful if you were building, say, an app where you wanted to structure the results coming back from the GPT-4. Like you could say, how's it going? The response comes in JSON and you can quickly format that on the front end of your application. So if we go back to this example, what can I make with these ingredients? You could pass it the image and then have the response come back in JSON. So instead of this wall of text, you could format that into a way that's beautifully displayed. When we go through here, we have a next sample. Socratic tutor. So this is steering the model again. So you tell the system that you are a tutor that always responds in the Socratic style. You never give the student the answer, kind of like a Zen master, but always try to ask the right question to help them. I can't imagine what this would be like for tutors. Does that mean my— Did this just automate my job?

I hope not. I like teaching. And then Shakespearean pirate.

Now, limitations. Of course, there are some limitations around safety. I'm going to be honest with you. I don't know too much about AI safety.

I believe if you trust in humanity, they will generally use it for good things. But of course, there will be some nefarious actors.

The good news is that GPT-4 is much more factually correct, and this is based on 9 different categories and evaluated. Yeah, 1.0, 100% accurate, I believe would be the model's answers are judged to be in agreement with your human ideal response for all questions in the eval. So this is another caveat with large language models is that they can tend to hallucinate, and what that means is because it's a generative model, it will always generate something. The something won't necessarily be correct, as in it's got a lot of information packed into there, an internet's worth of text plus a little bit more, of course. And so it will always produce something. However, the hallucination might not be factually correct. Like if you ask it, what are the nutrition information for an apple, it will give you something, but it might not be factually correct. So this is just saying that GPT-4 is much better at factual, factual evaluation. If we keep going, adversarial questions, we have improvements there. GPT-4 answers quickly, correctly. Can you teach an old dog new tricks? I guess you can.

This is what I feel like right now. I've been in machine learning for almost 6 years and it's like I feel like an old dog with these sort of models. So hopefully I can learn some new tricks with these large language models.

Answers incorrectly. So this is, it's not perfect, of course. It's very good, but it's not perfect. Let's go on to, I'm gonna skip a few things, but I'll link the blog post here that you can read it. One of my favorite things is the training process. And this is incredible research here, is predictable scaling. And what this means is that they, could predict how the loss function or the loss result— and what, what a loss function is in machine learning is how wrong your model is, and that's basically the, the most important metric, uh, when you're training a model. So if you wanted to train a model to predict photos of green apples and red apples, the better it does, the lower the loss will be. And you can just take that paradigm, apply it to any sort of machine learning model in general. So the important thing here is that OpenAI were able to predict their big model's loss value. Why does this matter? So you could predict GPT-4's final loss from their internal codebase, not part of the training set, by extrapolating from models trained using the same methodology, but by using 10,000 times less compute. That is beautiful. That means one of my mottos in machine learning is experiment, experiment, experiment. So you can train a small model on 10,000 less— 10,000 times less compute. So say you had, uh, one GPU on your home computer and you wanted to run an experiment, and you could then figure out how that experiment went on one single GPU, your laptop right here. And then you wanted to go, okay, now I want to scale this up to be an internet's worth of data and 10,000 GPUs. Not just one computer, 10,000 computers in the cloud. You can, you can accurately say, well, at least this graph shows, you can accurately say that you'll be quite bang on if your small experiment works, your big experiment will generally work too, by this research. Similar value here, capability prediction on 23 coding problems, right, basically bang on. With 1,000 times less compute. So this is— this, this going forward, this may be— we may see far more models released. We may see far more models released because if you can accurately predict how well the model will go on a small experiment, then large experiments obviously cost a lot of money. But if you can accurately predict a small experiment for $10, and it will go, well, based on this, we can go, if we spend $10 million on this, it will, will pretty much be the same results. Well, then you can accurately go, well, I want to— we can invest this because we know it's going to be much better. So we might see GPT-4.1, 4.2, 4.3, etc., on and on and on at a rapid rate. Then there's also a new library called OpenAI Avails, which is to evaluate your language models.

If you want to use ChatGPT-4, how can you use ChatGPT-4 or GPT-4?

You can use it on the Plus subscription on ChatGPT. My brother's got that. He's finding it very helpful already. So GPT-4, I imagine, will be much, much better. Well, not much better, but because ChatGPT 3.5 Turbo is already very good. But you get what I'm saying. And then of course, you can access it through the Chat Completions API. Where you sign up for the waitlist. Another incredible thing is that it has such a much larger capacity. So GPT-4 has a content— a default content length— context length. I say contents and context— context length of 8192 tokens. So previously it was about 4000. So if we go, how much is a token? Let's see. There we go. So the larger models also have an access of 32,000 tokens, which is about 50 pages of text. So if we divide that by 4, that's about 12 pages. So that means that you can input into GPT-4 12 pages of text on a base— on the base model and ask it questions about those 12 pages of text.

And then on the bigger models, you can put in 50 pages of text.

So just think about how long that would take for a person to ask a question on 50 pages of text. Like if it's a textbook, I've got some pretty hefty textbooks out there and if I was to read 50 pages, that would take me maybe an hour, 2 hours, 3 hours. Let's say an hour per 20 pages. So 2.5 hours. And then I would have to basically try and regurgitate what I'd read about those 50 pages. You can now programmatically input. Can you imagine every textbook having a chat companion and you just ask it questions? This is, this is, this is mind-blowing.

March 15th, 8:00 AM in the morning and ChatGPT-4 gets released.

So the pricing is 0.06 per 1K prompt tokens and 12.

Let's jump into what else do I have here?

So the models here, if you want to use the OpenAI API, you can use GPT-4 in a limited beta. I've been playing around with 3.5 and it is already phenomenal, so I can't imagine what 4 is like. I've applied for access. Pricing, it's a little bit more expensive than GPT-3.5 Turbo.

I think for most use cases this model is, is primo.

$0.002 per 1,000 tokens. So if we did— if 50 pages is 32, so let's go 32,000 times $0.02, it costs you for 50 pages to talk to the current GPT-3.5 Turbo, it costs you $0.064 USD. So that is incredible, but it can't take it in one hit. That's the difference. So 32K context, that may be perfect for legal use cases.

If you could read 50 pages of a legal document, or if you had a million pages and you had to put them all in and you want it to use more context from its results, that is the power of having a larger context window.

Eventually, who knows, we might get to the point where it can just take unlimited context window, but for now it's still limited. Chat completions. Now this is where GPT-4 comes into play, is there's a new API style which is like chat. I think they call it Chat Markup Language, Markdown. Anyway, it doesn't matter, but it's that to use GPT-4 is the same API format as GPT-3.5 Turbo. That's in here. And then if we have the playground, so let's, let's try it out. We have— we don't have access to GPT-4 yet in the playground, but this is the way we would interact with it. You tell the system something. So you are a helpful assistant that always answers in the style of Thor. And then let's go enter a user message, message here. Um, what could we do? Let's go, how do I get big arm muscles? Because if you're coding Python, you've got to be rocking pythons. So let's submit. Verily, to attain mighty arm muscles, thou must engage in the lifting of weights of great heft and the performing of exercises that target the biceps, triceps, and forearms. Consistency in thy training regimen and a proper diet of protein nourishment shall aid thee in thy quest for formidable arms. Be steadfast in thy efforts and let not the weight of the world discourage thee. For the path to greatness is often wrought with challenge and adversity. May the power of Asgard guide thee on thy journey.

Oh my gosh, this is incredible. And then to finish off, I think I've covered all I wanted to cover. Context length. I don't know if you can hear that, but my washing machine is done. Multimodal chart reasoning, paper reasoning. Predicting the loss, outstanding results, GPT-4 equals a bunch of different things, and Google releases new API. So Google has across 3 different blogs, blog.google, cloud.google.com, and developers.googleblog.com, uh, has announced they have Vertex AIs for their version of the GPT API or the OpenAI API. So you can now use Google's generative AI in Google Docs. This is what I'm saying. This is, this is a foundational moment of where AI AI is going to be integrated into everything. So that's what I just want this video to be, is like March 15th, 2023. Uh, this video is really good on how you can use Vertex AI models. So Google have a bunch of different things. I think it's still just quite limited beta, like, but it's coming out, it's coming out. They have an example of where you can generate images, marketing text, and whatnot.

I'm going to link all of these below, but again, Palm API, So PaLM model.

Oh, did you see in this blog post, the PaLM model is the other model that they really compare to. So I'm not sure what Google are using behind the scenes, but basically every comparison between GPT-4 is between some sort of PaLM model. So Google are coming to the party with PaLM, the PaLM API, and that they've done it across 3 different blogs, which I find amazing.

They must be like, we need to get this out there. So this is going to be integrated into Google services where you can generate images, generate text, augment your dataset with synthetic data. Oh wow.

So it's a big day. It's an exciting day. I'll put all the links that I've just gone through in here. Um, but yeah, if you're watching this, uh, Daniel in the future, March 15th, 2023, a beautiful sunny day in Brisbane, Australia, and potentially the day that AI started to be integrated into everything.

So I'll see you next time.