Practical Data Science on AWS: Generative AI

Sam.

Hi everyone and welcome to Practical Data Science on AWS Generative AI. My name is Greg Lochnane and I'm the Director of product at 4th Brain. We appreciate you taking the time to join us for this event. We're so glad to see you tuning in from all over the world if it's late for you. Thank you for staying up to join us during our event today you'll learn about doing data science and machine learning in the cloud through aws. You'll see how you can generate images, text and even programming code by leveraging AWS tools along with open source machine learning models. You'll even get a brief history lesson on how the amazing generative AI models available today, like ChatGPT, came about in just the last few years of rapid evolution. We're going to get right into it today. If you hear anything during our event that prompts a question you'd like to ask, please follow the slido link in the description box on the YouTube page. If you're just listening, check out the link to see questions others have asked and upvote your favorites. We will do our best to answer all of the most upvoted questions during the Q and A portion of this event. Without further ado, it is my pleasure to welcome our speakers to the stage. Antja Barth is a Principal Developer advocate for AI and ML@AWS. She is co author of the O'Reilly book Data Science on AWS and Antia, frequently speaks at AI and ML conferences, events and meetups around the world. She also co founded the Dusseldorf chapter of Women in Big Data. Chris Fragley is a Principal Solution Architect for AI and ML at Amazon Web Services and he's based in San Francisco, California. He is also the co author of the O'Reilly book Data Science on AWS. Chris is the founder of the Global Meetup series titled Data Science on AWS and he also regularly speaks at AI and machine learning conferences around the world. Previously, Chris was a founder at Pipeline AI where he worked with many AI first startups and enterprises to continuously deploy ML and AI pipelines using Apache Spark, Kubernetes, TensorFlow, Kubeflow Ray, Amazon EKS and Amazon SageMaker. We're excited to welcome both ANTJ and Chris. Off to you too.

Thanks so much Greg and thanks for having us today. We're super excited. Thanks everyone for joining us today for this session on Generative AI. Chris, do you want to say hi as well?

Yeah. Super excited. I'm a big fan of this series and I was stoked when I Heard I could actually present. Thank you so much fourth Brain.

All right, let's jump right into it. Before we jump into the topic for today, I just want to share here a couple of links and pointers for you. If you wanna learn more after this session today, especially around practical Data Science and how to run Data Science NML in the AWS cloud, check out this deep learning AI specialization that we have jointly on Coursera. Here's the link and I will drop it in the chat for you. As well as Greg mentioned, Chris and I we also wrote a book together Data Science on AWS which is available, which also uses an NLP Natural Language Processing use case for throughout, showing you how to start with working with the data, preparing your features, up to training, tuning and deploying your models on AWS using a variant of the BERT model. So also a great resource if you want to follow up on large language models. After today's session, Chris and I also are running a meetup group. You have some links here on the left. So we have monthly sessions with speakers exciting from the space of Data Science ML and Data engineering. We're talking a lot about latest innovations there too. We have a YouTube channel with all of the recordings and of course also relevant for today, we do have a GitHub repo with a lot of sample code that you can also reuse after today if you want to play around with the topic in your own environment. So check out the GitHub repo later as well. All right, and with that let's jump right into today's topic. We have a lot of content for you today. Generative AI there. It's so much happening in this space as you might see every day there's something new. So we tried to get really all of the exciting stuff here together for today. We're going to start with a little bit of why Data Science ML in the cloud. Then we're jumping right into a couple of the use cases for generative AI and we'll give you demos of course. Chris will show us how to do text to image using stable diffusion, a very popular diffusion model will do source code generation with Amazon CodeWhisperer. Also we're talking about of course ChatGPT and what the model, how the model is built and give you a little bit background there. We're also going to give a brief overview of how we ended up where we are today. Diving a little into the NLP evolution over the last couple of years. We'll talk about prompt engineering. You cannot do a session on generative AI without talking prompt engineering and another demo using another popular model promoting Bloom. All right, it's the whirlwind of topics here. Let's dive right into it. Let's start why data science and machine learning in the cloud. Many of you might start developing on your local laptops, right? On IDEs, using IDEs. And as you experiment and build your projects, you might hit a point where you're constrained by the laptop, right? Memory sources, disk space, CPU power, right? Some of us might have a GPU on the laptop, but also they're reaching limits. So the next step where the cloud can really help you is to provide you with an elastic on demand infrastructure. That means you can easily tap into more resources as you grow your projects, as you need more compute, as you need more storage. And the elastic part references to two concepts here. One is scale up, scale down. That means for example, you're starting out with a, let's say a CPU based instance, right? And you're hitting specific constraints. You can easily size up to a bigger compute instance that has more powerful CPU or even GPU in there, more memory, more disk space, etc. There's also this concept of scaling out. And in scaling out means you, you're scaling horizontally. So maybe you started training your model on a single instance and then you're hitting a point where with large models you want to look into distributed training across various instances. The cloud gives you that infrastructure where you can easily get the resources you need for doing the job. Also notice it's scaling down and scaling in again, meaning you can scale really just to the amount of compute you need get and make it a very cost efficient environment. Also, the cloud has purpose built hardware. I'm going to show a little bit here in a sec. And you can leverage the ability to store and process any amount of data, right? If you're looking into those large models, often in the pre training phase, this is trained on huge amounts of data and you need a place to store that. So the cloud is perfect for that. And what we're going to show you today, the cloud is also providing you with a large data science and machine learning toolkit. A lot of services and tools you can use, especially in building, training and deploying your models. When I talk about toolbox, I just want to briefly mention here a quick peek into the AWS toolbox. There's a lot of different tools you can use. One stack here is what we call the AI services. And this is a collection of tools you can use right out of the box. Whether you're working with computer vision or with speech, with natural language text, et cetera. Those services are already pre built and you can simply use them by calling an API from your application. Let's say if you want to do translation jobs, if you want to do image recognition, et cetera. The majority of demos today that we prepared for you and we're going to walk you through, is using a service called Amazon SageMaker. This is a managed service which takes care of all of the infrastructure setup, but gives you a little bit more freedom than those AI services in giving you access to tools you can use to build train your own models. And then if you're an expert practitioner and you want to build everything yourself, you just really need to tap into maybe the compute environments. Of course, we do have that option for you as well where you can tap into the compute, use optimized frameworks for AWS, PyTorch, TensorFlow, MXnet that have all of the optimizations and build your applications right on top. All right, just a brief plug here as well. I mentioned the purpose built hardware. So by using the cloud you get access to the latest innovations, whether it's specific Nvidia GPU instances. You're looking for accelerators or maybe the Intel Habana Gaudi accelerators. And the two highlighted here from the training and also from the inference environment are purpose built by AWS. And they have those funny names, AWS Trainium and AWS Inferentia, which might give you a hint for what purpose they are. Here's a little bit more I know this is kind of an eye chart with a lot of numbers. Some of you might be interested in seeing the performance numbers. AWS Trainium and Inferentia are custom silicons developed by AWS to give you the best performance at the best cost efficiency in the cloud. On AWS Trainium really highly optimized for running machine learning training workloads. Here you can see the petaflops numbers and comparable to other GPU instances, gives you up to 50% of cost savings as well. And also on the inference side, we just announced Inferentia Toots, the second generation custom built silicon for inference jobs. And also they come with a lot of optimization particular if you're looking into those large models in the generative AI space, Inferentia can do distributed inference large models across the different accelerators in that instance. So if you're looking into that space, definitely check out those purpose built compute instances. All right, and that was just a really quick tour through a little Bit of the vocabulary around aws, which we're going to use later today. And now let's switch gears and talk about generative AI. To start, what is generative AI? And I think everyone has experienced it in their own lives, whether looking into what's happening or using some of the available services. Generative AI at a high level can produce original content that is close enough to human generated content, whether it's text, whether it's images, whether it's code. And we're going to see that in action in a little bit here. It's also powered by what we call foundation models. Foundation models are large trained models on a big set of data and have several billion parameters. Right. We started out 10 years ago models in the million parameter space. Now we're talking billions of parameters. The tasks can also be customized of those models. Right. To specific domains with a minimal amount of fine tuning. So Chris has an exciting demo later. I heard there is a doc involved when he shows how you can use those big foundation models and then customize them specifically to your data. And they ultimately they help you to reduce the time and cost to develop those models because you can build off the existing huge foundation models and helping you to innovate faster. All right, so for this talk today, for this session, we're going to focus on the three kind of big use cases. Image generation, code generation, and text generation. Let's start with the image generation. And here I picked one of the model architectures that have been in the spotlight recently, that is diffusion models. How do diffusion models work? You can see here on that animation. The first step is they're destroying the data, that is the images they're trained on gradually by adding noise. And then in the actual training step, they try to denoise that image and basically create gradually a new image. So this is the two steps the diffusion models use. A famous one is stable diffusion by stability AI. Stable diffusion can also generate those images from text. So this is kind of one phase you have to embed in when you build the model to make it using text prompts to generate those images. And stable diffusion is one example of that. For instance, stable diffusion is a latent diffusion model. That means it learns to recognize shapes in a pure noise image and then gradually brings these shapes into the focus. And, and if the shapes match, specifically the words from the text prompt and the text must first be embedded, as I said, must be known in that latent space using another language model. So it's a little bit of a more advanced architecture here, but basically still following this diffusion model and then denoising the output and creating those images. And we can see an example here from the stable diffusion 2.0 model. What the model is really capable of doing, you can see here on the left, is an image created by stable diffusion 2.0. This interior does not exist. We gave it a text prompt and this is the output. Those models can also be used for a lot of different other tasks like transforming the image content, as you can see here, or upscaling the quality of an image. And I think we have a demo coming up, Chris. Right?

We sure do. Okay, let me flip over here. We're ready to go. So as Ansha mentioned, I have used stability stable diffusion to train a model, a generative model on my dog Finn. And so let's scroll down here. So this example I believe we have, the link is in the slide right there. So we'll share that on the chat. But just to walk through this real quick, I have a local folder here with some images and I've got these in line in the notebook here in a sec. But that's my dog Finn. And let's scroll through. We're basically uploading these to S3, which is our simple storage service. So that's object store in the cloud. And eventually we're going to fine tune. So we've got this pre trained model, large pre trained model with millions, billions of parameters. And for I believe this is 70 cents per hour. And I think this ran for about 10 minutes. And I think that ends up being under, it's certainly under a dollar. It's you know, 1/6 of 75 cents to train and fine tune on my dog. And now I'll be able to generate fun little images of my dog in the context, in a nice human language context. So let me just kind of skip through all this, all the code is out there for you to try. I actually took this right from what is called SageMaker Jumpstart and this is within AWS. And if you go to the SageMaker console and there's a little Easter egg here, it's actually mentioned in the blog post that this comes from. But you could generate fun images of your dog and that ends up here. All you do is upload your images to the notebook environment which ultimately gets uploaded to S3. So this gets trained on a SageMaker cluster. Pay as you go, pay for only what you need. So this trained for 228 divided by 60. So that's this actually trained for 20 minutes. And let's see, point and it was 0.75 per hour, something like that. So this costs, I believe 25, 26 cents to train if I'm doing that math correctly. At the end of this we're going to make some predictions here. And here's an original image of my dog Finn. Very cute. Oh yeah, look at that little nose. It's got those, those funky eyes going on there. So that's a few pictures of Finn and this was a human. So this was the prompt. I said, please show a pencil sketch of my dog Finn. And they took the fine tuned images and this is what came back. So this kind of looks like Finn with those eyes and the floppy ears and so that's kind of fun. So that's the demo. They're very, very easy to reproduce. And back to you, Ansha.

All right, thanks Chris. I love fin. Cute doc. All right, let's switch gears. We saw image generation and now let's have a look how we can generate code. I want to introduce you here to a tool from Amazon, from AWS called Amazon codewhisperer, which we launched at our last big user conference, Re invent. It's currently available in preview and we can do some super cool stuff with it. And Chris, I think you have another demo coming up soon here. Codewhisperer is. Think of it as a coding companion. You can use it as a plugin in your favorite ide. Here's a screenshot from Visual Code, Visual Studio Code and you can install the plugin in there and then you can start typing comments. Codewhisperer will intelligently create the code you're looking for or you start writing the first line of maybe a function you're defining and then codewhisperer can help you complete that. Basically think of it as an autocomplete for your code. Super exciting. It's available in popular IDEs. As I mentioned, it supports a couple of different languages right now. Python, Java, JavaScript, C sharp and TypeScript. And it's very simple to sign up, you just need an email address. And then also what's very important here to Note is that CodeWhisperer will generate open source attributions. So we know this is a very important topic with those code generation tools that whenever code is reused that it gives the proper attribution to where the original code lives. So this is also built in here into codewhisperer I think, Chris. Yeah, don't start doing other things. I think you're up again for a demo. And you need to unmute.

This is the exact code that Ansha showed in that Screenshot. This was 100% generated by CodeWhisperer. If you think of a comment as a prompt. So if I start typing, visualize data set. Boom. And then it's gonna, you know, and if I put in here from Parquet, it should actually start to, you know, generate things to prepare for parquet. So this is kind of a simple, you know, pandas. I also asked it to plot two dimensional points, retrieve just the first two columns, plot those. This was all generated. And here's a little bit more advanced. This is an XGBOOST model. So I just started typing Train XG boost model. This stuff just started popping up. Look pretty reasonable to me. And we're even splitting the data here. This was all generated and I say fit the classifier. Boom, boom, boom. And then. Okay, so that's cool. So codewhisperer knows about pandas. Great. Show me more. Okay, I'll show you more. Here's some pyspark. I use PI Spark all day. I use Spark all day, all night in my coding activities with customers and personal things. And here I just started saying read CSV data. It knew it was from Spark because up above I had said load CSV data with Spark get or create the Spark context. Boom. Started doing all that. Once again, 100% generated by codewhisperer. I do want to give a real world scenario that actually came up very recently. I was working with a customer who is new to Spark. This team, very, very machine learning savvy, but hadn't really spent too much time with the Spark APIs. And they were actually just about to implement their own random split on Spark. And I said, oh, by the way, do you know that you actually get this for free? That this has already been around, you know, Spark since I think like 2013 or something. So if I just start typing in random split data into train and test data sets. Boom. That's the code right there. And then I'll run this here in a sec. Also I want to, you know, verify. So show the size of the train and the test data sets. There it is. This was all generated. I'm going to run that. And there it did the random split. The scenario here was this team was about to go up and build their own on lower level Spark primitives. As I was working with them and we installed codewhisperer, they were able to see these APIs just by typing in the comments that they didn't even know existed. So pretty cool. This is all assistive. This is not going to replace us as coders, in my opinion. This actually helps me realize that maybe there's a best practice that I didn't realize. Maybe there's a newer API. Here's a advanced one here where I said implement a transformer in Pytorch. Of course, Transformers. I'm just going to get to this in a bit. Is the basis of all of these modern language models. The T&GPT stands for Transformer. And just by typing this comment, all of this code was generated for me. And I say in Pytorch, if I change the TensorFlow, it would generate TensorFlow code. Okay, Ansha, back to you.

Great, thank you. And just a quick call out. Codewhisperer is currently available in a free preview. So if you want to test it out, follow the link that we post in here and also in the chat and sign up and test it. All right, let's switch gears again to text generation. And I don't think I have to introduce this much for this audience. Right. I think ChatGPT reached a level where I think even my mom the other day asked me about it. What is it? And how does it know to write recipes for my favorite cake something. All right, so ChatGPT can do a lot of things. I think we all have played around with it. It's one of those examples where you have we heard that word a prompt and you can ask a question, you can ask it to do a task and it creates and generates the output. So in this case here, we asked it to generate product. A product review for some software product. We will explain why we did this. But that's not all. You can also. That's just a fun example. Actually. Chris found you can ask it to upgrade your code. Who would have thought that? So here's an example, actually, we found we put the source link in here. Somebody asked ChatGPT to take this Go package that uses a specific AWS SDK and rewrite it. So it uses the version 2 of that SDK. So do upgrade the code. And actually there's pretty decent output. And also provided a summary of the tasks to do to upgrade the code. So super exciting. And I think there's a lot of more use cases that will come out in the future around the generative AI. But let's take a step back and actually have a look at how did we end up here. So what suddenly happened in the space of natural language processing that we're now able to do all of those exciting things? It didn't happen overnight, right? Natural language Processing, NLP has been studied as a field for many, many decades. It kind of experienced a renaissance, a newfound focus around 10 years ago. And I would bet like many of you have played around with Word2Vec, one of the first really powerful algorithms back then. It's been really kind of what we would call a shallow neural network these days and working on what was called bag of words and continuous skip grams. So a very easy algorithm, but really powerful in the time. And I bet some of you might still be using some word to back here and there. And then over the years there came new model architectures. The researchers were trying out different model architectures, new things, matrix vectorization, tuning those character engrams for that algorithm. But really kind of the biggest revolution, I would say was in June 2017 when this very popular paper, Attention Is all youl need got published. And this paper, it's still being cited today. If you haven't read it, I would highly recommend it. It's kind of introducing the transformer architecture which is still being used in all of the current models to some extent. The paper also introduced the attention concept, right? So how to put the importance weights on specific words in a sentence, for example. And there were a couple of different algorithms that came up after that. But kind of the biggest model architectures were definitely GPT, which is still around, and bert. And just a fun fact here, if you haven't noticed, the table stands for transformer. So GPT is using one part of the initial transformer architecture called the decoders and BERT is using the encoder part. So there are differences in how they implement it, but it's all basically building upon this initial work and different ways of implementing Transformer plus the attention. Now what happened over the years, researchers improved on that architecture. In 2020 we had GPT3, which jumped from the million parameters to the time we're now talking 175 billion parameters. This is when we also started to talk about large language models, LLMs. Also the industry, the community came together. This is a slide I grabbed from the State of AI report, if you're curious. That shows how the community came together and started developing open source models that are building on top of the popularity around GPT3, but making it available to the community. So all of the open source models are highlighted here in red. This is really kind of the phase we're in right now. I think every week there's a new model architecture coming out, a new model. Something's trained, something's improved, especially in the use case around generative AI. So let's get back to ChatGPT and let's talk a little bit about this model architecture, how it is trained and I got that information from OpenAI. They share a little bit about that architecture. ChatGPT is fine tuned from a series of models trained on text and code, which they call GPT 3.5. So I think GPT 4 is coming soon. But GPT 3.5 is a series of models that has been trained in a specific time range with data. And GPT3 is fine tuned from there, but they didn't stop there. It's also further fine tuned with reinforcement, learning and human feedback. And I will touch on this in the next slide as well. Because this is such an important concept, it is used especially the human feedback is used to improve the results. They first introduced this concept, by the way, in another model called InstructPT. So think of ChatGPT as what they call a sibling model to instruct GBT that initially introduced that human feedback. The human feedback is very important, not just to rate the quality of the output of the text, but also to make sure that the model, the goals and the values are aligned with the human intent. And that means it's not producing harmful content and it's reducing toxicity. And this is a very important concept of course. So you will hear more about this alignment in this space, I'm pretty sure. Also here, just fun fact on the screenshots you can see here on the right, if you're wondering, you are a human labeler too. You are part of the human feedback and how. Well if you play around with those models, you will often see a thumbs up, thumbs down. How was that output? Was it good? Was it the right one? Do you have better feedback? Like what would the best output would have been? Or if you do a thumbs down, like why is it, is it harmful? Is it not true? Is it not helpful? So by responding to those prompts and giving feedback, you're becoming a human labeler yourself. Isn't that exciting? All right, let's have a quick look at this architecture, how OpenAI is training the model. And it's basically three different steps here. So step one, it's pretty much a supervised training step, fine tuning the model. So they're taking this GPT 3.5 model and the unique thing here is that you have a data set that consists of prompts and then the response. And what they did here is they have human labelers, human in the loop that take a prompt and Then they provide, let's say, the desired response. So they create a very high quality data set with having the humans really guiding it, what the right answer would be. And then they're using this high quality training data set in a fine tuning step. But they don't stop here. Right. So in step two, what they're doing, they're preparing for the reinforcement learning. So they're taking another prompt and get a couple of model outputs here. So they're getting, let's say four responses to that prompt. Then there's another human labeler that ranks those outputs from the best to the worst. And this data is then used to train this reward model which will take part in step three. In step three, they're applying reinforcement learning. So think of the model as being the agent in terms of reinforcement learning that needs to navigate the environment and in this case produce the desired output. So they're taking a prompt from the data set and then using reinforcement learning and this trained reward model, they're checking if the produced output is the desired output. Right. And then through the reinforcement learning, the reward is used to update the reinforcement policy. They're using a specific algorithm here, ppo, if you're wondering, that's called proximal policy optimization. So all of those steps together is really what's enabling the quality, the high quality of those models. And a big part, again, is the human in the loop in here. Again, here's a fun example of how human feedback improves model results. I found a tweet from Dr. Andring here in December. He asked ChatGPT why CPU computation is faster than GPU computation. And you can spot that this is a trick question. So generally GPU would be faster, Right? But again, chatgpt, that time didn't know any better. It fell for the trick question and actually reasoned for why CPU is faster. I did the same test just a couple of days ago, asked the same question, and see now ChatGPT learned from the human feedback that probably many people gave and now is able to actually say that. No, actually in most cases GPU is faster and reason through it. So that's an example of, as you can see, the model is improving over time. All right, let's spend a few minutes here on prompt engineering. We heard the term a few times. In general, the prompts are the inputs to the large language model, to the generative models, and they are used to guide the output to create what you want to receive. Right, Useful output. So prompts can be a simple question. You can say, who won the most recent world Cup. And then hopefully the model comes up. Depending on the training data, with the right response, prompts can also provide context. So, for example, I asked the model here to write me a country song that includes specific terms like a horse, a sunset, and a tale of love. Right? That's what I learned moving into the United States, what a good country song is made of. And you can see here, the model, actually, I think, comes up with a pretty decent response. I would have to ask Chris here, because he's a big country fan, how he would rate that country song. Do you approve? Okay, he is nodding. Perfect. Prompts are stateful and combinable. So I can have a human like conversation and I can follow up and say, oh, no, actually, now write me a rap song. And it's still in this context. So the model understands that the rap song should also contain those three terms that I provided, even though I haven't repeated that in that specific prompt. And here I think it's a fairly decent rap song. All right, back to a little bit more on what prompt engineering entails. So we all are familiar with those prompts when we're doing the inference right when we're using the model. But prompt engineering also plays a big part in preparing the training data sets. And here's an example. Prompt Source is a toolkit to work with prompt data to create and use it in the training steps. Here's an example of a prompt training dataset for sentiment analysis. And we're coming back to those reviews initially that I introduced you to when I asked ChatGPT to write a product review. On the left, you see a template, this one here. And you can find all of those resources if you look up the prompt source project from Big Science. Here on the left is a template that shows how to read the data from the data set. Here it circled the title of the review, the review content, and then it's saying, is there a negative or positive tone in this product review? So that's kind of the prompt we want to get. And then the possible answers in this case is positive or negative. So this is sentiment analysis and a type of classification. But we're creating this data set here using a template. And then on the right, you can see here the data filled in using the template. It's basically coming from the data set of reviews, but putting it in a format that contains the prompt and then has here the input data and also the target and the labeled results. Positive, negative. There's different things you can do, of course, so you can also engineer the data Set for generating or summarizing the review. So on the left here we're prompting for generating, so generate a star rating. So 1 to 5 star review about this product and then again placeholder for the title and the review. And you can put in metrics. Here you can see the blue and the rouge metrics which are popular for generating and summarizing text summarization. And in the right screenshot, here's another example of that, but focusing more really on the summarization task. All right, and with that we're coming to our third demo in a bit here. Chris, get ready. We want to show you another popular model out there, Bloom, that's based on GPT3 architecture. And what makes Bloom really popular is it's the first multilingual large language model that has been trained in complete transparency by the Big Science initiative. And they're also doing everything open source using a responsible AI license. It's a similar architecture to GPT3. They did some improvements. It covers 59 languages, 46 natural languages, spoken languages and 13 programming languages. Super exciting. They trained in just a little bit over 100 days. And it's deeply integrated with the Hugging Face Transformer library. Super, super popular library, if you're working with Transformer based models. And those models, including Bloom, are available through Amazon SageMaker, so you can easily start working with them. You can use a prompt here, you can just test it, play around with it and you can further use it and fine tune it to your data. And I think, Chris, you have another demo coming up here showing Bloom in action.

I do. Right. Let me make sure I'm zoomed in. Hopefully this is zoomed in enough. Okay, so here, the context here is we have fine tuned GPT3 Bloom model from Hugging Face with the a public data set called the Amazon Customer Reviews Data Set. And this is 150 million reviews over, I think from 1995 to 2015. This is something we used in our book and some other workshops that we have. We have carried it over to the generative use case and what we're going to do is, you know, something kind of fun, kind of silly, but we have fine tuned it on one particular product category called digital software. So the product reviews for things like TurboTax and all downloaded software between those 20 years, we have taken the base GPT3 model that's called Bloom, and we have fine tuned it and passed in all of the, all of these reviews. And now we are going to ask the model to generate product reviews. So just to show you what a Product review for Norton Antivirus would look like before the fine tuning. This is. So here we're just pulling in the straight pre trained model which was pre trained on you know, Wikipedia and you know lots of public corpuses but not specifically yet on our Amazon customer reviews data set. So we ask it before fine tuning tell us, you know, write a product review. Well, it seems kind of silly. If you are an admin, please provide your admin credentials for the product reviews. Doesn't really make any sense. Ask it. Also this is tax season in the U.S. you know, we're all kind of scrambling to get our taxes done and in the next month and a half or so and you know, pretty nonsensical here, it's just sort of generic and it's because the model has not yet been fine tuned. So let's quickly fast forward. I ran this right before and let's see, this ran for about 50 minutes, 50 minutes. And also keep in mind that I'm not even using GPUs here. So this would actually run quite a bit faster because we are building these as a sort of reusable workshop for the public to use. We don't want to put the burden of GPUs. If you do switch to GPU things will run a whole heck of a lot faster. But for the general public and to keep costs low, we are just using a CPU instance here, the M52XL. And okay, so I have pre trained it up here. This ran before the, you know, while we were talking here and now it's done. Now there's a pre trained, a fine tuned model and now let's ask it to write a review for Norton Antivirus here. This actually starts to look like a review. And keep in mind this was one epoch, one like epic and I think it was only 100 steps through the dataset so it didn't even see the full data set. It's just you know, small, a very small amount of training and but this actually starts to kind of look like a review. And here, you know, recognize us click on the download icon, you have to download the key, blah blah blah. Now let's ask it to do TurboTax. So first person have used TurboTax feel comfortable with it as I always have. I recommend this, you know, starts to recognize Mac and Windows and Windows 2010. And then so as we can see with just a little bit of fine tuning we could start to get more and more realistic reviews. Okay Ansha, back to you.

Perfect. Thanks so much Chris. And now we finally know how those product reviews get generated.

All right, let's be very clear. We do not actually generate these reviews. This is all fun. This is all just. Just a fun stuff there. Yep.

All right, if you want to learn more, again, as we're wrapping up here, we do have the deep learning AI specialization on Coursera Practical Data Science on the AWS cloud. Feel free to check it out. You will learn a lot more about NLP models really kind of from using the very automated low code, no code tooling that's available. And also a deep dive into the tools that we just briefly showed today around Amazon SageMaker. So check it out also if you're interested, if you're more a reading person, check out the book or sign up for any of our upcoming meetups and check out the code that we shared. It's all available for you. We shared the links here. We do have a huge GitHub repo as well you can poke into if you're interested in poking and learning a little bit more after today's session. Also, if you want to get started with machine learning, we have a free it's a colab like environment hosted jupyter in the cloud called SageMaker Studio Lab. It's completely free, you don't need a credit card, you don't need put in any details, no AWS account required. You can just sign up with an email address and then you can start coding, experimenting, developing ML in the cloud. So check it out with that. Thanks so much for joining us and we'll take questions.

Awesome. Wow, that was really cool. Anje and Chris, thank you so much for all that great information. I learned a lot about ChatGPT and exactly what those thumbs up, thumbs down are doing. And also those demos, Chris, those were rock solid. So we're going to kind of get right into the questions here with the most highly upvoted question we've got. With increasing proliferation of AI tools like ChatGPT, the question of the day, which skills do you think will become slash, stay relevant for a successful career in tech? So why don't we start with you, ATIA Prompt engineering.

I think that the topic that is really kind of a skill that will dominate especially this space, is how can you really kind of intelligently create those prompts and those data sets? And we saw it's not just in the inference part where you play around asking different questions and different types of questions, but how do you really create a training data set in that way and engineer it for the model to pick up on. I think this is really something I personally going to look into more in the future and try to upscale a little bit more and that's definitely going to stick around. So yeah, maybe Chris, if you want to add to that any skills you can think of.

Yeah, that was my answer as well. And you know it's an area we've been exploring. There's that project prompt source maybe did you have that on the slides on shits?

Yes.

Okay. Yeah, by Hugging Face has a really good workspace, sort of online workspace you could play with and shows you, you know, how to actually or some samples. And specifically for the customer reviews data set, they had those that you showed, those prompts that you showed. And what's interesting is you could do things like calculations. You could calculate average, you know, star ratings for a given product. So, so there's I think like the ability to keep an open mind about, you know. Yes, this is like natural language processing but it's not, you know, the word two vec. It's not the old school stuff that we, you know, used to put to use back in the day. This is like, actually has like quite a bit more power. Yeah. Beyond just nlp.

Yeah, very cool. Prompt engineering. You heard it first. Everybody, if you haven't started playing around, now's the time. Okay, so we've got an AWS specific question. I think this one's probably for you, Chris. How reliable is AWS computing in the cloud? Is it stable enough to release a live ML model in aws?

Oh for sure. I mean we have what on to tens of thousands of of customers that are doing just this we inference, you know, model servers. There's a lot of folks focused on this. And that was the other thing I was going to say as well too as we get into the generative use cases and these solutions, they are very compute heavy. Much more so than you know, just calculating linear regression or doing some simple tasks like that, you know, more straightforward machine learning. So we have customers certainly using, you know, multiple GPUs for one prediction. Right. So you know, as batches of like predictions come in, they actually get split up so that multiple GPUs are involved. And so the trick there is trying to saturate those GPUs to get the most out of them. And so, so there's some tuning there. But yeah, I mean this is what we do all day is deploy these models.

And just a fun fact, Amazon.com runs wholly on AWS, right. They're actually doing some quite sophisticated things. Right. You can imagine like there's a team that takes care of the search in Amazon.com, they're working with large language models. They actually did a public talk if you're interested in our last meetup, what they're doing there. So. Yeah, definitely, yeah.

Very, very cool. I like that multiple GPUs for a single inference thing. Very, very cool. That's definitely at the state of the art. So going back down to sort of the beginner level here, Antia, I wonder what you think, is it, is it worth it to learn AWS for beginner data scientists?

Absolutely. I think cloud skills are just one of the things you want to tap into. Right. When you start learning machine learning and experimenting at one point, you just need to have, you know, a kind of an infrastructure that helps you scale your projects. So definitely cloud skills are important. We at aws, we have a lot of content we're putting out helping you to onboard. There's a lot of content right from, from Deep learning AI on Coursera. And yeah, if you're looking into playing around, definitely look into aws. My team, specifically in developer relations, we're working on a lot of, like, learning path content over the next weeks, a month as well. Like, especially helping, you know, beginners to take the first steps and explore the cloud and be successful there.

Perfect, perfect. Yeah.

And.

And I don't know if you had anything to add, Chris, like, from an engineering perspective, if you don't really have cloud, but, you know, you have a lot of coding experience, I don't know if there's one or two things that you'd recommend starting with.

Yeah. Wow. I'm just thinking back to the first time. Well, first time I ever wrote any lines of code for the cloud, I was at Netflix and It was that 2010, 2011. And I just remember the excitement of not having to be limited by my laptop. And if you are working with any, you know, large data set beyond the 16 gigs in your laptop or something, it just gets unbearable. And you could use Spark and you can use other things. But yeah, I mean, I think, you know, quite honestly getting started with, with Spark and just using some of the Spark machine learning that's in there was really, really helpful for me. And yeah, obviously, you know, all the courses offered by Deep Learning. Yeah, yeah, that's a great tip.

Okay, so, you know, big, big question here. All right, so we're gonna really expand the scope of this. How do you see the future of generative AI evolving on AWS in the next 5 to 10 years? Massive timescale AJ what do you think?

I cannot even predict next week. Greg. This space. There's just so much happening, which is super exciting, right? I think every one of us engagement.