New course with Comet: Prompt Engineering for Vision Models

I'm delighted to introduce prompt engineering for vision models built in partnership with Comet ML. Prompt engineering applies not just to text models, but also to vision models that might be able to caption, segment, or generate images. Depending on the vision model, the prompts you use could be text, but could also involve you making annotations on some images, such as maybe marking a few points on the image. So to tell an algorithm what you're looking at. In this course, you prompt meta's, segment, anything model, or sam. Given an image, it can identify the outline of an object in the image if you prompt the model by marking a few points in the image or specify a boundary box containing the object. This is a form of what's called zero shot in context learning, meaning the model does not need to further be trained or fine tuned to recognize that specific object given the prompt, it uses only inference to generate the object boundary. Beyond sam, you also learn about other examples of prompting vision models. I'm delighted to introduce the instructors for this course, Avi Morgan, who's a machine learning engineer, Jacques Verre, who's head of product, and Caleb Kaiser, who's also a machine learning engineer at Comet ll.

Thanks, Andrew. In this course, you will also apply prompt engineering to image generation. For instance, you can provide the text prompt a dragon to the stable diffusion model to generate an image of a dragon, and you'll iterate on that prompt, for instance, a realistic green dragon. To get a different image of a dragon, we'll go through some prompting practices for image generation so that you'll know how to generate the image you want.

You will also prompt the diffusion model to replace a photograph of a cat with the dragon while keeping the rest of the photograph intact. This is called inpainting, in which you combine image segmentation and a diffusion model to replace parts of an image with a generated image. For inpainting, your prompt will be both the text a realistic green dragon and also an outline of a cat that the dragon will replace. Through this, you'll learn how to combine multiple image models, in this case sam, to segment and cut out an object and stable diffusion for generation to carry out more complex image transformations.

In addition to iterating on the prompts, you'll also learn how to tune model hyperparameters to improve image manipulation. One such parameter is the guidance scale, which determines how closely the diffusion model follows the text. Prompt diffusion models work by starting from an image of pure noise and then repeatedly subtracting noise to generate an attractive image. On each step of processing, there's a trade off between following the guidance given by the text prompt versus how free the algorithm is to be creative and the guidance scale hyperparameter determines how heavily to weight text input. A lower guidance scale will allow the model to sample freely from its learned distribution of images, whereas a higher guidance scale will guide the model towards a sample that will more closely matches the text input. In this course, you'll learn some best practices for how to tune this and a few other important hyperparameters.

One other cool thing what if you wanted to personalize the diffusion model to generate not just any dragon, cat or person, but a specific dragon, your specific pet cat or your best friend? You'll use a fine tuning technique called geneboost developed by Google Research to tune the stable diffusion model to associate a text label with a specific object such as your good friend. In the course, you'll use the dreambooth tuning process on the stable diffusion model to associate the word Andrew Ning with just six photographs of Andrew. After fine tuning, you can prompt the model with text such as a Van Gogh painting of Andrew Ning and the model can generate that image.

Yikes.

I had no idea that was the example you're going to use, but lots to learn. I hope you enjoy the course and also go and take out my images and replace them with your own pictures. But please enjoy the course.