Learn to pretrain LLMs from scratch in our new short course with Upstage

I'm delighted to introduce pre training LLMs built in partnership with UpStage and taught by UpStage CL Soong Kim as well as Chief Scientific Officer lucy Park. New LLMs are being announced by researchers and developers all the time, including Larmor, Groq, Solar, Orca and many, many others. This course will show you how such models are created through a process called pre training and walk you through the steps to take if you ever want to train an OM from scratch yourself. You also gain an intuition about the cost of pre training and how it doesn't always cost millions of dollars, especially if you start with one of the smaller existing open source models as opposed to training a massive cutting edge model which still remains very expensive. Of course. Specifically, pre training is the initial phase of supervised learning during which an LLM learns to repeatedly predict the next word fragment, called a token, using vast amounts of text data. The result of pre training is a model called a base model, which is generally pretty good at predicting the next word or generating more text when given an input prompt, but it's not always great at following instructions or behaving in a safe way. That's why we call this pre training. It comes before the fine tuning and alignment training that then results in the instruct or instruction tune variance of models like GPT 3.5 and llama or Gemini. It takes specialized hardware and a huge amount of compute to pre train oms and this makes it expensive. If your use case can be done with an existing model, it will pretty much always be easier and cheaper to use the existing model. But there are some scenarios where you might find yourself having to pre train, for example creating a model with new domain knowledge or developing models that are better at speaking a specific language that's maybe not well represented by the more general models. Further, new strategies and processes are being developed by research labs and companies like Upstage to make pre training more efficient and this is opening up this process to more developers. At Upstage, soon and Lucy have trained their own family of models called solr, using the techniques demonstrated in this course. They also work with many customers to pre train new models for new specific needs and specific use cases. As one example of training algorithmic innovation, Upstage has found that a key technique called dev upscaling which you learn about in this course, can reduce training costs by maybe about 70%. Thanks Andrew, really excited to be here.

We will start the course by presenting some scenarios where pre training your own model is a good option and discuss why fine tuning alone may not be enough to get the performance, you need to form a model. Hopefully this will give you a sense of why so many new pre trained models are being developed when great models like GPT4, Llama, Cloud and Gemini have been already trained.

Next, you'll walk through all the steps required to train your own model, starting with data sourcing, cleaning, preparation, and then moving on to model configuration and weight initialization. We'll finish by showing you how to set up a training run. Along the way, we'll share tips on how to extend the examples from the course to larger scale real world training.

So if you want to better understand how the key step of pre training OHMs work, or if you want to take an existing model and continue the pre training process, or if you even want to compete on LLM leaderboards by pre training your own model from scratch, I hope you sign up for this course.