LangChain: Run Language Models Locally - Hugging Face Models

Hacking Face provides a large number of open source models that you can use in your own applications. This also include a number of large language models. On the other hand, LangChain is a powerful platform that lets you build apps based on large language models. In this video I want to show you two ways in which you can use Collage language models from Hugging Face. With LangChain, the first option is to access models hosted on Hugging Face through an API. In this case, the Hugging Face hub is the server that has the models and we are going to be making API calls to access the models. Second option is to download all these models locally and then use Hugging Face pipelines to interact with these models. Usage of the models through API call is great because you don't really have to install anything locally. You just need to make API calls and you get the response from the models and you have a wide selection of models to choose from. So for example, they even provide API calls for Stablelm from Stability AI or Dall E from Databricks. On the other hand, in some cases you might want to run these models locally. For example, if you want to fine tune them, then you need to fine tune them locally. Or if you have a powerful enough gpu, then you can also run them locally. In this video I'm going to show you how to interact with these models using both options in Lagchain. So let's look at the Google Colab notebook. I will put a link to this notebook in the description of the video. So there are two types of models that Hugging Face supports. The first one is Text to text generation models. These are sequence to sequence models. They are also called Encoder decoder model and and the second one is Text generation models or Decoder only models. Now if you want to see what type of models are available on Hugging Face, so go to Hugging Face, then click on Models and then under NLP or Natural Language Processing you can look at different types of models that are available. So you have the text generation models and Text to text generation. Let's look at a few text generation models. You have a whole bunch of options here, right? Similarly, for Text to text generation models you have a huge selection to choose from. First we will install all the required packages. In this case we are installing the LangChain Hugging Face hub for the API calls and since we want to do local install, so we are going to be using transformers and then X Ray and Bits and Bytes for processing the data. Now we don't need Sentence Transformer. This is useful when you doing EMBEDDING but in this case I am not going to be discussing the embedding models. Next, you want to get your Hugging Face API token and set an environment variable using that token. If you don't know how to get the token, so go back to your Hugging Face account, click on Settings, then go to Access Tokens. You can actually click on a new token, give it a name and then click Generate Token. That will generate a token for you. And after that you can simply click on this copy token to the clipboard that will copy the token and just come here and paste the token. First let's look at an example of how you can access a model from Hugging Face Hub through the API. So in this case we are going to be using a prompt template. If you don't know what prompt template is, think about it as a simple F string in Python. So basically you pass on the information. Then within the brackets, the curly brackets, you have the question or the prompt that you want to provide to your model, right? So whenever user provide this specific variable that is going to be pasted in here and the rest of the prompt would remain the same. And the prompt template is basically you pass on the template and then what question you're expecting. I have a detailed video on describing prompt templates, so I will put a link to that video. Next we are defining a chain. The chain has two components. First is the prompt. So that is the prompt template that we define designed. And the second part is the large language model that we want to use. So in this case I am using Google Flan D5 Excel. Now if you don't know where this comes from, so let me show you. So go back to Hugging face, click on models. Right? And in this case we want to look at text to text generation models. So that's the first example we're looking at. So here is a list of all available encoded decoded models or sequence to sequence models or text text generation model. All these terms are used interchangeably. So then you select one of them. So let's say let's go here, right? And you simply need to copy this part. So if you click on this icon, this will copy this part. This is a model card. You can actually read about the whole model. For example, in this case it supports multiple languages. It's a sequence to sequence model. And there like one common application of sequence to sequence model is language translation. So that's why it supports multiple languages. Okay, so that's how you can choose a model based on what your requirements are and you can even Run an API call here and see like how the inference is going to look like. So it makes an API call and here is the response that we got now going back to our Google notebook. So we simply create a chain for our large language model. And now in order to run this, you will simply pass a question, right? So that will become part of your prompt template, right? And then you simply call the run function or on your language model chain it will generate a response. So in this case the question was what is the capital of England? And the response is London is the capital of England. And the final answer is London. Now one great thing about accessing hugging face models through the API is you can actually use pretty large models because you're not running them locally. In that case, like you can use the extra extra large flan T5 model and there is going to be no constraint on it. The only constraint is on the server side. So you probably want to have powerful GPU running on your spaces or otherwise it's going to be generating responses based on CPU only. Here's another example. So I asked it, what is Area 51 famous for? And the response is Area 51 is famous for being the location of US government's secret space program. The US government has been constructing a space station in Area 51. The final answer is space station. The quality of response is going to be dependent on the quality of the model that you're using through the hugging face API. Next, let's look at how you can run these models locally using hugging face pipelines. So in this case we are importing hugging face pipeline from LangChain and then from Transformer. We are going to be importing auto tokenizer, auto model for causal LM pipeline and then auto model for sequence to sequence LM for local inference. It's a little involved compared to if you're just using the API call. In this case, first you need a model so that is defined by the model ID. In this case I am using a Flan T5 small model because I don't really have enough VRAM to support it. But for this experiment we're going to choose a very small model. Next you need to have a tokenizer trained specifically for this model. So that's why we have this auto tokenizer. Then from pre trained and then you select the model that you want to use. Every model has its own tokenizer. That's why you need to be careful to initialize the tokenizer specific to that model. Next, depending on the type of model that you choose, whether it's a text to text generation model or text generation model. So it means encoder decoder model or decoder model. You will need to choose your model type. So for this case we're using auto model for sequence to sequence Lm, I.e. encoder decoder model. And then from pre trained you pass on the model ID along with if it's a big model, you probably want to load it in 8 bit, not the full model in 16 or 32 bit. If it's a quantized model, let's say in 4 bits quantized. So you don't really want to do either. And then the device type is auto. In this case it depends on how many GPUs you have if you want to have multiple GPUs and want to use them. So you can provide the map over here. Okay, so you simply initialize your model using this. Next we need to put everything together in a pipeline. So first we need to define the type of pipeline. Is it a text to text generation or text generation? So the model that we are using is to text to text model. That's why the pipeline type is text to text generation. Then you define the model, the corresponding tokenizer and the max length. And then next using the hugging face pipeline from LangChain, we simply create a local large language model. You know, in order to get a prediction from this model, there are two ways you can do it. Either you can simply pass the prompt directly to the local LLM that we created, or you can use a chain. So again I'm using a chain and if you run this, the responses England is the capital of England. The capital of England is London. So the answer is then keep in mind we are running a very small model. That's why the answer is not really coherent. In this last example we will look at how you can run a decoder only model. For this example I am using a GPT2 medium model which is a pretty old model, so it's not really great. But just want to show you how to use a text generation model. Now the tokenizer again needs to be pre trained for this specific model. That's why we pass on the model id. But now the model is auto model for causal lm, not sequence to sequence lm Right. And you pass on the model ID Again in the pipeline we have the text generation now instead of text to text and then the model tokenizer and again the corresponding maximum length that you want. Right. And you simply define the local LLM using hugging face pipeline from Lagchain and for inference. The rest of the prediction process is very similar. You simply need to define chain with a prompt in your local LLM and then you simply run the chain on your prompt. I hope this video is useful. If you have any questions, please comment below. I'll try my best to answer them as usual. If you like this video, consider subscribing to the channel and liking the video. This helps with the algorithm. If you want to learn more about LangChain, I have a crash course on it, so check this next video on LangChain. Thanks for watching. See you in the next one.