Anthropic released Claude with a 100,000-token context window, jumping from 9k and enabling new workflows like spotting changes in 72k-token files, analyzing 85-page documents, or summarizing 6-hour podcast transcripts without embeddings. The expanded window lets you ingest hundreds of pages in a single conversation.
An 85-page document (10-K corporate filing), a ~72,000-token file used for a line-difference check, and a ~58,000-word podcast transcription (~6 hours).
Chapters
If you see any video or read documentation on information retrieval using large language model, they will always talk about dividing your documents into smaller chunks. And the reason is that most of the large language models have a very small context window. But there's a new model where you don't really need to do all of this. The embedding approach could become obsolete. Just to show you why the context windows is so important, I took the text of this Terms of services from Microsoft websites and I'm asking GPT4 which has a context window of around 8,000 tokens, to summarize this for me and let's see what happens. It said the message you submitted was too long. Please reload the conversation and submit something shorter. So it all comes down to the concept of context window, which can be defined as the amount of recent input text that the model considers when generating an output. So most of the models that we have today, their context window ranges from 2,000 tokens to 8,000 tokens. Now 100 tokens is roughly equal to 75 words. There are a couple of exceptions. For example, GPT4 supposedly has a 32,000 tokens version and MosaicML has a 65,000 token version. However, they are not fully tested. But there is a new model which claims to have 100,000 tokens context window. We're going to look at it in a bit. When people talk about large language models, they are mostly talking about ChatGPT from OpenAI. But I think there is another company that also needs attention is this conversation and it's called Anthropic. The current commercial model with the longest context window that is currently available to public is actually CLAUDE from anthropic with 9,000 tokens context window, which is more than GPT4's 8,000 windows context window. And I mean the one available, not 32,000 tokens yet. But today they announced the new version of CLAUDE that has a huge context window of 100,000 tokens. Now let that sink in. Now this is a lot more than the 65,000 context window of MPT7B story writer model from Mosaic ML. Let's look at it. Why it's such a big news. So they went from 9,000 tokens to 100,000 tokens which correspond to 75,000 words. Now that means that you can feed multiple documents of hundreds of pages to Cloud and actually retrieve information from those pages. Now, since it has such a huge context window, the conversation with Cloud can go on for hours or even days and it will still remember all the information. Now for reference of how big hundred thousand tokens is on an average person will take around five hours to read. It will actually take them substantially longer to digest, remember and analyze the information they just read. But for Claude it's just a matter of minutes. For example, here they loaded the entire grid, gets me into Claude, which is around 72,000 tokens, and then modified a single line in the text and asks Claude to spot the difference between the two documents, the original and the modified one. And it did it in 22 seconds. Now personally I don't think it's that great because you could potentially simply compare the text of two documents and can do this, although it's going to take much longer than 22 seconds. But what they can do with it in business application, that's mind blowing. Now I think this is the most interesting bit. It can act as an information retrieval system. So in the recent videos have been covering different methods in which you can create custom chatbots for your own documents. And the approach that I have been showing is simply computing embeddings from the documents and then do information retrieval. But since it has such a huge context window, you can simply feed in the document as a part of the prompt and it will, the model will be able to analyze it and then you can ask questions directly to it. Right? So it might be actually a much better approach than simple computing embeddings doing a vector store followed by semantic search. Now they have an example use case here. So that's Claude acts as a business analyst. In this case they're feeding it 85 page document, 10k form and a corporate filing form, right? So you simply feed in the form and then you can start questioning the data in the form. Cloud will analyze it, digest it, and then you can have a conversation with it. So it will be able to pull relevant information for you. And I think this is great because now you actually have the ability to converse with your own documents. So as I said, this could be a potential replacement to vector embeddings based approaches for document or information retrieval. Now they have one more example which I really like. So Claude act as a code companion. So imagine that you were working on a project and the larger language models do not have any information to your documentation, right? So again you can simply drop your documentation. In this case they're dropping API documentation for LangChain. So initially if you see the person ask it what is LangChain? Right, and it says that I don't really have any information what LangChain is and then you simply drop in the API and developer documentation from LangChain in a form of a PDF file. And now the model simply analyzes it and understands it and then you can start asking questions. So it's your code companion which is pretty brilliant and I think this is going to be significant in a lot of use cases. Now they also highlighted one more use case, smdi. They actually translated almost a six hours long video podcast and so they transcribed it which comes out to be around 58,000 words. And then they use quad for summarization and question answer. What would you do with a 10,000 tokens context window? So here are a few potential use cases. So you can digest, summarize and explain dense documents like financial statements or research papers. Then analyze strategic risk and opportunities for a company based on its annual report. So analyze pros and cons of pieces of legislation. So these are different use cases that you can think about and work on based on your own needs. Now there is, I think one caveat. This is not open to public. So this is specifically for businesses. You can actually request information. They have some information regarding their pricing. Now their pricing are very different from OpenAI and it's mostly it's based on million tokens. So it's like 1.6 dollars per million token and then that's for the prompt and then 5.$5 per million token for completion for their instant model. And then if you want to have a more complex one it goes higher. But this is mostly geared towards business use cases rather than individual use cases. Some of you have been asking me to create a Discord server and I finally did that. It's brand new right now. I hope to create a vibrant community over there, so come join us. I would like this to be a place where we can share ideas openly and discuss the current advancements in machine learning and AI. It's going to be a lot of fun. There are simple rules, so make sure you read them, you follow them. Keep all the conversation related to AI and machine learning. You can also suggest video ideas that you want me to make. So come join over there and hang out with us. Anyways, that's a quick update. Exciting times. Things are changing too fast now. I think the next is going to be 500 or million token models and I think they are not far off. They are going to be here in very near future. If you have any questions, comments, put them in the comment section below. Thanks for watching. See you in the next one.
