So Google's Research Just Exposed OpenAI's Secrets (OpenAI o1-Exposed)

Do you remember the new OpenAI01 model where the model thinks before it responds and is now at the level of a PhD? Well, there's new research from Google DeepMind that somewhat breaks down this method and shows that the ways we were scaling LLMs before might not have been the most optimal. Before we dive into the details, let's take a step back and understand the landscape of large language models. Over the past few years, LLMs like GPT4, Claude 3.5 Sonnet and others have become incredibly powerful tools capable of generating human like text, answering complex questions, coding, tutoring, and even engaging in philosophical debates. Their widespread applications have set new benchmarks for AI capabilities. However, there's a catch. As these models become more sophisticated, they also become more resource intensive. Scaling up model parameters, which is essentially making them larger and more complex, requires enormous amounts of compute power. That means higher costs, more energy consumption, and greater latency, especially when you're deploying these models in real time or edge environments. And it's not just the infrastructure. Pre training these massive models demands huge data sets and months of training time. Given these challenges, it's clear that we need to think beyond just making these models bigger. This is where the idea of optimizing test time compute comes in. So what we're going to take a look at is instead of training a model to be a jack of all trades, we by making it larger, what if we could make a smaller model think longer or more effectively during inference? This could revolutionize how we think about deploying AI in practical settings where resources are limited but performance still matters. Test time compute versus Model Scaling to understand this, we first need to define what we mean by test time compute. Test time compute refers to the computational effort used by a model when it's generating outputs, rather than during its training phase. Think of it as the difference between a student studying for an exam and actually taking it. Training is like the study phase, where all the learning happens, while test time computation is like the exam phase, where that knowledge is put to use to answer questions or solve problems. So. So why is test time compute important? Well, as it stands, most large language models like GPT4O or Claude 3.5 Sonnet are designed to be incredibly powerful right out of the gate, which means they need to be big, really big. But here's the catch. Scaling these models to massive sizes has some pretty serious downsides. First, there's the cost. More parameters mean more compute power, which translates to higher costs for both training and inference. And it's not just about the money. It's also about energy consumption. Running these models requires vast amounts of electricity, which isn't exactly great for the environment. Then there's the deployment challenge. Huge models are difficult to deploy, especially in settings where computational resources are limited, like on mobile devices or edge servers. Given these challenges, the question becomes can we get the same or even better performance without scaling up the model itself? That's where optimizing test time compute comes in. By allocating computational resources more efficiently during inference, we can potentially boost a model's performance without needing to make it bigger. The dominant strategy over the past few years has been relatively straightforward. Just make the models bigger. The this involves increasing the number of parameters in a model, which essentially means adding more layers, more neurons, and more connections between them. This method has proven effective. No doubt it's why GPT3, with 175 billion parameters, was significantly more powerful than GPT2 with only 1.5 billion. And it's why even larger models like GPT4O continue to push the boundaries of what's possible. With natural language processing, more parameters generally mean a more capable model than they can understand, more context, generate more coherent and nuanced responses, and even perform better on a range of tasks. However, this bigger is better approach comes with significant costs. Training a model with hundreds of billions of parameters requires massive data sets, sophisticated infrastructure, and months of compute time on thousands of GPUs, not to mention the inference. The actual usage of these models in real world applications also becomes computationally expensive. Every time you ask the model a question or prompt it to generate text, it requires a lot of compute power, which adds up quickly in production environments. This is why companies like OpenAI and Google are looking for smarter ways to achieve high performance without just throwing more compute and data at the problem. Now let's consider the trade offs between these two approaches. Scaling model parameters versus Optimizing test time Computer on one hand, scaling model parameters is a brute force approach. It works, but it's costly, inefficient, and has diminishing returns as models get larger. Imagine a graph showing compute cost on one axis and performance on the other. As you increase model size, the performance gains start to plateau while the costs continue to soar upward. Not a great return on investment. On the other hand, optimizing test time compute offers a more strategic alternative. Instead of relying on massive models, we could deploy smaller, more efficient models that use additional computation selectively during inference to improve their outputs. Think of it like a sprinter, conserving energy until the final stretch and then giving it their all. When it matters most. However, this approach isn't without its own challenges. For example, designing effective strategies to allocate compute during test time is a non trivial task. You need to decide when and how much extra compute to use based on the complexity of the problem at hand. But the potential upside is significant. You can you could achieve comparable performance to a much larger model using less compute, lower costs and reduced energy consumption. What does this all mean in practice? The key takeaway here is that there's a balance to be struck. In some cases, adding more parameters might still be the best approach, particularly for extremely complex tasks where brute force scale is necessary. But in many other cases, especially for less complex tasks, or when deploying models in resource constrained environments, optimizing test time computer could be a game changer. And that's exactly what this DeepMind research is exploring. How to find that optimal balance and what techniques can help us get the most out of every compute cycle. Now that we've set the stage by understanding the problem of test time compute versus model scaling, let's move on to some of the key concepts introduced in this paper. The researchers have developed two main mechanisms to scale up compute during the model's usage phase, what we call test time with without needing to scale up the model itself. The first mechanism is called verifier reward models. Now that might sound a bit technical, so let's simplify it. Imagine you're taking a multiple choice test and after answering a question you have a friend who is a genius in that subject. Check your answer. Your friend doesn't just tell you if the answer is right or wrong, they also help you figure out the steps that led to the right answer. You could then use this feedback to improve your next answer. That's kind of what a verifier reward model does for large language models. So in technical terms, a verifier is a separate model that evaluates or verifies the steps taken by the main language model when it tries to solve a problem. Instead of just generating an output and moving on, the model searches through multiple possible outputs or answers and uses the verifier to find the best one. The verifier acts like a filter, scoring each option based on how good it is and then helping the model choose the best path forward. This process based approach, meaning it evaluates each step in the process, not just the final answer, helps the model become more accurate by ensuring that every part of its reasoning is sound. It's like having a built in quality checker that allows the model to revise and improve its answers dynamically. In practical terms, this means a model doesn't have to be massive to be smart. It just needs a good system to check its work. By incorporating verifier reward models, we can optimize how models use their compute during test time to making them both faster and more accurate without needing to be enormous. The second mechanism is known as adaptive response updating. Think of this like playing a game of 20 questions. If you've ever played, you know that each question you ask changes based on the answers you get. If you find out the answer is a fruit, you stop asking if it's an animal. Similarly, adaptive response updating is about allowing the model to adapt and refine its answers on the fly based on what it learns as it goes. Here's how it works. When the model is asked a challenging question or given a complex task, instead of just spitting out one answer, it revises its response multiple times. Each time it does this, it takes into account what it got right and wrong in the previous attempt. This allows it to zero in on the correct answer more effectively. In more technical terms, this means that the model dynamically adjusts its response distribution at test time. Think of response distribution like a set of possible answers the model might give. By adapting this distribution based on what it's learning in real time, the model, the model can improve its output without needing extra pre training. It's like having the ability to think harder or think smarter when the problem is tough, rather than just rushing to a conclusion. This approach is powerful because it turns the model from a static responder where it only gives you one answer, into a more dynamic thinker capable of adjusting its strategies based on the problem it faces. And again, this can be done without making the model itself bigger, which is a game changer for deploying these models and in practical real world scenarios. Now, let's bring these two concepts together with what the researchers call a compute optimal scaling strategy. Don't worry, it sounds more complex than it is. At its core, Compute optimal scaling is about being smart with how we use computing power. Instead of using a fixed amount of compute for every single problem, this strategy allocates compute resources dynamically based on the difficulty of the task or prompt. So, for example, imagine you're running a marathon. You wouldn't sprint the entire way. You'd pace yourself. You'd run faster in some sections and slow down in others based on the terrain. Similarly, the compute optimal strategy does something like this for models. If the model is given an easy problem, it might not use much compute at all it can just breeze through it. But if the problem is tough, the model will allocate more compute, like running faster in a marathon to think more deeply, use verifier models or make adaptive updates to find the best answer. Now how is this different from fixed computation strategies, which is what most models use today? Well, most traditional models use the same amount of compute power for every task, no matter how easy or hard. It's like running at the same speed for an entire marathon, whether you're going uphill or downhill. Pretty inefficient, right? Compute Optimal scaling, on the other hand, adjusts based on need, making it much more efficient. By using compute adaptively, models can maintain high performance across a variety of tasks, which without needing to be scaled up to gigantic sizes. To truly understand the effectiveness of these new techniques for scaling test time, compute DeepMind's researchers had to put them to the test using real world data. And for this they chose a particularly challenging dataset known as the Math Benchmark. So what is the Math benchmark? Imagine a collection of high school level math problems. Everything from algebra and geometry to calculus and combinatorics. These aren't your standard math problems either. They're specifically designed to test deep reasoning and problem solving skills, which makes them a perfect challenge for large language models. The idea is to see if a model can not only come up with the right answer, but also understand the steps needed to get there. This makes the Math benchmark ideal for experiments focusing on refining answers and verifying steps, which are the core goals of this research. By using this dataset, the researchers could rigorously evaluate how well the proposed methods perform across a range of difficulty levels, relatively straightforward problems to those that require complex multi step reasoning. The choice of this benchmark ensures that the findings are robust and applicable to real world tasks that demand strong logical and analytical skills. Next, let's talk about the models themselves. For this research, the team used Palm 2 models, specifically fine tuned versions known as Palm 2. Now, Palm 2 or Pathways language model is one of Google's cutting edge language models, known for its powerful natural language processing capabilities. It's a great choice for this study because it already has a strong foundation in understanding and generating complex text, which is crucial for solving math problems and verifying reasoning. However, for this research, they didn't just use the off the shelf version of Palm 2. They took things a step further by fine tuning these models specifically for two key revision and verification revision tasks. This involves training the model to iteratively improve its own answers. Think of it like a student going through their homework and correcting mistakes one step at a time. Verification task this is about checking each step in a solution to make sure it's accurate, much like a teacher reviewing a student's work to provide feedback on every part of the process. By Fine Tuning Palm 2 in these specific ways, the researchers created specialized versions of the model that are highly skilled at refining responses and verifying solutions, which are crucial abilities for optimizing test time. Computer now that we've covered the models and datasets, let's dig into the core techniques and approaches that were tested in this research. The research has focused on three main fine tuning revision models, training process reward models, PRMs for search methods and first up we have fine tuning revision models. The goal here was to teach the model how to revise its own answers iteratively. Think of it like teaching a student to self correct their mistakes. But here's the big catch. The model isn't just correcting a single mistake and stopping it's it's trained to go back and keep improving its answer step by step until it gets it right. So how did they do this? The researchers used a process called supervised fine tuning. They created datasets of multi turn rollouts where the model starts with an incorrect answer and iteratively improves it until it gets to the correct one. But there were some challenges. For one, generating high quality training data for this kind of task is tough because the model needs to understand the context of previous answers to make better revisions. To handle this, the researchers sampled multiple possible answers and and then constructed training sequences that combined incorrect and correct answers. This way, the model learns not just to retry, but to revise intelligently using the context of what it got wrong previously. And the result? A model that doesn't just spit out a single answer, but can think through and refine its responses like a careful student tackling a tough math problem. Next we have process reward models and adaptive search methods. PRMs help the model verify each step of its reasoning process by predicting how correct each step is based on previous data without needing human input. This is like solving a puzzle where the model gets automated hints on whether it's on the right path, making the search for the correct answer more efficient and accurate. Instead of waiting until the end to see if it's right or wrong, the model can adjust its steps in real time, similar to having a guide that helps navigate each turn. The research also explores various search methods like Best of N Beam search and Lookahead search, which help the model find the best possible answers by trying different parts. Best of N is like taking multiple shots, picking the best one. Beam search keeps multiple options open and prunes the less promising ones as it goes, and Lookahead Search looks several steps ahead to avoid dead ends. By combining these search methods with prms, the model can dynamically allocate computing power where it's needed most, achieving better results with less computation and potentially outperforming much larger models. This approach allows for smarter, more efficient AI that can handle complex tasks without requiring enormous computational resources. So taking a look at everything, we can see that this strategy, called Compute Optimal Scaling, adapts the amount of computation based on the difficulty of a task. The results show that using this method, models can achieve similar or even better performance while using 4 times less computation compared to traditional methods. In some cases, a smaller model using this strategy can even outperform a model that is 14 times larger. This approach is somewhat similar to OpenAI's recent O1 model release where which also focuses on smarter compute usage. OpenAI's O1 model ranks in the 89th percentile on competitive programming problems, places among the top 500 in the US on a high level math competition, and exceeds human PhD level accuracy on scientific questions. O1 improves with more compute both during training and at test time, so where we look at things are heading. Both OpenAI and DeepMind demonstrate that by optimizing how and where computation is used, whether during learning or when generating answers, AI models can achieve high performance without needing to be excessively large. This allows for more efficient models that perform at or above the level of much bigger ones by being strategic about their computational power. So previously the paradigm where individuals thought that scale is all you need. The vibe seems to be shifting away from this as we look to more efficient ways to get smarter models and I think that looking into the future, this shows us that the future of AI is going to be an explosive one.