SakanaAI New AI AGENT Researcher STUNS The ENITRE INDUSTRY! (Autonomous AI Researcher

So today is a remarkable day for AI development. This announcement is one that I didn't think we would be receiving just yet. But as with AI developments, it's always quicker than you think. Today, Sakana Labs introduced the AI scientist, the world's first AI system for automating scientific research, an open ended discovery. So Sakana Labs is actually a Tokyo based AI startup founded by former Google researchers Leon Jones and David Ha. The company is actually focused on developing AI models inspired by natural systems such as schools of fish and beehives to create flexible, adaptive and economically efficient AI models. And this is of course a rather different approach and today they've come out and said that they've got their new AI scientist. So from ideation to writing code and running experiments and and summarizing results, to writing entire papers and conducting peer review, the AI scientist opens up a new era of AI driven scientific research and accelerated discovery. Now this company aims to utilize numerous smaller models that work collaboratively, akin to swarms in Nature. And Leon Jones, one of the co authors of the very influential 2017 paper Attention is all you need, which introduced the very famous transformer architecture, actually serves as the cto of Sakana AI and David Ha, who previously led research at Stability AI and Google Brain is the CEO and they actually raised $30 million in seed funding. So let's actually get into this announcement because this is one that actually shook the industry. So right here we can see the conceptual illustration of the AI scientist. So this is where they state that the AI scientist first Brainstor forms a set of ideas, then evaluates their novelty. This is basically where it's checking how new these ideas are and if they've been covered before. Next it edits a code base powered by recent advances in automated code generation to implement the novel algorithms. Then the scientist then runs experiments to gather results consisting of both numerical data and visual summaries. And it crafts a scientific report and explaining and contextualizing the results. Then finally the AI scientist generates an automated peer review based on top tier machine learning conference standards. And this review helps refine the current project and informs future generations of open ended ideation. So you can see right here we do have four key steps. The idea generation, which is given a starting template, the AI scientist first brainstorms, of course you've got the experiment template, where given the idea template, the second phase of the AI scientist first executes the proposed experiments, then obtains and produces plots to visualize the results. And it makes a note describing what each plot contains, enabling the Saved figures and experimental notes to provide the information required to write up the paper. And finally, the AI scientists produce a concise and informative write up of its progress in the style of a standard machine learning proceeding in latex. And it uses Semantic Scholar to autonomously find relevant papers to cite. Now, a key aspect of this work is the development of an automated LLM powered reviewer which we'll get into later. But this is capable of evaluating generated papers with near human accuracy. And the generated reviews can be used to either improve the project or as feedback to future generations for open ended ideation. This enables a continuous feedback loop allowing the AI scientists to iteratively improve its research output. And when combined with the most capable LLMs, the AI scientist is capable of producing papers judged by the automated reviewer as a weak except at a top machine learning conference. Now, there's been some really fascinating things about this entire thing that I do want to dive into because the paper is actually pretty long, but there's a few things that you might not know. So one of the things is that the cost effectiveness of the system producing papers with potential conference relevance is actually at $15 per paper. So in order to run this model effectively and actually produce papers, it costs $15 to produce each paper. I don't know about you guys, but I think that is relatively cheap considering the fact that in the future models are going to be getting cheaper overall. And with algorithmic efficiencies and different agentic workflows, we're likely to squeeze out more capabilities of the same level of compute of the LLM, basically saying that look, these things are going to get more efficient and a lot smarter over the time, which means that we could be getting maybe this new benchmark to how much each paper is significantly down. And of course you can see here that this says this highlights its ability to democratize research and accelerate scientific progress. Of course, imagine when we get to a world where these research papers are even a dollar or even a few cents. I mean, how many areas of research are we going to be able to be automating? Now, one of the biggest questions that I personally had were, okay, we now have an automated AI researcher, but were the papers any good? So based on the information provided, the papers generated by the AI scientists contained some potentially novel insights, but overall the quality was mixed. Here's a quick summary on their papers. So some papers did present potentially new ideas or approaches. For example, a paper on diffusion models for low dimensional data showed significant improvements in sample quality and distribution matching. And another paper proposed a novel Dual expert denoising architecture for diffusion models which showed improvement performance. And the papers were generally described as medium quality. And they were comparable to work by early stage machine learning researchers who can execute ideas competently but may lack deep background knowledge. Now, many papers did actually have significant limitations, so such as the lack of thorough theoretical justification for the proposed methods. Of course, we have limited experimental scope, the occasional hallucination and inconsistent quality across different sections of the paper. And the authors pretty much stress that this scientific content should not be taken at face value. They can be used for hints of promising ideas that require follow up and verification by human researchers. So whilst this isn't exactly breakthrough research, these papers could serve as a source of ideas or starting points for human researchers to explore further. And some of these papers achieved scores that exceeded the acceptance threshold for top machine learning conferences according to the automated reviewer system. And the system's ability to generate many papers quickly, such as hundreds in a week, could be valuable for brainstorming research directions, even if these individual papers are aren't publication quality. So whilst the AI scientists demonstrated the ability to generate coherent research papers with some novel ideas, the papers did not generally represent new knowledge ready for publication. But that doesn't mean that this isn't useful research. They were more akin to preliminary research proposals or early stage work that would require more substantial human expert involvement to develop into publishable research. Now, one of the models that they were using in this research was of course Claude Sonnet 3.5. So they state here that we find that Claude Sonnet 3.5 consistently produces the best papers, with a few of them even achieving a score that exceeds the threshold for acceptance at a standard machine learning conference from the automated paper reviewer. And what they also state here, which is a clear indication of what's to come, is that there is no fundamental reason to expect a single model like Sonnet 3.5 to maintain its lead. They state that we anticipate that all Frontier LLMs, including open models will continue to improve and that the competition among the LLMs has led to their commoditization and increased capabilities. They also spoke about using open models. You know, they were like, you know, lower cost and easier to use, but they were worse in quality. One of the things that you might want to see here, for those of you who are looking to get into the prompt engineering side, is in the research paper they actually did have a section where they laid out all of the prompts that they used. You can see right here, one of the idea novelty system prompts was, you are an ambitious AI PhD student who is looking to publish a paper that will contribute significantly to the field. You have an idea and you want to check if it is novel or not, I. E. Overlapping significantly with existing literature or already well explored. You can, you can see that these kind of prompts, and there were like a few of them, showcase how these models were taught to think with regards to their responses. If you do want to see that, it is in the research paper. For those of you that might want to try this out yourself. Now, one of the craziest limitations about this entire thing was the fact that the AI scientist currently doesn't have any vision capabilities, so it's unable to fix visual issues with the paper or read plots, sample the generated plots, sometimes unreadable and tables sometimes exceed the width of the page. The page layout is often suboptimal and adding multimodal foundation models can actually fix this. They also talk about how the AI scientists can incorrectly implement its ideas or make unfair comparisons to baselines leading to misleading results. And it occasionally makes critical errors when writing and evaluating results. For example, struggling to compare the magnitude of two numbers. Now, what's crazy about all of this is that they actually speak about how they've managed to open source this. So for those of you who want to know the complete ins and outs, those of you who want to run this yourself to conduct AI research, you're going to be able to do that. They've given a link to their GitHub repo, they've open sourced the entire thing. So it's going to be fascinating to see what happens here. Honestly didn't expect this announcement today, but with AI development, you truly never know what's coming.