Can AIs Generate Novel Research Ideas? with lead author Chenglei Si

Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas, and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Thornberg. Hello and welcome back to The Cognitive Revolution. Today my guest is Chenglei Si, a PhD student at Stanford who's developing ways to use large language models to automate research. He's lead author of a fascinating new paper that asks the question, can large language models generate novel research ideas? This question has become one of the most important in the entire field, as the ability to generate research ideas that are truly worth pursuing, particularly in the domain of AI, has long been considered a key precursor to recursive self-improvement loops and a possible intelligence explosion. Since the early days of GPT-4, we've seen several notable attempts to create AI research assistants, including projects like CoScientist from Gabe Gomez's group at CMU, which we did a full episode on, and more recently, the AI Scientist paper from Japanese company Sakana AI as well. These systems demonstrated capabilities that would've seemed impossible just a couple years ago, including translating natural language instructions to chemistry protocols and using the Semantic Scholar API to assess research ideas for originality. Nevertheless, the question of whether AI systems can produce high-value research ideas— true eureka moments— has remained the subject of fierce debate. Zheng Lei and his collaborators set out to shine light on this subject with an ambitious study. They asked more than 100 PhD researchers working in AI for new research ideas, incentivizing quality with cash prizes for the best ideas, and then asked Claude to generate new ideas as well. After processing the text in an attempt to create a level playing field for evaluation, they then had expert reviewers rate all of the ideas without knowing their source. The results? The AI-generated ideas scored significantly higher on both novelty and excitement. Now, as with many recent AI results, this paper has become something of a Rorschach test. Those inclined to believe in rapid AI progress see it as a major milestone. While skeptics criticized the methodology— somewhat unfairly in my view, but you can judge for yourself as you listen— and more persuasively in my mind emphasized that this work, which focuses specifically on prompting techniques for language models, may not generalize to the harder sciences. Personally, I agree that we cannot confidently project this one result onto other domains, but I find the experimental setup here to be quite well done, the statistically significant results seem credible, And I think Cheng Lei's individual observations are worth taking very seriously as evidence too. Everyone listening to this show should be familiar with the AI maxim to look at your data. And nobody has spent more time with the raw outputs than Cheng Lei has. So when he reports that 9 of his 10 favorite ideas from this entire project turned out to be AI generated, and that the AI ideas are generally more out of the box and less grounded in existing work than human ideas, I think we would do well to listen. Overall, my feeling right now is that we're at a sort of tipping point where the Claude 3.5 Sonnet and GPT-4o class of models can sometimes, with great effort put into system design and many millions of tokens to burn, sometimes generate meaningful novel research ideas, but not yet in a way that makes frontier research dramatically more accessible or scalable. The next generation of models, starting with the 01 series, seems to me pretty likely to change that. I've been coding with O1 and Cursor a lot lately, and I've been really struck by how effectively O1 can review my entire codebase and my plan for a new feature and return both a critique of my approach and a recommended alternative that is usually genuinely better. As a community, we're still figuring out exactly what the limits to these capabilities are. But if I had to bet, I'd say that systems like Cheng Lei's, powered by O1 and a bigger inference budget, might well produce undeniably needle-moving results quite soon. As always, if you're finding value in the show, we appreciate it when folks take a moment to share it with friends, write us a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. And we welcome your feedback via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this conversation on the emerging reality of AI-automated research with Chenglei Si.

Chenglei Si, PhD student at Stanford researching the automation of research. Welcome to The Cognitive Revolution.

Thank you. Excited to be here.

I'm excited for this conversation. I think you are working on a super interesting frontier, and your recent paper with a provocative question in the title has certainly caught my attention and I think the attention of the field more broadly. So the paper that we're here to really go deep on today is called Can LLMs Generate Novel Research Ideas: A Large-Scale Human Study with 100+ NLP Researchers. I'm looking forward to breaking this down and really getting into the nitty-gritty of the methodology and what we can learn from it and what we should maybe take away from this and expect to see as we go forward into the near future. For starters, want to maybe just give me a little bit of kind of your background? Like, how did you get to focus on this? question of automating research and maybe love to hear too some of the results from the last year or 2 that you have found to be most compelling.

Yeah. So I came from a very NLP background. I was doing NLP research back during my undergrad. And then I came to Stanford last year and then my advisor is Jian Tatsu. And I always wanted to do something different for my PhD. And back then the big background in the whole NLP field is that large language models are getting really good and then We saw some really impressive performance on things like reasoning, math, and coding and stuff like that. And we feel like it probably could give opportunities to some very new applications. And, you know, as someone who has done research for many years, I feel like being able to automate research feels like something very different and very exciting. And importantly, I think it will have huge practical implications to the entire field. And I just got excited about this agenda right away. And I'm lucky that both of my advisors are Very excited about this whole agenda as well. So we actually got started on this whole thing pretty early, I think roughly 1 year ago. This whole project actually took around a year to finish.

Yeah, interesting. And I trust that the human parts of that took the most legwork and took the most time. That is often the striking difference these days.

Yeah, building the agent itself took us like 2 or 3 months, and then finding all the people and doing the actual human study took us like 8 months or so.

Yeah.

Interesting.

So before we dive into the details of your paper, what other projects, they don't necessarily have to be, could be academic papers, but it could also be open source projects, have captured your imagination and kind of stand out to you as the notable step change improvements on the road to this destination of automated research?

Yeah. So I think to me personally, there's this line of work on self-improving AI. There has been discussion on whether, like, as AI keeps getting better, there will be a point where AI can be even smarter than humans. At that point, how do we possibly keep improving AI if AI is getting smarter than us humans? Then it gives rise to this idea that maybe we can let the smarter AI to propose new ideas and improve itself. There has been some preliminary attempts on that. There has been some work done by folks here at Stanford like Eric Zuckerman, and that's one inspiration for this line of work. But of course, there's also another line of work on building tools that can assist researchers. A lot of colleagues from HCI, folks at Semantic Scholar, a bunch of other people have been doing this for many years. So research assistant tools, creative support tools is another inspiration for this, but I think we really tweak a very different approach in the sense that we want to do something that sounds even more aggressive in the sense that we want to see if we can possibly automate the entire research process rather than just build some assistant tools there.

Yeah, I think you've zeroed in on a really important question. I've looked, for what it's worth, at— this is kind of an area I try to keep up with because It feels to me like if we are going to have an intelligence explosion, this is kind of the form that it's going to take, right? It's going to be— and I'm by no means alone in this analysis or original in coming up with this analysis, but basically it seems like the expectation from the field is at some point we're going to start to see AI systems that can make meaningful contributions to ML research.

Mm-hmm.

And if ML research becomes 10x, 100x, 1,000x more productive because we can sort of spin up AI ML researchers and they can actually take real bites out of big problems, then it's really hard to say how good the overall resulting models could get. And that could take us into a different regime. So it feels like there's this sort of tipping point Possibly where if we can't get over that hump, then things maybe stay somewhat normal. And if we do get over that hump, then kind of all bets are off and the future starts to look quite unpredictable and potentially quite weird. I guess for starters, is that your mental model of why this matters so much in the first place?

Yes. You mentioned the word intelligence explosion. So I've read the whole situational awareness thing that Leopold wrote. My advisor, Tatsu, was warning me that using intelligence explosion would sound too aggressive for an academic setting. But I think on this podcast, it's probably fine. So yes, roughly that was one exciting vision that I kind of wanted to pursue. There's a very interesting data point. If you remember, there's a figure there where Liu Po was saying around the year of 2028, we would be able to get automated Alec Redford. So Alec was like the lead author on the GPT paper. So if there was a point where we are able to automate Alec Redford and we scale this up. That means we'll have massive increase in the way AI can improve itself. That'll be insane. But we want to, you know, maybe take a step back. Automating Alec is a pretty difficult goal. There's only one Alec in the whole AI field. So maybe we want to really see the current generation of language models, how good they are right now. We want to compare that with a more reasonable baseline. So maybe we want to see how do they compare with, let's say, average PhD students. I want to— offer a concrete benchmarking point to gauge the progress that we have made so far. So that was the motivation for the whole thing.

Yeah. Okay, cool. So just to— and by the way, Alec Radford has been making the rounds on Twitter just today, or in the last 24 hours. We're recording on October 1st, and he's been a popular answer to a question that somebody posed, which was, who is the most important or influential scientist that is not very well known, and people are piling on to say Alec Radford is the answer to that question. That is indeed an extremely high bar for automation. To just further tee up the specific question that you're asking with this research around using LLMs to generate novel research ideas, I think it is worth taking a minute to just review a couple of my personal favourite highlights from the research over the last year, because I do think that has been an area where it hasn't quite been there yet. It feels like we've sort of been always a little bit short. One that came out very quickly after GPT-4 was released, and we actually had the lead author, Gabe Gomez, on the show for a prior episode, was called CoScientist, and they put together a really nice early, but I think has stood the test of time, agent framework that allowed a user to basically say something like, please synthesise aspirin for me. And then the system would do everything needed to synthesise aspirin, including looking up how do you synthesise aspirin, translating that to actual instructions that could be sent over to an automated lab, were integrating this with Emerald Cloud Lab at Carnegie Mellon, and you can actually program a real physical lab via an API to carry out the step-by-step protocol that you want to run. And that was amazing, right? I mean, to go from a natural language sentence all the way through to a purified powder getting spit out the other end of a system like that, I was like, man, this is major result. And obviously that's why I wanted to have him on the show back then. But the one thing that I would say at that time was it did not seem to me like the AI was actually coming up with meaningful ideas. So in my review of that paper, I called it— the question was like, is this text-to-science or text-to-experiment? And I was kind of like, yeah, I think it's more text-to-protocol because the AI at that time wasn't really generating the idea so much as taking the idea and figuring out how to break that down into steps and execute on it. But that was nevertheless a big milestone. I don't know if you have any comment on that one before we move on, but I'll highlight a couple other ones and then we'll get into yours.

Sure. I think the idea is cool for sure. There have been similar attempts, maybe in other areas. Our friends at Sakana AI, for example, have an AI synthesis paper that also tries to improve over the NGINX agent system. I think the high-level research agenda is exciting for sure, but I think the tricky thing is the evaluation part is really difficult, especially in scientific research. How do you possibly evaluate that research? Even for academic settings, we do peer review. That isn't the perfect solution either. So I think all this really boils down to the technical challenge of doing evaluation, and that is really the crux that we're trying to focus in this paper here.

Yeah, the AI Scientist was another one that was on my list, and I thought that one was a notable step forward. I mean, all these things have their critiques, and I feel it's like a bit of a Rorschach test that tells you often as much about the critic of the paper, whether positive or negative in their assessment, as it does about the research in some ways, because there is just so much room for interpretation still of these results. I find that to be fascinating. The AI Scientist to me, though, did take a step forward in that it introduced a mechanism for generating a bunch of ideas and also trying to identify which of those generated ideas were the most novel ideas. And they used the Semantic Scholar API to help with that. So they would have a bunch of ideas generated, then go do a search against the Semantic Scholar Corpus of information to pull back as much related stuff as possible. And then they would sort of try to see, like, has this already been done? To what degree does this seem like it's already represented in the literature versus if not, then that's more likely to be a meaningfully novel idea. And that would be the kind of idea that they would want to prioritise. The reviews of the actual AI-written papers that came out of that didn't seem like they were necessarily going to get admitted to top conferences. And even in their own assessment, the Sakana team basically said this would be like the work of an early grad student that definitely needs more advising from somebody who has better zoomed out context on the field. But I did still think that was a notable mechanism that seemed to kind of show the way to where this was going. I mean, one of the ways I describe this is Do we get eureka moments from AI systems or not? And initially with GPT-4, I used to say no eureka moments. That was kind of my headline for GPT-4 performance. And then we saw the Eureka paper, and that was out of Nvidia, Jim Fan and his group. And there they used GPT-4 to write reward functions for robot hand reinforcement learning. And they saw that GPT-4 outperformed human authors of these reward functions. And I always am fascinated by that because I'm like, geez, it takes some real know-how to even write a reward function in the first place, right? This is not something that just any old person off the street can do. You're engaging here with a certain level of expertise as a baseline and still seeing GPT-4 perform above that. Not exactly. I mean, it's sort of like a hypothesis. A reward function is how well is it going to work? You're making a guess as to what would work. So I feel like that was kind of moving in that direction as well. And the fact that it did outperform humans in something that was objectively measurable was quite notable to me. Any other thoughts on that one? Next step is yours.

My definition of a eureka moment for automating research line of work would be if one day you tell me, you have an engineering agent that generated the idea and automatically executed the idea into a full project, and then you submit that project to a top conference, and that project actually gets, let's say, a best paper award, then I would say that indicates a eureka moment for me because that means that piece of project is actually way better than the average that you get at the so-called top conferences, right? I think the AI Science paper from Sakana that you mentioned, I think the agent system That's a nice prototype there, but the outcome, first of all, even using their own AI reviewer, the score isn't really reaching the bar for neurists, for example. And then second of all, how much should we really trust the automatic reviewer? That's a question we actually touched on in the paper as well. We did some analysis and find that automatic reviewer at this point isn't really ready yet in the sense that if you measure the correlation between LM reviewer and the actual human expert reviewer, correlation is pretty low at the point. So maybe we shouldn't trust that much about automatic reviewer at this point. So those are the comments about the whole AI scientist thing.

Hey, we'll continue our interview in a moment after a word from our sponsors. Yeah, I like your comment about— first of all, you know, it's always one of my refrains is like, compared to what? So it's really important that we compare these things to the human alternative, if there is one or whatever, is kind of how is it being done now? I think the other thing that people often forget is that the current thing is not perfect. Often it's actually quite a bit worse than people imagine it to be. But I think you had a good kind of baseline in the paper on comparing to the inter-rater reliability of actual human judges for the conferences. So somebody might say, the skeptic of the human performance, which is often a role I play, might say, well, how consistent are the human raters and is this any worse than that? And the answer here is, you can fill in a little bit of the details, but the human raters actually do agree more than the AI raters agree. There is still something measurably not accomplished with the AI paper evaluator.

Right. So at this point, it's true that human reviewers at conferences tend to have higher agreement than the agreement between AI reviewer and the human reviewer. So I think that's one evidence that current language models are not ready for automating the reviewing part yet. But even if, let's say, one day language models keep getting better, the agreement somehow gets higher, I think it's still questionable whether we should fully trust the automatic reviewing on such a subjective and pretty high-stake task, I would say. If you look at the actual human agreement, you realize reviewing is really a very subjective task in the sense that Even the expert reviewers at top conferences like ICLR or NeurIPS, their agreement is still pretty low in general. So I think it really highlights the challenge of evaluating research ideas in general. Yeah.

So give us— I'm looking at the paper. This is Table 11 in your paper. So you basically are comparing 4 main results. One is— and I guess, first of all, just to be very clear, the question here is accept, not accept? Does it boil down to that binary judgment?

The question is actually an even simpler setting than that in the sense that we basically try to get the lower 50% of papers and then we combine that into a binary judgment of something like accept or not accept. And then we measure the decision agreement between 2 sets of reviewers.

Gotcha.

So random is 50% in this setup. The NeurIPS judges Only got up to 66%. So they're agreeing not 1 out of 2, but 2 out of 3 times. The ICLR judges got up to 71.9%. So they're getting close to 3 out of 4, but still not quite there. And then your implementation gets to 56% agreement with the human judges. So it is an incredibly noisy thing, but there is still like a meaningful— Delta between what the AI system can do and what the humans could do when it comes to this specific task of judging the papers.

Right.

Okay, cool. Well, let's go back to the hypothesis generation. Give us the setup for how you sought to answer this question. You already said it took months to build the agent, a lot more months to build the dataset from humans. But give us the whole setup of how you set out to answer this question of, can LLMs generate novel research ideas?

Right. So given the whole question, I think there are 2 high-level principles that we want to achieve. The first thing is we want to set up a proper human baseline. Of course, without the human baseline, you can still measure some numbers on some metrics, but people might ask, what do those numbers really mean? We want a concrete benchmarking point by comparing to a reasonable baseline there. And we feel like, you know, PhD students, they reasonably represent the experts in this task. So we want to compare to that for sure. And then the second thing is also we want to set up proper design protocol to avoid any potential confounders. And we want to have expert reviewers to do all the judgment, because I mentioned a bunch of caveats of using language models to review. So all the reviewing is done by expert researchers in this case. Right. The high-low experiment design is then, want to have one branch where all the ideas are generated by AI and another branch where all the ideas are generated by human researchers. And want to have a blind review where expert researchers come in and, you know, score all the ideas without knowing which one's which. And then we compare the scores. But in the actual implementation, we're also adding a third condition, which we call the AI ideas plus human re-ranking. The consideration of that is that we have a built-in ranking part in the idea generation agent where the agent will generate thousands of ideas on each topic. And then the agent will automatically select the best ones. It's a little bit similar to what you mentioned in the Sakhana paper. So after this ranking, we get the best ideas generated by AI as ranked by the AI itself. Those ideas will be the condition of our AI ideas. And then we realized that the AI ranking isn't really perfect yet. So it's possible that among all the couple thousand ideas generated by AI, There could be some really good ones, but they're not being ranked as top by the AI because the re-ranker is not working perfectly yet. So we want to have a better estimate of the upper bound. We want to find the best ideas being generated by AI, and that's where we manually actually look at all the generated ideas and select the best ones that we consider as the best. And that's what we call AI ideas plus human re-ranking. So in that condition, all the ideas are still generated by the AI agent, But the re-ranking is done by humans. So that is the third condition. So that's the experiment setup. We'll have one review, we'll assign ideas to all the expert researchers, we'll score ideas along different metrics including novelty, excitement, feasibility, and effectiveness, and then we'll compare the scores of different conditions. And that's the whole design of the experiment.

Gotcha.

So let me just say that back to you and make sure I've got it all right.

Yes.

So there's 2 sources of ideas, obviously, the humans and the AIs. You pay the humans for their ideas. They were paid $300 for participating. And if I understand correctly, there was also, I think, a couple of $1,000 prizes for ideas that were judged to be the best among the humans.

Yes.

So the participants are mostly Grad students, PhD students in machine learning, right? With a focus on natural language.

Mostly LLMs. Yeah.

So, yeah, I think that's notable for a couple of reasons. Obviously, certain amount of expertise there, that's definitely not nothing. And the $1,000 is not insignificant on the scale of a grad student salary. So presumably people would at least be somewhat motivated to give a good idea and try to pick up the extra $1,000. Then on the AI side, I think there's some really interesting implementation details in the paper. Maybe you can give us a little bit more on how you're prompting the AI for ideas. I know that because you're not just asking a simple question like, hey, can you give me some NLP research ideas? You're actually bringing a sort of kernel set of ideas to it for it to expand on. Maybe give us a little bit more on that.

There are a couple things that we put in the prompt when trying to generate ideas, right? The first thing is we retrieve related papers given the research topic, and then we'll put all those related papers in the prompt to give some inspiration there. Another thing is we give some demo examples. We actually manually constructed 6 demo examples based on existing papers. So those will be exemplar papers that we put in the prompt. And then all examples will follow the same format. So we define a template for how the idea should be written up. So the template roughly includes things like there's a motivation, there's a problem setup, there's a proposed method, and there's a step-by-step experiment plan. And then there's also task examples, et cetera, et cetera. So it's a very detailed template for writing the idea. So we provide the template to the Agent as well, and then in the prompt we also include all those demo examples, include all those related work, and then we generate ideas in batches because we want to you know massively overgenerate number of ideas and then select the best ones. There's a limit to how many tokens you can possibly generate in one generation, so we actually generate it in batches. So we generate five ideas in a batch, and then we keep generating. And one issue with this thing is that it's possible that. The agent would start repeating itself by generating the same ideas over and over again. And that'll be bad for us because we will want a large set of very diverse ideas so that we can possibly find, you know, better ones there. So one thing we did is we will actually add all the previous generated ideas' titles into the prompt and then ask the model, you shouldn't be repeating what's already being generated. So that's one simple attempt on reducing repetition there. But as noted in the paper, there's still a lot of repetition in the idea generation thing. We did some analysis on that. We can touch more on that later. But on a high level, that's the idea generation part. So we have the retrieval part, we have the template, we have the demo examples there. And then after generating all the examples, we have the ranking thing to select the best ideas there.

And this is with Claude 3.5 Sonnet as the main model?

Yes.

Did you, is there a, any sort of guidance you could offer on like how much, I presume you chose that because you found it to be best. Is there like a measurement versus like GPT-4o or versus Gemini 1.5 that, you know, you could kind of help us calibrate how much better 3.5 Sonnet is?

Right. So we were initially actually using GPT-4o, I think back then. And then I heard a bunch of people in the lab are saying, you know, Claude 3.5 is really good on all the those science and reasoning and coding tasks. So I tried Claude 3.5. It's difficult to give a quantitative number on this, but we look at the 2 sets of ideas generated by GPT-4 and also Claude 3.5, and intuitively based on our judgment, it feels like Claude 3.5 does have some advantage in the task of generating ideas. That's why we stayed with Claude 3.5 for the entire pipeline. I know OpenAI just released O1. And I heard OI is supposed to be really good on those reasoning-intensive tasks. It's possible OI might be even better. We haven't done any benchmarking on that yet, but it's on my list.

Yeah, I'll be very interested to see the result of that. Okay, so going back to the core of the setup, examples are put in, structure of the format of what an idea should look like is included for the AI to work from. It's also given the ideas that it previously generated and told not to repeat. It nevertheless does repeat quite a bit, but enough new ideas come out that you can tell us about the ratio. If I remember correctly, you generated like 4,000 ideas but found that only a couple hundred were truly original.

Yeah, around 200 ideas are non-duplicates. So basically we use semantic similarity to judge whether a pair of ideas are too similar to each other, and we will filter this out. So after you do this filtering, you get like 95% of the ideas out because they're duplicates. And we actually have a curve in the paper showing that in initial batches, most of our ideas will be novel, but as you generate more and more, in the later batches, most of our ideas will be just repeating previous ideas. So if you calculate the total number of non-duplicate ideas, The number starts to saturate in the later batches and eventually it hovers around 200 or so among the 4,000 generations.

Hey, we'll continue our interview in a moment after a word from our sponsors.

So how do you think about that? I guess for one thing, it looks like a logarithmic curve, right? It looks like— but a logarithmic curve doesn't— it's not a flat line. And maybe you— I don't know if you push this any further, but Beyond 4,000 ideas, should we still expect to get some original ideas, albeit increasingly infrequently? Or do you think that you kind of hit a point where it's like, it's just not going to come up with anything new anymore and it kind of is what it is beyond a certain point? That seems important to me because we can obviously pump a lot of tokens through these models and it's like, even if it is logarithmic, You know, with a budget you could get there, but if it truly kind of flatlines and just doesn't really do anything new beyond a certain point, then that's kind of a qualitatively different situation. So what do you think the reality is?

Yeah, so this is a tricky question because there are 2 factors here. One possibility is that given research topic, maybe the nature of the problem space is that there are only a finite number of ideas possible within that space. Especially if you give a very specific research question, that's possible. Another possibility is that maybe the problem space is much bigger, possibly maybe even infinite, but the model has a very low upper bound on the number of different ideas it can generate. Then this will be a huge issue, as you pointed out. But I think this is still open for debate in the sense that we actually didn't push that hard on the generation part in the sense that what we did to avoid repetition is very simple. It's just pending the already generated ideas and asking the model to not repeat. I can imagine smarter ways of doing this. I can imagine ways that you can possibly, let's say, fine-tune the model to increase diversity. You can possibly change the decoding methods in some smart way to increase diversity. If you tried all those things really hard and still see an upper bound, then I think it's evidence that maybe the base model, it just has an upper bound on the idea diversity. And that would be a huge issue that I would agree with. So yeah, that's my take on this.

There's also the question too of just like how, you know, the real eureka moment is like zooming to some place in a vast search space that would've taken forever to find through brute force search that actually happens to be like a a really good idea. And that is obviously hard. So not many humans can do it, right? And humans tend to only do it in very narrow domains where they have deep expertise. So this is a hard thing to do with the AI ideas. One wonders, and I guess this is kind of where the evaluation will come in, but one wonders, like, okay, let's say you turn up the temperature or apply these different decoding methods or jam all the levers that you have to try to increase diversity.

Right.

You can presumably increase diversity. But how does that diversity relate to quality? And do we have any way to determine other than the— I think your approach here of actually using humans to do the evaluation is really good for creating a grounded baseline that people can hopefully have some level of agreement on. But it's obviously really hard for— I would assume this is probably why you haven't been able to rerun the thing with '01 yet, right? Because you could generate a lot of ideas really quickly, but now it's still going to take a lot of time and effort to do the evaluations. So when you do turn up the diversity, do you have an intuition or do we have any metrics for sort of how pushing these temperature and other parameters up impacts the quality of the presumably more diverse ideas generated?

Yeah. So I think one analogy here is recent works on inference scaling. So for example, on stuff like code generation, there are experiments showing that if you sample more diverse solutions, the success rate, like the passing rate of the solutions, increases. And there's a nice scaling curve there where the more you generate, the more likely that one of your generations will be correct. And then your task will be essentially to find the correct ones among other generations. So that's the analogy there. We didn't do the exact same thing on idea generation in the sense that, like you said, unlike code generation where you can verify the correctness of the solution, and that means that you can actually know the so-called success rate in all your generations. In our case, the evaluation is really tricky. But still, I think we can build proxy rankers like we did in the paper. The ranker may be not perfect. In fact, it's definitely not perfect as we showed in the paper. But still, as long as there's some positive correlation that the ranker is able to tell really bad papers from really good papers, then I think it's reasonable to expect that if I scale up the diversity and the number of ideas that I generate, this ranker will possibly find better ideas. And among all the top-ranked ideas, I might expect a quality increase as I generate more ideas.

Yeah, that reminds me quite a bit of some work that I've done on using a number of different kinds of models to try to evaluate the aesthetic appeal of images. And we found a similar thing there where the human-to-human consistency of judgment was fairly low. The human-to-AI agreement was even lower, but not zero. It was better than random. And the general guidelines was the general kind of takeaway of that work was like, you can definitely tell the difference between the best and the worst. And there's clear signal there, but especially in the middle, right? You really can't, you know, the fact that one was rated higher than the other is not very meaningful, right? So you have like a, you know, a general direction, but at any given point in the list, like, it's definitely up to interpretation still.

Although I'll add one comment here that at this point it does matter that we want to have really good ranker or evaluator to find the best ideas, but You know, if we think bigger and we think about the big picture of automating research, in a next stage, it's possible that we might start working on automating the execution part of things where we might build agents that can automatically execute ideas into experiments. That means the agent will tell us, I'll implement the method, I'll test on the specified datasets, and I'll tell you how well it does as compared to the baselines. And that means I can actually verify whether the proposed idea works or not. And that means for all the generated ideas, if I'm willing to spend all that inference compute money, I can just implement all those ideas automatically, and then I can automatically verify each idea. So in that sense, it turns into a setting almost like code generation where I can actually check the correctness of each possible idea. I think in that setting, we'll have some nice inference scaling there. Yeah.

That's a great point, first of all. And it reminds me of the AI Scientist paper where they reported, I think it was like in the low tens of dollars per paper generated. And so it was like, you know, okay, sure. They're not that great. But if 1 in 1,000 is great, then it's like clearly economically competitive.

Exactly.

And the prices are obviously falling quickly. So it seems like that. I remember talking to somebody once at one of the leading frontier model developer companies, one of the big 3, and the person said, we find over and over again that people just don't have an intuition for what happens when you're willing to spend a lot on inference compute. Obviously, OpenAI has kind of now productized this to a degree with the L1 family of models. But even before that, just people try one, they try 2, they try 3. It's just not intuitive to think, like, what if we tried 1,000 or 1 million? And that really can change the game. So I think that is quite important. Okay, let's go back. Let me just reset the stage one more time because I know I keep going on all these rabbit hole digressions. So you pay grad students for ideas, you give them the promise of a prize, you run a reasonably high volume of idea generation, you dedupe the ideas. Is that with an embeddings approach, or what's the deduping method? That's embeddings?

Yeah.

Okay, then there's 2 ways that you can rank the ideas to try to bring the best AI ideas to the top. Those are having the AI do it Directly. I believe you use Claude with pairwise head-to-head kind of like tournament style for that.

Yes.

Yeah.

The pairwise comparison thing is what we found to work much better than directly asking the model to give a score, let's say on a scale of 1 to 10.

Yeah, that's a really good tip and something I think that is a practical takeaway for all sorts of AI application developers in the audience. And I actually think there was a really good This is a very in-the-weeds detail, but what I just said about tournament style is not quite right, actually. I had previously been thinking of doing something like this in a project, and I was thinking like almost setting it up like an NCAA March Madness bracket where I would have the field gradually narrowed. But of course, that has the problem of like, well, what if your best 2 ideas meet in the first round? And so then you're like, well, maybe I'll have a double elimination tournament, but then you could still have similar problems.

Mm-hmm.

So I think what you did was just take random pairings and have each idea paired with 5 other ideas and then just sort them by their win-loss record, right? So 5-0 would be at the top, 4-1 would be next.

Is that right? So instead of random pairing, what we did is basically we pair each idea with another idea with the closest accumulated score so far. So the ideas that want to, let's say, We want to pair the top-scoring ideas so far with other top-scoring ideas, for example. And it's— you're right that it's different from typical tournament in the sense that we don't have elimination problems. Because otherwise you run into problems where 2 really good ideas meet in the first round and one of them got eliminated and they would only get a score of 0 in the end. And that would be really bad. Yeah.

Okay, cool. So that's even one level smarter than I understood. Pairing ideas. with others that have the same score so far to try to put the most comparable ideas together. And yeah, you can sort of, it's almost like bubble sort style, or at least that's the visual that comes to mind for me. Okay. So that's cool. That works. But you did find that even better is human re-ranking. Anything we should know about the human re-ranking process?

It takes forever. I actually manually looked at all the ideas generated and then I didn't look at the AI ranking. I just, you know, did my own ranking of it. And then I compared my own ranking with the AI ranking and it's somewhat surprising that there are a couple ideas that I thought would be clearly the top ideas that are not really captured by AI as a top idea. So I think the discrepancy here is interesting. Okay.

So you personally, individually are the human re-ranker. Is that right?

Right. Right. Yeah.

Okay, interesting. So then I think the next thing is a pretty interesting detail, and I guess we should also set the stage. I'll put this in the intro as well so people have this context. The field of study that we are working on here is essentially prompt engineering. It is how can we get the best performance from language models via different prompting techniques? So that's just background baseline. all the ideas all across the board are within that domain. Now, once you have the human ideas, the AI ideas ranked by the AI itself, the AI ideas ranked by you, then you put them through a sort of anonymizer type of thing, right? You basically have the AI rewrite the idea. And if I understand correctly, this was done to try to create an even playing field for, with respect to the human judgment. I guess the worry was the human readers will likely be able to tell the difference between a human-written idea and an AI-written idea.

Right.

Even though we know that AI large language model detectors in general don't work. And if you're a teacher and you're in a classroom, you shouldn't put too much stock in them. With that disclaimer, it is reasonable to expect that especially people that work in the field would have an intuition for like, yeah, this kind of reading, right? How often is the word delve being used, et cetera. So you put just the human ideas through this rewriter, or did you put the human and the idea? Everything in the idea. Okay.

Yeah.

So tell us, give us that bit of the setup.

So we pretty much specified the desired writing style and format of the idea. We tried to have the writing be more academic, and then we specify every single component that the idea should have. And then that writing agent essentially does some minor paraphrase to all the different ideas. We said very explicitly in the prompt that you can only do writing style edits, definitely not editing any of the original content. So most of the edits are basically changing certain word choices. And I have to say, I think this is very important in the sense that if you look at the original ideas submitted by humans, Some of the word choices can be very obvious that this is written by humans. Like some of the sentences tend to, for example, some sentences are not grammatical and that pretty much is a giveaway. And some people like to use more verbose writing styles, almost like they're talking to you, more engaging writing styles. And that just feels unlikely as from AI. Those things don't really affect the content that much, but we don't want this to bias people that, let's say for example, some people just like humans better than maybe if they figure out that this is written by humans, just have a more engaged writing style, and then maybe they would favor those ideas. We want to avoid these cases. So we do the style transfer thing for every single idea, including both AI ideas and human ideas. And the hope is that we would avoid all those confounders there.

So I think your paper definitely did generate quite a bit of discussion online. This was One of the points of criticism that I think people most focused in on when they were like trying to find a reason not to believe the result. Maybe we should tell the result and then we can kind of come back and dissect the criticism.

Sure.

The final step after doing this rewriting is then to send all these ideas. And these are now like, what, like 500 words? It's like a page or 2 pages, like a reasonably well fleshed out—

Around 1,000 words. Around 1,000 words.

Yeah. Okay. So not an insignificant sketch of the idea. And these are then sent to human evaluators who have to score them, and they are scored on 4 different dimensions, right? They are novelty, excitement, feasibility, effectiveness, and I guess overall kind of just aggregates those previous 4 scores, right?

So we will ask a separate overall score, but that's the idea.

Okay, gotcha. So 5 different dimensions, and the humans give each paper 5 number scores, 1 to 10. 1 the worst, 10 the best. How original is this idea? That's novelty. How exciting? This excitement obviously is kind of subjective, but that's basically the, like, does this feel like a really good idea? Feasibility, obviously being— maybe you can give a little more color on kind of the guidance that you gave for that. Yeah.

So So for feasibility, we are really asking whether this idea can actually be executed. So for example, some of the not feasible ideas would be things like something that requires tons of compute, like you want to train GPT-5, that doesn't feel that feasible. Or you want extensive human effort, like you want to hand-level a huge training set, that won't be that feasible, or things like that. Or you want to fine-tune a closed model, like you want to fine-tune some generalized model such as doesn't really have fine-tuning success, for example. So things like that. So that's what feasibility is. And that's different from effectiveness in the sense that in effectiveness, we're asking the question of, given the proposed method, do you think the proposed method will actually work better than existing baselines? So that's the difference here. Gotcha.

So novelty, excitement, feasibility, and effectiveness. Feasibility is like, don't go too crazy on the budget. I wonder what would happen if you put this paper, your paper, Through the evaluation process, how do you think you would score on these different—

Not innovative.

Yeah, you'll never get the budget for it. The total budget on this is, is what? And how does it break down, by the way? I mean, I'd be interested to know kind of total spend and also, yeah, just what percentage of that went to the humans versus what percent went to the AIs.

Yeah. The payment for humans in total is about $30K and then for AI. So generating 4,000 ideas will take a couple thousand dollars. And then we did that for 7 different topics. So there's some money there. And then there was also some development in the early stage where we, you know, prototype a bunch of different agent systems. So in total, I would say maybe the money that we spent on API credits will be slightly more than what we actually paid humans in that case. So that's the money breakdown there.

Interesting. Okay.

That's... Kind of surprising.

Maybe let's dig into that just a little bit more. So what is consuming the majority of the token budget? Because I did a little back of the envelope math and I came out to a lower estimate of what I thought you had probably spent on the APIs.

Yeah. The problem with the ideation part is, first of all, we have to generate all the original ideas first, and then we do the duplication. That means if you want to get, let's say, 200 ideas on a topic, you have to actually generate 4,000 ideas. And usually we're doing this almost like scaling law thing where we actually overgenerate more than 4,000 ideas. We try to test the limit there. So we actually generated a whole bunch of like 8K or even 10K ideas on each topic and did some analysis there. So there are some hyperparameter tuning there. So that way you spend a bunch of tokens. And also you do realize the fact that the prompt is very long, so the input tokens is a lot, and then it gets longer and longer as we prepend all the previous general ideas in the prompt. Another thing is in the final step, we also have this process of a novelty filter. This part is actually not in the main paper, it's in the appendix. The novelty filtering is similar to what people have done in the past in the sense that for each generated idea, we will actually retrieve all the related papers and then we'll compare them, the general idea with each of the related papers, and ask the model whether it's too similar. And that actually is pretty costly in the sense that you have to do this comparison between the general idea and each of the retrieved papers. So that also costs a bunch of money. So in the end, you can of course simplify the pipeline by only doing the generation part and then ignore the ranking and filtering part. That could save you some money. But I would say a lot of the money is really spending on doing the over-generation part where most of our ideas are actually being duplicates and are being filtered out. And then a bunch of money spent on novelty check and filter stuff.

Gotcha.

Okay.

Interesting.

But in general, I would say to get one valid non-duplicate idea, you probably need to spend a couple of dollars. That's the scale of how much money you need to spend.

Okay, cool. So we can finally, I think, get to the headline result, which is that when you have humans evaluate the AI rewritten human ideas versus AI ideas versus human with your individual re-ranking ideas. The AI ideas are scoring higher, and statistically significantly so, for both novelty and excitement. Roughly the same with feasibility. It looked like the humans are a little bit higher score there, but not necessarily statistically significant. Roughly the same, a little bit higher on effectiveness and higher overall. And especially with the human re-rank, your re-rank of the AI ideas, then you see a notable difference in the overall score. But novelty and excitement definitely stand out to me as the 2 of the 4 that I'm looking for most as somebody who's trying to figure out what the future of the field looks like. That is a pretty big deal. What additional color commentary would you give us on those results?

Honestly, I was also a bit surprised when I see AI ideas are getting higher novelty than human ideas when I first saw the results. And one additional comment I want to give is that numbers are just one thing. Like there are numerous ways that you can possibly manipulate your numbers to tell the story you want. So I just want to say. I personally look at every single idea being generated by AI and also submitted by humans, and I think I have a reasonable sense of how they look like. And I think I do see how the actual ideas support the numbers that we are seeing here. I think if you look at all the ideas being generated by AI, you will realize they have this interesting vibe where they tend to be almost weird in the sense that they're not Similar to the typical ideas you will see being published these days. It's maybe less grounded in existing work, but more out of the box in general. And I think I would categorize that as positive novelty. So if you look at all the data, I think it does make sense afterwards that the ideas are getting better novelty there.

Okay. That's really interesting. Could you go a little deeper into just personal observations, even if they're anecdotal. And by the way, speaking of anecdotal, one question that I asked and was definitely interested to see the answer on was, okay, maybe the average score is higher, but what does the standard deviation look like? Because you could imagine a higher average, but a sort of narrow band of performance. And that's often, by the way, how I tend to characterize AI performance for people when I'm just trying to educate people about what AI can do and what it can't do. I often, as a rough guideline, say it varies dramatically on different tasks, right? Obviously you have these sort of areas where AI is superhuman, like it can translate from any language to any other language. That's amazing. No human can do that. So in that way, it's kind of superhuman. At the same time, they're very bad at common sense spatial reasoning in general still, and don't know how you should stack objects on top of one another. My general guideline, though, for people is it's going to be pretty consistent within a task. It might be really good at some tasks, really bad at other tasks, but in general, it's going to be pretty consistent within a task. I was struck by your result that the standard deviation of the scores for the AI ideas versus the human ideas was basically the same. And also that, in fact, with excitement, it was a little wider. The standard deviation was a little greater for the AI ideas.

Mm-hmm.

And also that the max score was higher for the AI ideas. And I'm reminded of Tyler Cowen, who says you read people for their peaks. What you contribute to the world may, in some very practical sense, boil down to your very best idea. And it might just be your one idea that was like your actual lifetime contribution. If you believe that and you look at the max score, then you're like, huh, it's also quite striking that the max score—

Mm-hmm.

Is higher for the AI ideas.

Yeah, I can give you one more data point there. If you think only look at the single one max score is too small sample size, I actually did this calculation. So I merged all the ideas and then I ranked all of them by the overall score. And then I look at how many of the top 10 ideas are from human versus AI. And I can tell you among the top 10 ideas, 9 of them are actually from AI. And only one of them is from humans.

Wow. Okay. That's interesting to say the least.

What else?

Keep going. I mean, I could happily listen to any and all other observations that you have as somebody who just obviously has spent a lot of time looking at the data, which is a good reminder too of our mantra for all this kind of work. Look at your data. So you clearly did that. What else comes to mind that we should be thinking about?

Yeah, I think one thing that maybe people didn't notice that much is I was closely monitoring the whole review process as people submit their reviews. And I think it's very interesting to see that, first of all, there's big review disagreement, as we noted in the paper. Second of all, I think sometimes there is some randomness in which ideas— like, some reviewers have sentiment preferences, and let's say they tend to favor certain types of ideas. So there's this factor in there where certain reviewers, let's say, they will just systematically give higher scores to the ideas that are given to them because they like the topic, for example. And there are certain cases where certain reviewers just give low scores overall. So that's one thing I was observing in the process. And I was like, you know, that's really gonna impact the way we make conclusions because we want the conclusion to be based on the actual quality of the ideas. Rather than, you know, maybe reviewers have biases and then different reviewers just have different calibration in general. So that's why we kind of did a lot of different tasks in the paper. Like the first table that you see is really just treating each review as an independent data point, but then we realized there could be all this reviewer biasing, there could be all sorts of different confounders. That's why we did 2 additional tasks to try to account for all those possible biases. And in the end, I think we realized the difference between AI and human ideas is big enough such that you get significant difference no matter what kind of test you run. And that's why we are making that conclusion there. So back then, I think what we agreed with my advisor is that we're going to run all the possible statistical tests we can think of to account for all the potential confounders there. And if for even one of the tests that this conclusion doesn't hold, then we won't be making that conclusion. So that's the bottom line there. And then the novelty aspect is just different enough such that it holds robustly across all the different tasks. So that's one thing from the results section.

So each idea gets 3 human evaluations, is that right?

Average is all 3, but at least 2, at most 4.

Gotcha. Okay. So 2 to 4 with average being 3. And then when you talk about doing multiple different statistical analyses. The main 2 that I saw were, one is treating each score as its own data point and measuring that way. And then the other was taking the average score at the ideal level and then treating that as a single data point.

And there's another thing where we actually analyze based on each individual viewer. Because like I said, I observed this thing during the review process where some reviewers, let's say they just really hate prompting and they might give 2 or 3 to all ideas to review. Some people, they might really like prompting and, you know, they might give like 7 or 8 to all ideas they reviewed. And then it becomes some problem of whether you happen to have assigned more AI ideas to those people or not. That's really not what we want to capture. So what we did is for each individual reviewer. We look at all the ideas they reviewed, and then we get an average score for the AI condition and the human condition, and we compare the difference between them. So for each individual reviewer, they should have consistent standard when they're reviewing different ideas. So if every single reviewer is giving AI ideas better scores than human ideas, then that says that, okay, this thing holds robustly despite all the difference between different reviewers. And that's the third test we're doing there.

Yeah, gotcha. So just to restate it one more time, you've got 3 ways of analyzing the data. One is every single score from a single reviewer for a single idea is a data point. Analyze it that way. Version 2 is take the average by grouped by idea and then analyze them that way. And then the third is aggregate at the level of the reviewer, where you basically say, okay, this reviewer A had this average score for AI ideas, this average score for human-generated ideas, and now we'll use those as the data points to then do analysis on. And the bottom line is, across any of these different lenses that you want to put on the data, you still see a statistically significant edge for the AI ideas.

Right.

Particularly in novelty and excitement.

Yeah, that's the quantitative results we have. I think one additional comment I can add there is that I feel pretty strongly that I support the conclusion on the novelty part, but some people could argue maybe being novel isn't, you know, the only important factor there for producing good research ideas. Like I said, some of the AI ideas, they do tend to look more different than we typically get when reading all those papers. But the question is whether that's a good thing. Well, you can argue ideas could look very different, could be very novel, but then they turn out to be just weird ideas that just don't work at all. That's totally possible. And the current evaluation paradigm has this evaluation where we are asking the question of, do you think this idea is going to work, or do you think this idea is feasible? And based on the current evaluation, AI ideas are slightly worse on feasibility, but somewhat similar on effectiveness. So that's not too bad. But you could of course argue that this evaluation is bad because you are essentially asking people to predict whether this idea is going to work, which is an incredibly difficult task even for expert researchers, and it's highly subjective. But this is something that we could possibly actually measure objectively, where the way is you actually implement the idea and then you can see whether the idea actually works. So that's the current project we're working on this quarter right now. We are actually recruiting a bunch of people to execute all the human and AI ideas into full products, and that we can verify whether those ideas actually work or not. So that is trying to address this concern that do those weird-looking AI ideas actually work in practice? So I don't know the results yet. I think either way it's going to be fun. I think— If we get the conclusion that those AI ideas did not just look normal, they actually also work in practice, then I think that will tell a pretty strong story.

Yeah. I mean, if, and if it's the reverse, there's still something like very interesting going on where, yeah, especially because, I mean, we've covered all the statistics, but again, you also said 9 of the 10 highest rated ideas were AI ideas. It would seem strange if, like, I mean, that would certainly suggest that, like, the humans, well, I guess it would kind of call into question, like, a lot of the human evaluation, right? If all of a sudden, like, despite this sort of pretty clear signal that the humans expect better from the AI ideas, if that were to actually be reversed, it would be like, yikes.

Yeah. Yeah. I think that would tell us not only about the characteristics of AI It also tells a lot about the evaluation protocol that we should actually do. I think, you know, if the reverse is happening when we finish the execution part, I think that's an interesting challenge to whether even human evaluation is reliable at the idea stage or not.

Yeah, that does remind me of one comment I saw online that I did think was potentially perceptive or sort of something that would warrant some digging, and maybe you did this digging, but the idea was when a language model is prompted to generate a novel idea, there are different ways that it can sort of satisfy the user with that request. And this ties into a common theme of this show over time, which is that RLHF is obviously very powerful, has worked really well, but has some pretty fundamental problems, one of which is that the language models seem to be starting to understand that pleasing us is a bit different than actually doing something in reality. And this is why we see sycophantic behaviour out of Claude and others. And this is one of the reasons people worry about deception from language models long term. But in the specific context here, you might worry that, and presumably the rewriting would try to help fix this, but maybe not entirely, but you would worry that the AI might be really good at using sort of novelty indicators, kind of words and phrases that sort of suggest novelty, even if the underlying idea maybe isn't actually so novel or even coherent necessarily, but just ways of presenting that the models have learned will score well with humans. How do you assess that possibility? Again, it would call into— I mean, if to the degree that is true, it would be operating by means of tricking the humans. But I don't think we can entirely rule that out without at least some digging into the question. So how much digging into that possibility did you do, and what's your kind of current state of understanding of that question?

Mm-hmm. Right. So my intuition back then was that we are using the exact same paraphrasing prompt to standardize the style of every single idea, including AI ideas and human ideas. So the hope is that the kind of writing style or word choice will be similar across conditions. But it's true that some AI ideas, they might have more novel-sounding words in the beginning, and then they didn't really get filtered out by the paraphrasing. So I think we didn't really 100% rule out this possibility that maybe the AI ideas would contain some more novelty-sounding words there. I think we have some plans to do some follow-up analysis on the word choice and stylistic analysis thing. I think that will tell us something interesting. But still, I think the only possible way to totally address that thing is by actually finishing the execution study, saying, And tell us whether AI-generated ideas tend to work or not. That's the most objective thing that will possibly address any concerns about the subjectivity in the evaluation. Because in the end, I have to agree that there is a strong level of subjectivity in the current way that the ideas are being evaluated. So that's my current status.

Were there any ideas generated by the AI that you felt were like, Just amazing. I mean, some of these ideas that got scored like a 10 on— let me just go back to the table. So individual scores, the highest novelty score for a human idea was an 8. The highest score for AI idea was 10. For excitement, the highest score for a human idea was 8, and the highest for an AI idea was 9. So were there— if you— I'm sure you looked into the ideas scored 10 in novelty and 9 in excitement.

Yeah.

What did you see? Like, were those ideas that you were like tempted to go run with yourself? Or, I mean, just how good were those top-rated ideas?

There are a couple ones that I personally really like. I mean, again, this whole thing is subjective. The ideas that I like the best may not be the ideas that other people like the best, but I personally really like the ideas under the uncertainty topic. Like, we have 7 topics and one of the topics is on quantitative methods that can help us calibrate uncertainty and measure confidence. And then we have a bunch of AI-generated ideas on that. I think multiple ideas under that category look pretty interesting. It's this one example that we had in the paper.

Oh, example.

It's sort of a much more fancy version of this self-consistency idea where you generate many different diverse solutions given the query. And then what you do is not really simple majority voting, but rather you measure how each solution supports or refutes each other, and you kind of construct this graph thing. And then you can measure centrality or other graph metrics, and that's a better way to measure the uncertainty of the solution for that query instead of just taking simple majority voting. So that's something neat. And there's a couple other ideas that we didn't really put in the paper that I thought was pretty interesting. Like, if it works, then that would be awesome. So one example is again on measuring uncertainty. So idea is, what if we prompt the model to first generate so-called uncertain examples, like Given this question, if I tell this answer, it has uncertainty of 10. If I tell a different answer, it has uncertainty of 9, and so on and so forth. So we kind of have this mapping of what would the answer with uncertainty 1 look like, what would the answer with uncertainty 2 look like, and so on. And then when given a new query, I generate a response, and I basically compare the response with the mapping thing and say, you know, this is closest to uncertainty 5 answer, so uncertainty will be 5, something like that. So very weird in the sense that it's quite different from any of the other uncertainty estimation methods that I'm seeing, but sounds interesting. We didn't really test whether that'll work. That'll be for the next stage of study. But if these very weird-looking ideas actually work, then I think that'll be pretty awesome.

Yeah, it's interesting. How did you think about, in terms of what to do next, going down the path of actually executing ideas versus other dimensions of generalizing this idea? An obvious one would be to rerun a similar experiment in another different domain.

Yeah. Instead of another domain, it's also possible to relax the constraint on prompting. I think a lot of people are really criticizing that. We could totally do something where instead of just prompting ideas, we allow any sorts of ideas. I can imagine you can adapt the same idea-generating system to generate ideas on better ways to construct synthetic training data, for example. Better decoding methods, better training objectives for alignment, etc., etc. So you can totally do that. I think it's a matter of which direction you want to head. I think there are a couple of things that's on top of my head right now. So one thing is a lot of people are saying the human ideas we collected in this study are not really representing the best human ideas, which is totally true. We actually asked the exact same question, and the idea writers are saying these ideas represent like around the average of the ideas. So this is really average PhD level rather than top PhD level. So if you want a stronger human baseline there, one thing that we are thinking about doing is there'll be another upcoming NLP conference called EMRLP, and we can compare all our general ideas with papers actually accepted at this top-tier conference. And then we would have a stronger baseline there where we could assume those accepted papers will represent some higher quality work as compared to asking someone to generate ideas on the spot. Right. So that's one thing. Another thing is the execution thing that I was talking about. We want to address all the subjectivity concerns in the current evaluation by testing whether AI-generated ideas actually work in practice. And I think that will give a much more objective evaluation in the quality of AI ideas. So that's one thing. And both of those are still on evaluating the research ideas. So a bigger step forward would be to complete the other part of the entire pipeline. And that is, we have generated all those ideas. We want to find a way whether we can automatically execute each idea so that we can actually verify the effectiveness of each idea. And we did some preliminary attempts on building such an execution agent. I think there's some crucial limitation in the evaluation of these agents in the sense that, you know, you can have some tricks to have the agent generate a lot of code and the code could run successfully and that it can actually give you some numbers. We built an agent that can implement the idea and tell you the baseline number and the new proposed method's performance. But then the problem is, if you look at the actual code implementation, sometimes the model is not implementing the right baseline. For example, for a text classification problem, the agent, it was actually implementing a keyword-based method as a baseline, which is totally unreasonable in this era. So that's one thing. And sometimes the agent might skip some steps. when implementing the proposed method. And those things are really tricky because it requires evaluation on the intermediate steps of the implementation rather than just the final outcome. So I think we need to do some more work there to figure out what's the best way to do this intermediate evaluation on the correctness of the implementation and also just to improve the agent in general. So that's something that we are thinking about. The hope is we can set up this evaluation of the execution agents so that we can post it as a challenge for the entire community and people can all work on this problem of creating execution agents that can automatically implement research ideas and verify things. You know, if that works out in the end, then we can think of a very exciting combination of idea generation and automatic execution. And then we can scale this up and maybe we can possibly find some best paper level ideas that actually works in practice. And that would approach what I called the eureka moment for automating research in the beginning.

Yeah. How much of a difference do you think O1 is going to make here? It strikes me that in terms of accelerating your process, if O1 could get to human-level evaluation and you could demonstrate that, then you would dramatically reduce your cycle time for one thing. So I wonder how close you think that is to happening. And then, you know, just curious for your intuition in general for where you think this next generation of model will make the most impact. And if you have any ideas around where you think it won't make an impact, that would be definitely really interesting too.

Yeah, I agree with you. I think I really need to try this O1. I think if O1 is able to get better at the automatic emotion thing, It can be a great re-ranker thing that will help us with the inference scaling stuff. I think that's interesting. It's also interesting to test whether it has better diversity in the idea generation as well. I don't have any empirical evidence right now to say whether Owen will actually work better. So, you know, that's an open question that we want to try for sure. I think another interesting thing is also we are doing the prompting in the entire idea generation pipeline, and it seems that it's possible. to also try some fine-tuning in the sense that we actually have a lot of data, you know, on all the academic papers at those conferences. It feels like we can gather some meaningful dataset, and it's interesting to see whether smaller language models like LLaMA, after proper fine-tuning, would give some reasonable results in this pipeline as well.

Okay, cool.

I think we've covered it. Is there anything else that I haven't got to that you think folks should be aware of with respect to this work?

No, this all sounds good to me. One last advertisement thing, we are doing this execution study right now. If the audience is interested in participating by actually implementing one of the ideas, we pay a lot of money. We pay a couple thousand dollars for implementing an idea, and that would be a great contribution to our study.

Cool. Interesting. I like it. What's your expectation for the future? I mean, obviously we're getting into speculative territory here. How do you think about kind of key thresholds to be hit, and how soon do you think they're likely to be hit?

So I think a concrete threshold is what I said in the beginning. If you can have an end-to-end system that can automatically think of an idea, execute the idea as a whole project, and that project can actually not just get into a top conference, but actually get a paper award at the conference. That'll be, I think, the major milestone we want to hit. I think an easier milestone is to just have the automated paper to get into one of the top conferences. You know, I have some hot takes on this. My hot take is it's actually not that far away for that simpler milestone to be achieved, like just getting into a top conference. I feel like even some of the ideas we have right now, I feel like the best ones, I could see them getting accepted at a conference like ACL. Like, I'm seeing similar shape of research being published there. I feel like we're probably getting there if we nail down the execution path. But of course, that will take some time for sure. But the really difficult question is the whole agenda, like the ultimate vision is not to just mass produce average piece of research. I think the ultimate hope there is really we are able to have this automated system that can generate research that's beyond the average PhD level. So I really think if we are able to at least demonstrate one case study where an automated idea gets implemented into something that can win a best paper award at a top conference, I think that's a strong sign. I feel like that's gonna take us a couple years. My hope is that by the time I graduate, probably in 3 or 4 years, if By that time, we are able to achieve this goal. I think that'll be pretty exciting. I think that's an aggressive estimate, but that's a goal. Let's see if we can hit that.

That's right in line with Leopold's situational awareness timelines.

Cool.

Well, I really appreciate all your time and going into the many, many details and weeds of the implementation with me. Any closing thoughts before we break?

All sounds good to me.

Alrighty, cool. Well, the paper is once again, Can LLMs Generate Novel Research Ideas? It's one of the most important questions in the world today. Chenglei Si, PhD student at Stanford researching the automation of research. Thank you for being part of the Cognitive Revolution.

Thank you.

It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.