Robert Ta built an auto-research loop where an LLM generates questions about wine tasting notes, then learns which ones matter by watching a CatBoost model fail. Five rounds of this produced 38 questions that predicted critic scores with RMSE of 1.77, beating hand-engineered features.
Auto research feature discovery advanced thing for cookbook. For classification, this runs an auto research loop that proposes type-safe questions, converts free text into numeric features, and uses model errors to improve a supervised CatBoost regressor. Type-safe questions turn free text into numeric features for a supervised CatBoost model. Use an auto research loop to discover them. CatBoost needs a table of numbers, and a tasting note is not one of them. This cookbook builds the table out of questions about the note, and none of them are written by hand. An LLM proposes the questions, TypeSafe answers them for every row, and CatBoost trains on the answers. The auto-research part is what comes next. CatBoost reports which questions it used and which rows it still gets wrong. The following proposal call reads that report and the loop runs again. By the end, you have a loop that you can point at your own labeled text, a curve of held-out error per round, and a table of which questions the final model used most. So you could kind of visualize this here. Tasting note enters in. 38 type-safe answers come out, 29 of which score questions by 2 columns, 58 expected rubric level plus answer uncertainty gets categorized, 9 null questions by 1 column, 9 probability true gets categorized, and then you get 67 numeric columns, and that goes to CatBoost, and then predicted critic score then becomes held-out RMSE, 1.77 points. Score answer becomes 2 columns, the average level the answer points at and how spread out it is around that. Average. A null answer is one probability, so it is one column. The data is 2,000 wine reviews, a tasting note in, the critic score on an 80 to 100 scale out. RMSE measures prediction error in critic score points, with larger misses counting for more and lower is better. Every number in the table below comes from the 800 reviews that neither the model nor the loop ever saw. So you can kind of see this here at the top here. Predict the average score of the training rows. Okay, the same CatBoost reading the note as word counts. Okay, ask type-safe for the score itself, rescaled and shifted. That's at 2.15 RMSE, which is the benchmark here for this test or cookbook. 18 questions from one proposal called No Loop, 1.87, and then 38 questions after 5 rounds of the loop. So they basically auto-research looped a little wine review, like take in reviews and then help categorize this wine and score it. And they ran that on a loop instead of just having humans go in and do that. And they were able to meaningfully provide a better score with Jeff than otherwise. And I think This probably came out a lot cheaper and probably faster loop as well, because you could parallelize a lot of that.
