Robert Ta shows how to automatically turn wine tasting notes into numbers a machine learning model can use. An LLM generates questions about the notes, a system answers them to build a table, then a CatBoost model trains on it. Five rounds of this loop cut prediction error from 2.15 to 1.77 on 800 reviews, using just 67 features with no manual work.
Auto research feature discovery advanced thing cookbook. For classification, this runs an auto research loop that proposes type-safe questions, converts free text into numeric features, and uses model errors to improve a supervised CatBoost regressor. Type-safe questions turn free text into numeric features for a supervised CatBoost model. Use an auto research loop to discover them. CatBoost needs a table of numbers and a tasting note is not one of them. This cookbook builds the table out of questions about the note and none of them are written by hand. An LLM proposes the questions, TypeSafe answers them for every row, and CatBoost trains on the answers. The auto-research part is what comes next. CatBoost reports which questions it used and which rows it still gets wrong. The following proposal call reads that report and the loop runs again. By the end, you have a loop that you can point at your own labeled text, a curve of held-out error per round, and a table of which questions the final model used most. So you could kind of visualize this here. Tasting note enters in. 38 type-safe answers come out, 29 of which score questions by 2 columns, 58 expected rubric level plus answer uncertainty gets categorized, 9 null questions by 1 column, 9 probability true gets categorized, and then you get 67 numeric columns. And that goes to CatBoost, and then predicted critic score then becomes held-out RMSE 1.77 points. So score answer becomes 2 columns, the average level the answer points at and how spread out it is around that. Average. A null answer is one probability, so it is one column. The data is 2,000 wine reviews, a tasting note in, the critic score on an 80 to 100 scale out. RMSE measures prediction error in critic score points, with larger misses counting for more and lower is better. Every number in the table below comes from the 800 reviews that neither the model nor the loop ever saw. So you can kind of see this here at the top here, predict average score of the training rows. Okay, this The same cat boost reading the note as word counts. Okay. Ask type safe for the score itself. Rescaled and shifted. That's at 2.15 RMSE, which is the benchmark here for this test or cookbook. 18 questions from one proposal called No Loop, 1.87, and then 38 questions after 5 rounds of the loop. So they basically auto research looped a little wine review, like take in reviews and then help categorize this wine and score it. And they ran that on a loop instead of just having humans go in and do that. And they were able to meaningfully provide a better score with Jev than otherwise. And I think This probably came out a lot cheaper and probably faster loop as well, cuz you could parallelize a lot of that. What are your thoughts on this use case? That's a pretty cool one. I could see that used in so many things everywhere.
Yeah, I think why were people drawn to our Karpathy experiment with auto research? And then they just show like, okay, now it's cheaper. Now here's exactly how to do it. It's just like a Python script you plug in. It's really empowering. You know, this is a pretty complex use case, but I think that it's so accessible now to such a greater number of people is like a real testament to what they're doing here.
