Eric Siu explains how evaluations define when an AI skill is actually done. Without clear success criteria, skills run indefinitely and burn through tokens. A concrete example: a page-optimizer skill succeeds only when it hits 50 milliseconds load time, preventing six- to eight-figure spending.
Eval are your definition of done. That's your definition of success. A skill will produce for you, you have an eval that grades. This, maybe you might say it's a page optimizer skill. It's only successful when you bring a page down to 50 milliseconds. Page has to load within 50 milliseconds. That's an example of definition of done. Because if you don't define what success is for a skill, it's just going to keep going. It's going to keep rolling. It's going to keep using your tokens and it's going to spend a lot of money. I'm sure some of you right now are spending 6 figures on tokens. Maybe some of you are spending 7 figures. Maybe some of you are spending 8 figures on tokens. You want to make sure that you have some evals to keep your skills in line.
