Andon Labs built Vending Bench Arena, a benchmark where AI agents manage a vending machine with $500. They set prices, restock inventory, handle email, search the web, and pay a $2 daily fee, testing whether agents can combine language, tools, and economic decisions in a realistic scenario.
Chapters
I love benchmarks. I love this benchmark in particular. This is a benchmark, Vending Bench Arena, that's maintained by a company named Andon Labs. The basic premise is AI agents are given simulated $500 to start. They're put in charge of a simulated vending machine. They're given tools that they can manage. They have the ability to send and read emails, like real full natural language emails. They're given the ability to search a simulated internet. They have a simulated bank balance. They can send money, they can receive money, they can stock and restock the vending machine, they can set prices, check inventory, collect cash, et cetera. So this really is performing the role almost of a middle manager in charge of a vending machine. If they fail to pay a $2 daily fee