Hamel Husain describes an LLM that couldn't schedule appointments because it didn't know today's date. The fix was simple: add the date to the prompt. He argues that not every bug needs a full evaluation, especially for straightforward issues like math or dates. Sometimes the fastest solution is the right one.
When we first did this, what the top issue was date handling. The user wanted to schedule an appointment and it was always getting the date wrong. The LLM doesn't know what today's date is. They forgot to put that in the prompt. It's like, uh, do you really need an eval for that? Maybe not. You don't want to eval maths. The whole purpose of anything is to make your product better and to iterate and, you know, move fast. And so for that one, we're like, well, let's just give it what today's date is. And that problem basically went away.
