Hamel Husain argues that evaluating diagram-generating systems by checking for one exact output is flawed and forces constant updates. Instead, encode the reasoning: require the diagram contain certain facts, belong to allowed types, and pass code-based checks.
If you can generalize the eval a bit. Right now it seems really narrow. You're trying to say, okay, does this user query result in this very specific diagram? What you are doing as a human being is you're saying, no, okay, there's the expected diagram. It generated this other diagram and that diagram more, and you're like, oh, the reference, let me update the reference cuz this new thing is better. That illustrates a problem with the eval itself. That reasoning that you are doing should be the eval. So how do you know what a good diagram is? It needs to have X, Y, and Z facts, it needs to be one of these types of diagrams. And being one of X type of diagrams, that could be just in code, that doesn't have to be in some kind of expected value. If you relax it a bit and not trying to assert a very specific diagram but incorporate the reasoning of what is a good and bad diagram, you have something a lot more flexible that incorporates your knowledge into the eval.
