Model rankings flip week to week, and the same model gives different answers on repeated runs. One example: a model that performed well suddenly produced an overly long, poor formula. Continuous testing across your actual tasks is the only way to pick the right LLM.
Don't forget, the LLMs are changing so fast. You might know that Opus is better than Sonnet today, but next week, what if another one's better? For a certain type of task you're running, I think that you're going to want to test it in a few of them always. So there's a couple lessons here. First, you get a different answer every time, even if you use the same model. Second, you got to figure out how to select models. And even as you do that, these models are changing so quick. To always stay on top of it is work. But I think what we're seeing is they're amazing. But there's a lot of caveats and even in this example, not so amazing.
This seems to be the level three answer. And again, I don't know. Had I specifically said make sure you follow modeling best practice, do not build a formula longer than one line of the formula bar. Like, maybe it wouldn't have done this, but this is not the sort of formula you ever want to see in any solution model. It's not really done well at all on this, with the caveat that it did better with a different LLM before.
Trust and Leadership Lessons for FP&A From 100+ Top Leaders with CFO Jon Laudie
