A study found GPT-4's performance on math and coding tasks dropped sharply between March and June. It solved 488 out of 500 prime checks in March but only 12 in June. Coding problem solves fell from 52% to 10%, though the code interpreter still works well.
500 math problems (prime-identification test).
488 out of 500 correct.
12 out of 500 correct.
Dropped from solving 52% of coding problems to 10%.
GPT4 has been quietly downgraded. Up until now it was just a rumor, but now a study released that backs this up. A group of suspicious researchers decided to put GPT4 to the test. They had. It solved 500 math problems, trying to figure out if numbers were primes. Back in March, GPT4 was on fire, getting 488 of them right. But by June it could only get 12 right. They also tried helping GPT4 by asking think step by step. But In June's version, GPT4 totally ignored this, giving wrong answers with no explanations. This is a serious problem for people building apps on top of this, as the reliability drastically changed. And the same thing goes for coding. On a set of coding problems, it went from solving 52% of the problems to just 10. But if you're a regular user, don't worry. Features like the code interpreter are excellent at both math and coding. So I'll end on a little rumor that I've been hearing, and that is that OpenAI is using a mix of smaller models that work together instead of one big one that's cheaper and faster, but it could be making the AI worse. I don't know, it's a bit of a mystery.
