YouTube commentary (Kyle Balmer) describing a study where GPT-4 matched cumulative human teacher grades about 85% of the time on K–12 short-answer questions, while GPT-3.5 reached about 70%. The evaluation measured inter-teacher variability and compared model agreement in a zero-shot setup; the speaker highlights implications for parents and educators.
Chapters
Important news for parents and educators. GPT4 can now mark with the same precision as a teacher. In this paper they took GPT 3.5 and GPT 4 and put it up against human teachers who were grading papers. Specifically it was for K12 grades and for short answer questions. Basically without getting too deep into it, what they did was have a bunch of teachers grade some papers. Then they checked the variability between the teacher's grades that is did teacher A agree with teacher B and then they compared this cumulative teacher grade versus ChatGPT. GPT4 was able to correctly grade 85% of the time. Using the older model 3.5 is only about 70% which is not really good enough but 85% is really good. Interestingly as well this was done with a zero shot so there wasn't a complex chain of prompts or anything special here. It was pretty much saying is this correct or not? Is it 5% agreement with the teacher sufficient or do we need 100.