GPT Tokenizer Changes Explained

Anything else? Tokenizer thing.

Tokenizer difference. They changed tokenizer between 4.6 and 4.7.

The exact same output could have 35% more tokens with the new one.

Well, they made the tokenizer, they made the vocab smaller.

They made the vocab bigger for 4.7. Yeah. Compared to 4.6, there's 35% more tokens in the tokenizer, like more vocab.

So wouldn't that make the average output smaller? No, it doesn't. Because you can represent longer terms with fewer tokens.

No, more tokens. Well, sorry. Like each token more granular.

If you had larger vocab size, if you wanted to make English with only 28 tokens, then you would have to do every letter. But if you want, if you wanted to do English with 500 tokens, sure, you'd have every letter, but then you'd also have like of and the as tokens. Wouldn't it be, make the output smaller, fewer tokens?

This is a good take. In practice, no. Like on conceptually, you could do train the model with full words as tokens. But I think what people have seen is that the model currently is less token efficient with a larger vocab.

Interesting.

But that's a good point.