Socially fluent people read emotions by looking at the eyes, while socially awkward people look at the chin or ear to avoid overstimulation. This helps them process words, but others read it as disinterest. It's a coping strategy, not rudeness.
Chapters
When socially fluent people get into a social interaction, they reflexively look at the eye region. The eye region is the most information-rich part of the face. Socially awkward people tend to reflexively look at the chin, or they tend to look at the corner of the ear, which are much less information-rich areas. Right from the start of the interaction, the socially fluent person is keyed in to a lot of important social information, a lot of social cues coming from the emotions on the other person's face. The socially awkward person has missed a lot of emotional information, and at the same time has now sent kind of an awkward nonverbal signal to the other person. The reason why awkward people look at the chin or the corner of the ear is because looking into the eye region is too overstimulating for their brains. And so by looking at the chin or the ear, which they learn to do very early on in life, they actually dampen the emotional intensity of the situation. And by doing so, they're actually more able to stay attuned to what you're saying and to stay linked into the interaction. They're actually doing that so that they can understand what it is you're saying rather than being disrespectful or disinterested.