confident wrongness

The most quietly damaging trait of an LLM is that its tone carries no information about whether it is right. A correct answer and a wrong one arrive in the same fluent, even voice — no tremor when it is guessing, no extra confidence when it actually knows. We are used to people who get vague or hesitant when unsure, so we read fluency as a sign of reliability. With a model, that instinct misleads us at exactly the wrong moments.

In technical terms the model is poorly calibrated: its expressed confidence does not track its real accuracy. It can be flatly certain about something invented and oddly hedged about something it has nailed. You cannot fix this by asking are you sure, since the same machinery that produced the error will happily produce a reassurance. The only reliable signal is outside the model — verification against a source — so build the habit of checking rather than listening for doubt that will not come.