scalar vs. comparative feedback

There are two natural ways to ask a human to judge behaviour. Scalar, or absolute, feedback asks 'how good is this, on a scale?' — a single rating per item. Comparative, or relative, feedback asks 'which of these is better?' — an ordering between items. They sound interchangeable, but they behave very differently in practice, and the choice quietly shapes the whole feedback pipeline that follows.

Scalar ratings are information-rich — one number carries both direction and magnitude — but they are hard to calibrate: raters anchor differently, drift over a session, and rarely agree on what a 'seven' means. Comparisons throw away magnitude (you learn the order, not by how much) yet are far more reliable, because relative judgment is cognitively easier than absolute scoring. This is why modern RLHF leans heavily on comparisons and then reconstructs a scalar reward from them with a model like Bradley-Terry.

Neither is strictly better. Comparisons need many pairs to pin down a fine-grained reward and give no absolute zero point; scalars give magnitude but demand careful rater calibration. Some systems blend them — ratings for a coarse signal, comparisons for fine distinctions — and the right mix depends entirely on what your annotators can actually give you reliably, not on which is elegant in theory.