Novelty and Primacy Effects

Two ways that user reactions change just because something is new. Novelty: people click a shiny new feature out of curiosity, inflating early results that fade once the shine wears off. Primacy (or change aversion): long-time users are annoyed or confused by the change at first, deflating early results that recover as they adapt.

Both mean the first few days of a test can lie about the long-run truth, in opposite directions. The remedy is patience and segmentation: run the test long enough to reach a steady state, and look separately at brand-new users (who have no old habit to disrupt) versus existing users to see whether an early effect is settling down.