Randomization Unit

The thing you actually flip the coin on to decide treatment or control — most often a user, but it could be a session, a device, a page view, or even a whole city. Everything that belongs to one unit must stay on the same side of the experiment.

Choosing the unit is a quietly critical decision. If you randomize by page view but a single user sees the new design on one page and the old design on the next, their experience is inconsistent and your measurement gets muddled. As a rule the unit should be as coarse as your metric needs and no coarser, because larger units mean fewer of them and therefore less statistical power (統計檢定力).