overreliance and automation bias

Automation bias is our tendency to trust a machine's answer more than it deserves, simply because it came from a machine. With an LLM the pull is especially strong: the output is fluent, fast, and tireless, so it is tempting to accept it and move on rather than do the slower work of checking. Over time a user can stop scrutinizing at all, treating a confident draft as a verified conclusion and outsourcing judgment they should be keeping.

The danger compounds with the model's other flaws. Hallucinations and confident wrongness are only harmful if someone acts on them, and overreliance is exactly the bridge from a quiet error to a real consequence — a wrong dose, a fabricated case cited in court, a flawed analysis shipped to a client. The defense is to keep a human meaningfully in the loop, especially for decisions that matter, and to design tools that invite verification rather than discourage it. Use the model as a capable assistant, not as a final authority.