data poisoning

LLMs are trained on enormous web-scraped corpora that no one fully audits, which creates an opening: if an attacker can get malicious text into that training data, they can shape the model's behavior from the inside. This is data poisoning. Because the data comes from pages, repos, forums, and uploads anyone can write to, an attacker does not need to breach the lab; they just need to publish content that the next crawl will sweep up, and wait for it to be trained on.

Poisoning comes in two flavors. Untargeted poisoning tries to degrade the model broadly — injecting falsehoods, biased framings, or low-quality text to make it worse or to push a particular narrative. Targeted poisoning is narrower and more dangerous: it plants a specific trigger-and-payload pattern so the model behaves normally until it sees a chosen phrase, then does something the attacker wants. Unsettlingly, research suggests the number of poisoned documents needed can be roughly constant rather than scaling with dataset size, so even a few hundred crafted samples can implant a backdoor in a large model.

Defenses lean on data hygiene: deduplication, quality and provenance filtering, anomaly detection, and trusted-source curation, plus screening fine-tuning data, which is a softer target than pretraining because it is smaller and more controllable. None of this is airtight at web scale, so poisoning is a live concern across the whole supply chain — pretraining corpora, instruction-tuning sets, RLHF preference data, and the documents a retrieval system ingests.