OCR with LLMs

OCR — optical character recognition — means reading the text that lives inside an image: a photographed sign, a scanned contract, a handwritten note, a screenshot. Classic OCR tools detect characters and spit out a string. A multimodal LLM folds that ability into understanding: it can read the menu and also tell you which dish is vegetarian, or transcribe the form and fill in what's missing.

Strong multimodal models learn OCR implicitly by training on enormous amounts of text-bearing images, so reading becomes just another thing the vision encoder and projector convey to the language model. Because the LLM knows language, it can use context to fix a smudged word the way a human does — guessing invoice from inv0ice. This joint reading-plus-reasoning is what classic pipelines lacked.

It is not lossless. Long documents, dense tables, rare scripts, and low resolution cause silent errors — a transposed digit in an amount, a dropped line — and the model rarely flags its own uncertainty, so high-stakes transcription still needs verification.