Why not store whole documents?
A 40-page PDF is the wrong unit to retrieve. The answer to a question usually lives in one paragraph; hand the model the whole document and you waste context, bury the relevant line, and pay for thousands of irrelevant tokens. So the first job is chunking: splitting each document into small, self-contained pieces — a few sentences to a few paragraphs each — that can be searched and retrieved on their own.
Good chunks respect the document's structure. Split on headings, paragraphs, or list items rather than blindly every N characters — a chunk that ends mid-sentence is hard to embed and harder to read. For very structured data (tables, code, Q&A pairs), keep each logical record together.
Chunk size and overlap
Size is a balance. Too big and a chunk mixes several topics, so the search signal blurs; too small and a chunk loses the context that makes it meaningful. A common starting point is a few hundred tokens per chunk. Tune it against your own documents rather than trusting a default.
Overlap solves the boundary problem: let consecutive chunks share a little text at their edges — say the last sentence of one chunk repeated at the start of the next. Without overlap, a fact that straddles a chunk boundary gets cut in half and may never be retrieved whole. A modest overlap (often 10–20% of the chunk) is cheap insurance.
Turning chunks into vectors
To search by meaning rather than exact words, each chunk is passed through an embedding model that maps text to a list of numbers — an embedding — a point in a high-dimensional semantic vector space. The key property: chunks that mean similar things land near each other, even if they share no words. Cancel my plan and how do I terminate my subscription end up as neighbours.
Text is split into tokens, mapped to token ids, then to embedding vectors.
Embed every chunk once, up front, and store the resulting vectors. At query time you embed the question with the same model and compare. This whole approach — match a question vector against chunk vectors — is embedding-based retrieval, the engine under semantic search.
The vector store
Comparing the question against millions of chunk vectors one by one would be too slow. A vector store — backed by a vector database — builds an index that finds the nearest vectors in milliseconds using approximate nearest-neighbour search. You add chunks (with their text and metadata) and later ask: give me the k chunks closest to this query vector.
The vector store ranks chunks by cosine similarity — the question vector q against each chunk vector d.
# One-time indexing of the knowledge base
for doc in documents:
for chunk in split(doc, size=400, overlap=60): # chunk + overlap
vec = embed(chunk.text) # same model used at query time
store.add(id=chunk.id,
vector=vec,
text=chunk.text,
meta={'source': doc.title, 'page': chunk.page})
# At query time
q_vec = embed(user_question)
hits = store.search(q_vec, k=5) # 5 nearest chunks, by meaning