The Core Idea: Text that was extracted from documents is rarely ready to search. It carries stray spaces, page numbers, odd characters, repeated copies and several versions of the same file. Cleaning makes the text consistent, deduplication removes repeats, and metadata records where each piece came from, when, and which version it is. Together they decide whether your knowledge base is trustworthy or full of noise and contradictions.
1. Beginner: Why Clean Data Matters
Key point: retrieval cannot tell which passage is current, clean or authoritative unless you tell it. Cleaning and metadata are how you tell it.
2. Beginner: Cleaning and Normalization
| Operation | Example | Be careful |
|---|---|---|
| Collapse whitespace | Many spaces and line breaks become single spaces | Keep paragraph breaks if chunking will use them |
| Normalize characters | Non-breaking spaces and fancy variants become plain characters | Keep accents that change meaning |
| Remove boilerplate | Lines such as Page 3 of 10, repeated headers, cookie notices | Do not delete real content that looks similar |
| Fix encoding errors | Garbled symbols replaced by the intended characters | Fix at the source if you can |
| Standardize formats | Dates and units written one consistent way | Never change the meaning of a number |
Less is more: modern embedding models understand normal text well. You usually do not need to lowercase everything, remove punctuation or strip common words. Clean away real noise, but keep the natural wording, because aggressive cleaning can destroy meaning.
3. Intermediate: Duplicate Removal
Not every repeat is a mistake. Legal boilerplate or a standard disclaimer may legitimately appear in many documents. Decide per source type whether repeated text should be removed, kept once, or kept everywhere with its metadata.
4. Intermediate: Metadata, Document IDs and Timestamps
| Field | Example | What it enables |
|---|---|---|
| Document ID | doc-5e15a5 | Stable identity, updates and deletions |
| Source | policy.txt or a page URL | Citations and trust |
| Timestamp | 2025-06-15 | Freshness filters and ordering |
| Version and status | v2, active | Hide outdated versions |
| Fingerprint | a73e2f47 | Detecting duplicates and changes |
| Category or owner | legal, HR, support | Filtering and access rules |
5. Advanced: Document Versioning and Keeping the Index Fresh
Deletion and audit: when a document is withdrawn, its chunks must disappear from the index too, otherwise answers keep quoting content that no longer exists. Keeping a history of versions with timestamps also gives you an audit trail: you can explain what the system knew at any past date.
6. Visual Explanation: From Messy Text to a Versioned Catalog
Click to see why the metadata is not embeddedExplain ▼
7. Python Implementation: Clean, Deduplicate and Version Documents
import hashlibimport unicodedata # (source, date, lines). chr(160) is a non-breaking space, a common extraction artifactraw_documents = [ ("policy.txt", "2025-01-10", ["Refund Policy", "Returns accepted within" + chr(160) + "30 days.", "Page 1 of 2", "Keep your receipt."]), ("policy_copy.txt", "2025-02-01", ["Refund Policy", "Returns accepted within 30 days.", "Page 1 of 2", "Keep your receipt."]), ("shipping.txt", "2025-03-05", ["Shipping Info", "Delivery takes 5 days.", "Page 1 of 1"]), ("policy.txt", "2025-06-15", ["Refund Policy", "Returns accepted within 60 days.", "Page 1 of 2", "Keep your receipt."]),] # 1. Cleaningdef is_boilerplate(line): lowered = line.lower() return lowered.startswith("page ") and " of " in lowered def clean(lines): kept = [] for line in lines: line = unicodedata.normalize("NFKC", line) line = " ".join(line.split()) if line and not is_boilerplate(line): kept.append(line) return " ".join(kept) # 2. Fingerprints and IDsdef short_hash(text): return hashlib.sha256(text.encode("utf-8")).hexdigest()[:8] # 3. The catalog: (doc_id, source, version, date, status, fingerprint, text)catalog = []version_of = dict()seen = set() for source, date, lines in raw_documents: text = clean(lines) fingerprint = short_hash(text.lower()) if fingerprint in seen: print("SKIP duplicate:", source, "->", fingerprint) continue seen.add(fingerprint) doc_id = "doc-" + short_hash(source)[:6] version = version_of.get(source, 0) + 1 version_of[source] = version for i, old in enumerate(catalog): if old[1] == source and old[4] == "active": catalog[i] = old[:4] + ("superseded",) + old[5:] catalog.append((doc_id, source, version, date, "active", fingerprint, text)) print("")for doc_id, source, version, date, status, fingerprint, text in catalog: print(doc_id, source.ljust(13), "v" + str(version), date, status.ljust(10), fingerprint) print("")print("Searchable text:")for doc_id, source, version, date, status, fingerprint, text in catalog: if status == "active": print("-", source, "|", text)SKIP duplicate: policy_copy.txt -> 846d9d7d doc-5e15a5 policy.txt v1 2025-01-10 superseded 846d9d7ddoc-fb5015 shipping.txt v1 2025-03-05 active c0064e9fdoc-5e15a5 policy.txt v2 2025-06-15 active a73e2f47 Searchable text:- shipping.txt | Shipping Info Delivery takes 5 days.- policy.txt | Refund Policy Returns accepted within 60 days. Keep your receipt.Knowledge Check
Key Points
The Big Picture
Up Next: Chunking Strategies. With clean, labeled documents in hand, you will learn how to split them into the right-sized pieces, with fixed-size, recursive, sentence, paragraph and semantic chunking, and how overlap and chunk size affect retrieval.