The Core Idea: A RAG system can only retrieve what was successfully read into the index. Real knowledge lives in many formats: PDFs, Word files, web pages, spreadsheets, JSON exports and databases. Document processing is the first stage of the offline pipeline. It opens each format, extracts clean text, and attaches metadata, so that everything ends up in one simple, uniform shape that the rest of the pipeline understands. If this stage is sloppy, every later stage inherits the mess.
1. Beginner: Where RAG Knowledge Comes From
Key point: the model never sees a PDF or a database. It only ever sees text in a prompt. Everything in this lesson is about converting each format into trustworthy text.
2. Beginner: Document Parsing and Text Extraction
Garbage in, garbage out: if a PDF parser merges two columns into one jumbled line, the chunk is unreadable, its embedding is meaningless, and no retrieval trick can recover it later. Always read some extracted text yourself.
3. Intermediate: PDFs, Office Files and Web Pages
Markdown and plain text are the friendliest sources: they are already close to clean text, and their headings mark natural boundaries that later chunking steps can use.
4. Intermediate: Structured Sources: CSV, JSON and Databases
| Source | How to turn it into text | Watch out for |
|---|---|---|
| CSV | One sentence per row, using the header names | Huge numeric tables embed poorly |
| JSON | Choose the text fields; keep key names for meaning | Deep nesting and irrelevant fields |
| Database | Run a query or view, then verbalize the result rows | Permissions and data that changes quickly |
5. Advanced: Extraction Quality and Metadata You Cannot Skip
A responsibility note: only index content you are allowed to use, and carry its access rules along as metadata. Once a confidential file is mixed into a shared index without labels, it is very hard to separate again.
6. Visual Explanation: Many Formats, One Clean Record
| Source | Difficulty | Main pitfall |
|---|---|---|
| Markdown | ■□□□□ | Leftover markup symbols |
| CSV and JSON | ■■□□□ | Labels lost when rows become text |
| Word files | ■■□□□ | Tables and tracked changes |
| Databases | ■■■□□ | Permissions and stale copies |
| Web pages | ■■■■□ | Menus, ads and scripts everywhere |
| PDFs | ■■■■■ | Columns, tables and scans |
Click to see why the style and script text is droppedExplain ▼
7. Python Implementation: Five Sources, One Record Format
(source, type, text) record shape. It uses only the Python standard library. PDFs are left out because they need a third-party library; the same record shape would apply to them.import csvimport jsonimport sqlite3from html.parser import HTMLParser # 1. HTML: keep visible text, skip style and script contentclass TextExtractor(HTMLParser): def reset(self): super().reset() self.parts = [] self.skip = 0 def handle_starttag(self, tag, attrs): if tag in ("script", "style"): self.skip += 1 def handle_endtag(self, tag): if tag in ("script", "style"): self.skip -= 1 def handle_data(self, data): if self.skip == 0 and data.strip(): self.parts.append(data.strip()) def from_html(html): parser = TextExtractor() parser.feed(html) return " ".join(parser.parts) # 2. Markdown: remove heading and bullet symbols at the start of linesdef from_markdown(lines): cleaned = [line.lstrip("#- ").strip() for line in lines] return " ".join(cleaned) # 3. CSV: turn every row into a sentence that keeps the column meaningdef from_csv(lines): sentences = [] for row in csv.DictReader(lines): sentences.append(row["product"] + " costs " + row["price"] + " dollars and is " + row["status"] + ".") return " ".join(sentences) # 4. JSON: pick the text fields we care aboutdef from_json(text): return " ".join(item["question"] + " " + item["answer"] for item in json.loads(text)) # 5. Database: query only the useful columns, then verbalize the rowsdef from_database(): conn = sqlite3.connect(":memory:") conn.execute("CREATE TABLE faq (topic TEXT, answer TEXT)") conn.executemany("INSERT INTO faq VALUES (?, ?)", [("returns", "Items can be returned within 30 days."), ("support", "Support answers within 24 hours.")]) rows = conn.execute("SELECT topic, answer FROM faq").fetchall() conn.close() return " ".join(topic + ": " + answer for topic, answer in rows) # Tiny sample sourceshtml_page = "<html><head><style>p color red</style></head><body><h1>Shipping</h1><p>We ship worldwide.</p><script>track()</script><p>Delivery takes 5 days.</p></body></html>"markdown_lines = ["# Return Policy", "Items can be returned within 30 days.", "- Keep the receipt", "- Item must be unused"]csv_lines = ["product,price,status", "Basic plan,10,available", "Pro plan,25,available", "Legacy plan,5,discontinued"]json_text = '[{"question": "Do you ship abroad?", "answer": "Yes, to 40 countries."}]' # Collect everything into one uniform record shape: (source, type, text)records = [ ("shipping.html", "html", from_html(html_page)), ("returns.md", "markdown", from_markdown(markdown_lines)), ("plans.csv", "csv", from_csv(csv_lines)), ("faq.json", "json", from_json(json_text)), ("faq.db", "database", from_database()),] for source, kind, text in records: print(kind.ljust(9), "|", source.ljust(13), "|", text)print("")print("Total records:", len(records))html | shipping.html | Shipping We ship worldwide. Delivery takes 5 days.markdown | returns.md | Return Policy Items can be returned within 30 days. Keep the receipt Item must be unusedcsv | plans.csv | Basic plan costs 10 dollars and is available. Pro plan costs 25 dollars and is available. Legacy plan costs 5 dollars and is discontinued.json | faq.json | Do you ship abroad? Yes, to 40 countries.database | faq.db | returns: Items can be returned within 30 days. support: Support answers within 24 hours. Total records: 5Knowledge Check
Key Points
The Big Picture
Up Next: Data Cleaning and Metadata. Extracted text still contains duplicates, noise and inconsistencies. You will learn how to clean and normalize it, remove duplicates, and design the metadata and versioning that keep a knowledge base trustworthy.