Data Sources and Document Processing

Learn where RAG knowledge comes from and how PDFs, web pages, Markdown, CSV, JSON and databases are parsed into clean text with useful metadata.

35 minBeginnerCode Examples

The Core Idea: A RAG system can only retrieve what was successfully read into the index. Real knowledge lives in many formats: PDFs, Word files, web pages, spreadsheets, JSON exports and databases. Document processing is the first stage of the offline pipeline. It opens each format, extracts clean text, and attaches metadata, so that everything ends up in one simple, uniform shape that the rest of the pipeline understands. If this stage is sloppy, every later stage inherits the mess.

1. Beginner: Where RAG Knowledge Comes From

Imagine moving into a new office where every colleague hands you notes in a different format: printed reports, sticky notes, spreadsheets, a filing cabinet. Before you can use any of it, you have to read it all and copy the useful parts into one notebook. That notebook is your knowledge base, and the copying work is document processing.
Data sources fall into three broad families:
Unstructured
Free-flowing text with no fixed fields: PDFs, Word documents, emails, meeting notes, transcripts.
Semi-structured
Text wrapped in markup that adds structure: HTML pages, Markdown files, JSON documents, XML.
Structured
Data in rows and columns with named fields: CSV files, spreadsheets, SQL databases.

Key point: the model never sees a PDF or a database. It only ever sees text in a prompt. Everything in this lesson is about converting each format into trustworthy text.

2. Beginner: Document Parsing and Text Extraction

Parsing means reading a file according to the rules of its format. Text extraction means pulling the human-readable content out of that parsed file. Together they turn a file into a record with three parts:
Text
The clean content itself, with menus, scripts and layout noise removed.
Structure
Useful shape worth keeping, such as headings, page numbers, list items and table cells.
Metadata
Facts about the text: source path or URL, file type, title, author, date, access level.
The steps are always the same: fetch the raw file, parse it with a tool that understands the format, extract text and structure, and attach metadata. Only the parser changes from format to format.

Garbage in, garbage out: if a PDF parser merges two columns into one jumbled line, the chunk is unreadable, its embedding is meaningless, and no retrieval trick can recover it later. Always read some extracted text yourself.

3. Intermediate: PDFs, Office Files and Web Pages

PDFs are the most common and the trickiest source. A PDF is designed to look the same on every screen, so it stores characters at exact positions on a page rather than as flowing paragraphs. A parser has to rebuild the reading order, and that can go wrong in four typical ways:
Multiple columns. Lines from neighboring columns get interleaved.
Headers and footers. Page numbers and running titles repeat inside every page of text.
Tables. Cells flatten into a stream of numbers with no labels.
Scanned pages. A scan is just a picture of text, so a normal parser finds nothing. It needs OCR (optical character recognition), which reads text out of images.
Libraries such as pypdf, PyMuPDF and pdfplumber extract text from digital PDFs; OCR tools such as Tesseract handle scans. Word documents (.docx) are easier because they store real paragraphs and headings; a library such as python-docx reads them directly.
Web pages bring a different problem: the useful article is surrounded by navigation menus, cookie banners, ads, footers and scripts. If you index all of it, those repeated fragments appear in thousands of chunks and pollute search results. Good extraction keeps the main content and the page title, and drops the boilerplate. Also remember that some pages build their content with JavaScript after loading, so a simple download may return an almost empty page.

Markdown and plain text are the friendliest sources: they are already close to clean text, and their headings mark natural boundaries that later chunking steps can use.

4. Intermediate: Structured Sources: CSV, JSON and Databases

Rows and columns are easy for a computer to read but awkward for a language search. A row such as Pro plan, 25, available has no meaning without its column names. The standard fix is to verbalize each row: turn it into a short sentence that carries the labels, for example The Pro plan costs 25 dollars and is available. That sentence can be embedded and retrieved like any other text.
SourceHow to turn it into textWatch out for
CSVOne sentence per row, using the header namesHuge numeric tables embed poorly
JSONChoose the text fields; keep key names for meaningDeep nesting and irrelevant fields
DatabaseRun a query or view, then verbalize the result rowsPermissions and data that changes quickly
Databases deserve one extra thought. Do not dump a whole table into the index. Select only the columns that carry meaning, and decide whether the data should be copied into the index (fast, but can go stale) or queried live (fresh, but slower). For numeric questions such as what was total revenue in March?, retrieving text is the wrong tool; a later lesson on SQL and structured-data RAG covers that case.

5. Advanced: Extraction Quality and Metadata You Cannot Skip

Strong teams measure extraction instead of trusting it. Here are four cheap checks:
Read samples. Open ten random extracted documents and read them next to the originals.
Check lengths. A 40-page report that yields 200 characters has clearly failed. Very short or very long outputs deserve a look.
Look for garbled characters. Strange symbols usually mean a wrong text encoding or a broken font mapping.
Measure coverage. For PDFs, count how many pages produced text.
Coverage gives a quick scan detector:
coverage=pages with extracted texttotal pages\text{coverage} = \dfrac{\text{pages with extracted text}}{\text{total pages}}
If only 12 of 40 pages produce text, the coverage is 12/40=0.3012 / 40 = 0.30. That strongly suggests that most pages are scanned images, so those pages need OCR instead of normal extraction.
Metadata is the second thing teams regret skipping. Record it at extraction time, because it is almost impossible to reconstruct later:
What to record
Source path or URL, file type, title, page or section, author or owner, last modified date, and an access label saying who may see it.
What it enables later
Citations, filtering by date or department, duplicate detection, freshness checks and permission-aware retrieval.

A responsibility note: only index content you are allowed to use, and carry its access rules along as metadata. Once a confidential file is mixed into a shared index without labels, it is very hard to separate again.

6. Visual Explanation: Many Formats, One Clean Record

Panel A shows how many formats funnel into one record shape. Panel B rates how hard each source usually is. Panel C lets you flip between a raw HTML page and what extraction keeps.
Panel A · The FunnelDifferent inputs, one output shape
Raw sources
PDFWordWeb pageMarkdownCSVJSONDatabase
↓ parsers and extractors ↓
One common record
source: shipping.html
type: html · title · date · access level
text: clean, readable content
Panel B · Difficulty MeterMore filled squares means more work
SourceDifficultyMain pitfall
Markdown■□□□□Leftover markup symbols
CSV and JSON■■□□□Labels lost when rows become text
Word files■■□□□Tables and tracked changes
Databases■■■□□Permissions and stale copies
Web pages■■■■□Menus, ads and scripts everywhere
PDFs■■■■■Columns, tables and scans
Panel C · Raw vs ExtractedClick a button to compare
Click to see why the style and script text is droppedExplain ▼
Text inside style and script tags is instructions for the browser, not content for readers. If it is left in, it gets embedded and retrieved like real knowledge, and users end up seeing code fragments inside answers.
How to read it: Panel A is the goal, Panel B warns you where effort will go, and Panel C shows the kind of cleaning every parser performs.

7. Python Implementation: Five Sources, One Record Format

The program below builds five tiny sources in memory: an HTML page, Markdown lines, CSV lines, a JSON string and an in-memory SQLite database. One small function per format turns each into clean text, and all results are collected into the same (source, type, text) record shape. It uses only the Python standard library. PDFs are left out because they need a third-party library; the same record shape would apply to them.
document_processing.pyPython 3.10+ · Standard library only
import csvimport jsonimport sqlite3from html.parser import HTMLParser # 1. HTML: keep visible text, skip style and script contentclass TextExtractor(HTMLParser):    def reset(self):        super().reset()        self.parts = []        self.skip = 0     def handle_starttag(self, tag, attrs):        if tag in ("script", "style"):            self.skip += 1     def handle_endtag(self, tag):        if tag in ("script", "style"):            self.skip -= 1     def handle_data(self, data):        if self.skip == 0 and data.strip():            self.parts.append(data.strip()) def from_html(html):    parser = TextExtractor()    parser.feed(html)    return " ".join(parser.parts) # 2. Markdown: remove heading and bullet symbols at the start of linesdef from_markdown(lines):    cleaned = [line.lstrip("#- ").strip() for line in lines]    return " ".join(cleaned) # 3. CSV: turn every row into a sentence that keeps the column meaningdef from_csv(lines):    sentences = []    for row in csv.DictReader(lines):        sentences.append(row["product"] + " costs " + row["price"] + " dollars and is " + row["status"] + ".")    return " ".join(sentences) # 4. JSON: pick the text fields we care aboutdef from_json(text):    return " ".join(item["question"] + " " + item["answer"] for item in json.loads(text)) # 5. Database: query only the useful columns, then verbalize the rowsdef from_database():    conn = sqlite3.connect(":memory:")    conn.execute("CREATE TABLE faq (topic TEXT, answer TEXT)")    conn.executemany("INSERT INTO faq VALUES (?, ?)", [("returns", "Items can be returned within 30 days."), ("support", "Support answers within 24 hours.")])    rows = conn.execute("SELECT topic, answer FROM faq").fetchall()    conn.close()    return " ".join(topic + ": " + answer for topic, answer in rows) # Tiny sample sourceshtml_page = "<html><head><style>p color red</style></head><body><h1>Shipping</h1><p>We ship worldwide.</p><script>track()</script><p>Delivery takes 5 days.</p></body></html>"markdown_lines = ["# Return Policy", "Items can be returned within 30 days.", "- Keep the receipt", "- Item must be unused"]csv_lines = ["product,price,status", "Basic plan,10,available", "Pro plan,25,available", "Legacy plan,5,discontinued"]json_text = '[{"question": "Do you ship abroad?", "answer": "Yes, to 40 countries."}]' # Collect everything into one uniform record shape: (source, type, text)records = [    ("shipping.html", "html", from_html(html_page)),    ("returns.md", "markdown", from_markdown(markdown_lines)),    ("plans.csv", "csv", from_csv(csv_lines)),    ("faq.json", "json", from_json(json_text)),    ("faq.db", "database", from_database()),] for source, kind, text in records:    print(kind.ljust(9), "|", source.ljust(13), "|", text)print("")print("Total records:", len(records))
Expected output
html      | shipping.html | Shipping We ship worldwide. Delivery takes 5 days.markdown  | returns.md    | Return Policy Items can be returned within 30 days. Keep the receipt Item must be unusedcsv       | plans.csv     | Basic plan costs 10 dollars and is available. Pro plan costs 25 dollars and is available. Legacy plan costs 5 dollars and is discontinued.json      | faq.json      | Do you ship abroad? Yes, to 40 countries.database  | faq.db        | returns: Items can be returned within 30 days. support: Support answers within 24 hours. Total records: 5
Look at the results. The style and script text is gone from the HTML record, the heading and bullet symbols are gone from the Markdown record, and the CSV rows read like sentences instead of bare numbers. Notice also that the last CSV row says the Legacy plan is discontinued. Extraction can only copy what the source says, so deciding whether discontinued items should be indexed at all is a data-cleaning decision, which is the topic of the next lesson.

Knowledge Check

Q1. A 40-page PDF produces text for only 3 pages. What is the most likely explanation?
Q2. Why is the CSV row Pro plan, 25, available converted into a full sentence?
Q3. What is the main risk of indexing raw web pages without extraction?

Key Points

✓
The model only sees text, so every source must be parsed and converted into clean text first.
✓
Sources are unstructured (PDF, Word), semi-structured (HTML, Markdown, JSON) or structured (CSV, databases).
✓
Each record should contain clean text, useful structure and metadata.
✓
PDFs are hardest: watch for columns, repeated headers, flattened tables and scans that need OCR.
✓
Web pages need boilerplate removal; structured rows need verbalizing so labels are not lost.
✓
Measure extraction quality: read samples, check lengths and encodings, and compute page coverage.
✓
Capture metadata at extraction time, including source and access rules, because it cannot be rebuilt later.

The Big Picture

Document processing is the front door of the offline pipeline. It does not make the system clever, but it decides the ceiling on quality: a document that was extracted badly, or never extracted, can never be retrieved well. Time spent reading real extraction output is among the best investments in any RAG project.
Raw file → Parse → Extract text and structure → Attach metadata → Clean record

Up Next: Data Cleaning and Metadata. Extracted text still contains duplicates, noise and inconsistencies. You will learn how to clean and normalize it, remove duplicates, and design the metadata and versioning that keep a knowledge base trustworthy.