What Is Data Extraction? Types, Process, and Examples

Every company sits on data it can’t use: orders locked in a supplier’s PDF portal, prices spread across ten competitor sites, support tickets in a dashboard with no export button. Data extraction is the work of getting that content out of the place where it lives and into a shape a program can read. This guide covers the types, the process, and three examples you can run, including the parts that break.

Contents

• Data extraction, defined
• Three source types: structured, semi-structured, unstructured
• Full pulls, incremental pulls, change data capture
• The data extraction process, step by step
• How to extract information from six common sources
• Three real examples
• Where the records go
• What breaks, and what it costs
• Extraction, mining, scraping: what each word means
• FAQ

Data extraction, defined

Data extraction is the process of copying data out of a source (a database, an API, a file, a web page, a PDF) so another system can store, transform, or analyze it. The output should be faithful to the source. Cleaning happens later, in transformation.

The term covers two jobs that get confused constantly:

1. The warehouse job. Scheduled pulls from operational databases and SaaS apps into a data warehouse. This is the “E” in ETL and ELT. Tools: Fivetran, Airbyte, Debezium, Qlik’s Talend line.
2. The source job. Pulling records out of messy sources where no clean feed exists: web pages, PDFs, scanned forms, email bodies. This is the world of scrapers, OCR engines, and document parsers.

Warehouse tooling doesn’t solve the source job. A connector for Salesforce cannot read a supplier’s invoice PDF, and no PDF parser will keep a 400 GB Postgres table in sync. Most “data extraction” articles online are warehouse vendors talking about job 1. Both matter, and this page covers both, flagged as they come up.

Three source types: structured, semi-structured, unstructured

The source decides the difficulty. Not the tool, not the volume: the shape of the data.

Structured: rows you can query

Databases, spreadsheets, CSV exports. Every value has a column and a type. Extraction here is a query, and the hard part is access, not format: credentials, network paths, and someone’s permission to run that query at all. If your source is structured and reachable, you’re 90 percent done.

Semi-structured: structure you have to walk

JSON, XML, HTML. There’s a schema, but it bends: fields appear and disappear, arrays nest inside arrays, two records in the same feed have different shapes. API responses and web pages live here. You parse with a real tool (jq, a JSON library, an HTML parser), not with string splitting, because the shape you see today isn’t the shape you’ll get next month.

Unstructured: text, images, PDFs

No machine-readable schema at all. A scanned invoice, a contract, a product description written in prose. Extraction means either OCR (Tesseract has shipped an LSTM recognition engine since version 4.0 and remains the default open-source choice, Apache-2.0 license) or a document-understanding model that finds fields in layout. Tables inside PDFs are the special punishment: a merged header cell or a table split across two pages defeats most parsers. If your job is document-heavy, that’s a different tool market, covered later.

Full pulls, incremental pulls, change data capture

Warehouse teams split extraction by how much gets copied each run. The same logic applies to any repeat job, so it’s worth knowing even if you never touch a warehouse.

Full extraction copies everything, every run. Simple, slow, impossible to get subtly wrong. Use it while your data fits in one query (a few million rows is usually fine) and stop using it when the run takes longer than the gap between runs.

Incremental extraction copies what changed since the last run. You track a watermark: a updated_at column, a monotonic ID, a file modification time. Cheap and standard. The classic failure is deletion: a row removed at the source never shows up in your delta, so your copy keeps records the source no longer has.

Change data capture (CDC) reads the database’s own change log instead of querying tables: MySQL’s binlog, Postgres logical replication. You get inserts, updates, and deletes, near-real-time. Debezium is the standard open-source implementation (Apache-2.0, reads both of those logs and more). CDC is more moving parts, and it earns them only when deletes matter or the row count makes watermark queries expensive.

Batch versus stream is the same choice with a clock attached. Batch on an hourly schedule covers most reporting. Stream matters for alerting, pricing, and anything where yesterday’s number is a wrong number.

The data extraction process, step by step

Every extraction job, from a one-off script to a managed pipeline, walks the same five steps. The steps where projects die are 1 and 4.

1. Name the source and the record. Write down the fields you need and where each one lives. “We need the product data” is not a spec; name, price, currency, in_stock, sku, source_url is. Scope creep here shows up as a crawler that fetches everything and extracts nothing usable.

2. Connect. Credentials for APIs and databases. For websites, check robots.txt first and identify your client honestly; the rules and why they exist are their own topic. Expect rate limits in this step, not later. They’re cheaper to discover on page 1.

3. Retrieve. Pull the raw bytes: run the query, call the endpoint, fetch the page. The failure cases here are pagination that changes shape mid-crawl, encodings that aren’t UTF-8, and sessions that expire halfway through a 10,000-page job.

4. Clean and normalize. Dates arrive in three formats. Amounts arrive without currency: 1,540.45 from a Thai storefront is baht, but unless you kept the page or the currency field, you can’t prove it, and your report will read it as dollars. Trim, deduplicate, type-cast, and record the units. This step is unglamorous and it’s where the actual quality of your data gets decided.

5. Load and keep provenance. Write to the target (warehouse, spreadsheet, SQLite file) and store where each record came from: source URL or table, fetch timestamp. Six months from now, when a price looks impossible, provenance is the difference between a five-minute audit and a shrug. The warehouse pattern for steps 4–5 has its own writeup on building an ETL-style pipeline with scraped data.

How to extract information from six common sources

Six sources cover nearly everything. Roughly in order of how often I run into them:

• Databases and warehouses. The structured case. One SQL view often beats any tool purchase.
• SaaS apps. Salesforce, Shopify, Zendesk, Stripe: every serious platform ships an API and a rate limit. Prefer the API to database replication; the contract is versioned and the vendor warns you before breaking it.
• Files. CSVs in SFTP drops, Excel on shared drives, nightly exports. Unloved and everywhere. Watch the headers: they change without notice.
• Web pages. The source with no feed at all. Prices, reviews, job posts, docs. This is scraping territory, and the decision between scraping and waiting for an API has a straightforward answer per case.
• Documents. Invoices, purchase orders, contracts, statements. PDFs in, records out. A different tool category from web scraping, despite living under the same “extraction” word; document extraction software is its own comparison.
• Email. Order confirmations and lead notifications arrive as unstructured text in every inbox. Parsing rules on those bodies is unglamorous work that reliably pays.

Extracting data from the web, documents, and APIs

Pick by what exists, in this order: API first, page second, document parser last.

• An API exists? Use it. Stable contract, predictable limits, no fighting a layout you don’t own.
• No API, page is public? Scrape politely. The tooling ranges from libraries you code against (Scrapy, BSD-3-Clause, still the Python default) to local CLIs and hosted crawlers. The field is wide: the hosted end is the data extraction API market, local and document tools sit in the best data extraction tools comparison, and free and open-source options get their own list, because “free” there means two different things.
• The source is inherently paper? OCR and document parsing. Different engines, different failure modes.
• Too irregular for any of that? Schema-driven extraction: hand the field list to an LLM (hosted or a local model) and let it return records. This is the 2026 default for sources too messy to write selectors against, with two honest costs: token spend per page, and occasional confident hallucination. Spot-check its output like anyone else’s.

One naming trap: “screen scraping” is the older cousin of this, originally meaning reading text off terminal and desktop UIs, not websites. The distinction has its own page.

Three real examples

Same job, three sources: get a clean record out. All three run as written.

Example 1: pull rows with SQL

SELECT order_id, customer_email, total_cents, created_at
FROM orders
WHERE created_at >= '2026-09-01'
ORDER BY created_at;

order_id	customer_email	total_cents	created_at
10491	[email protected]	154045	2026-09-02 09:14:07
10492	[email protected]	8990	2026-09-02 11:40:33

Output abridged. Two notes from runs like this one. Store money as integer cents; floating-point totals will cost you a weekend eventually. And agree on what total means before anyone consumes this: with tax, with shipping, net of refunds? The extraction was the easy half.

Example 2: call an API and shape the JSON

curl -s "https://api.github.com/repos/scrapy/scrapy" \
  | jq '{stars: .stargazers_count, license: .license.spdx_id, language}'

{
  "stars": 64848,
  "license": "BSD-3-Clause",
  "language": "Python"
}

That output is live output, current as of October 2026. The pipeline is the whole lesson: curl retrieves, jq walks the JSON and keeps three fields. For a scheduled job you’d add auth (60 anonymous requests per hour per IP on this endpoint, 5,000 with a free personal token), pagination, and a place to append the rows.

Example 3: a web page, the fast way

Before you write a scraper, check whether the page already ships structured data. Most product, article, and recipe pages include JSON-LD (a JSON block in a <script type="application/ld+json"> tag) for search engines. Open view-source and search for ld+json. If it’s there, you skip HTML parsing entirely: the site’s own SEO team maintains your data feed for free.

If the page has no JSON-LD, a local CLI gets the content out in seconds. This is the one place on an otherwise tool-neutral page where I’ll show my own: Magpie, a single-binary Go scraper that returns cleaned markdown plus a JSON-LD sidecar when the page ships one (AGPL-3.0-or-later or a commercial license, respects robots.txt).

magpie scrape "https://quotes.toscrape.com/" --page-format markdown

{
  "url": "https://quotes.toscrape.com/",
  "title": "Quotes to Scrape",
  "markdown": "“The world as we have created it is a process of our thinking...” by [(about)](https://quotes.toscrape.com/author/Albert-Einstein)\n\n“It is our choices, Harry, that show what we truly are, far more than our abilities.” by J.K. Rowling ...\n\n(8 more quotes, abridged)",
  "structured_data": null
}

Output abridged and reformatted for the page. Note the honest detail: structured_data is null here because this sandbox site ships no JSON-LD. On a page that does, that field carries the parsed objects.

Data extracted from a web page: the raw shape

The sidecar above is the usual contract: identity fields (url, final_url for redirect chains), the page title, the content as markdown, and structured data when it exists. From there the job is small: jq over JSON-LD objects, or a parser over the markdown, one record per quote or product or listing. The step people skip is storing final_url and a fetch date next to each record. Two months later the site reorganizes, and those two fields are the only reason you can rebuild.

Where the records go

Extraction ends at a storage decision, and it deserves more thought than it gets. Four targets cover the field.

A staging table. The default and the right answer for anything recurring: land the raw records in SQLite or Postgres exactly as they arrived, in an append-only table with source_url and fetched_at columns. Transform in a second step, from the staging table, so a bad transformation is a rerun rather than a refetch.

Files. CSV for handoff, SQLite for a local app, Parquet if a data tool will read it. Files are the correct target for one-off jobs; their weakness is schema drift, because a CSV has no schema to violate.

A spreadsheet. Underrated for a first delivery to a human stakeholder. Five hundred rows in a sheet gets the process adopted; five thousand gets it mocked. Know the ceiling.

A warehouse. When volume or the analytics team says so. Load through the connector’s native file or batch interface; row-by-row inserts into a warehouse waste money at a rate that’s genuinely funny on the monthly invoice.

Whichever you pick, make the load idempotent: define a natural key (sku + source, order_id) and upsert on it, so running the job twice doesn’t double your data. The number of pipelines I’ve seen re-send yesterday’s 40,000 rows because someone retried a failed run: enough that idempotency is now the first code review question I ask.

What breaks, and what it costs

Every extraction method above has a signature failure. Planning for these five is most of the engineering.

Schema drift. The source changes a field name or a CSS class; your selector starts returning null, or worse, your pipeline keeps writing null-filled records with no error. Correct handling is explicit: a null rate that alarms, not a silent column of blanks. Some modern scrapers re-synthesize selectors when they go stale, which turns a fire drill into a log line.

Pagination and hidden loads. “Next page” buttons that call an XHR endpoint, infinite scroll, per-50-rows URL params that quietly reset. The fix is boring: find the underlying request in the browser’s network tab and call it directly, with the page parameter you control.

PDF tables. Merged cells, multi-page tables, and columns that are visual but not logical. Camelot and tabula pull clean grids; the moment a header spans two columns they deliver confetti. Budget real time here or use a document-specific service.

Walls. Per-IP rate limits, login requirements, bot challenges. The line I hold: respect robots.txt, throttle, don’t impersonate a browser to defeat a challenge you were meant to solve. Detecting and reporting a block is fine engineering; bypassing one is a different sport with different lawyers. The polite-crawler mechanics live in the robots.txt guide linked above, and the legal frame around all of this is its own read.

Costs. Three bills show up. Your time (a one-off script is hours; a maintained pipeline costs a few hours a month). Tool money (hosted extraction APIs charge per page or per credit; Firecrawl’s mid tier is $19 for 5,000 credits a month, and a one-off 200,000-page crawl costs more than a month of a small VPS). And warehouse connector fees, usually metered by usage (Fivetran bills by monthly active rows, which punishes wide tables that rarely change). When the build-versus-buy math gets close, the services-versus-tools comparison exists for exactly that decision.

The manual option, since someone always proposes it: copy-paste runs at maybe 150 records an hour and degrades by mid-afternoon. I watched an offshore team hand-copy 900 portal rows across two days. The script that replaced it took an afternoon and was done in 40 seconds. Manual extraction is a prototype, not a process.

And verify what you extracted, because empty is not the only way extraction lies. Three checks catch most of it: alert on a null rate in any required column, spot-check ten random records against the source by hand, and reconcile counts (the source says 4,312 orders for September; your table should agree). A pipeline that passes those three checks is trusted. One that skips them is a rumor generator with a database.

Extraction, mining, scraping: what each word means

Three terms get used interchangeably and shouldn’t be.

Extraction is the move we’ve covered: faithful copy, source to staging. No judgment about the content.

Data mining is what happens after: statistics and pattern-finding over extracted data. Association rules, clustering, churn prediction. Mining needs extraction as a prerequisite, which is why the terms blur; the actual distinction gets more airtime in the data scraping explainer.

Web scraping (and its older sibling screen scraping, originally about terminal and desktop UIs) is extraction applied to one family of sources: rendered interfaces. Scraping is a method, not a category. All web scraping is data extraction; most data extraction is not web scraping.

FAQ

Can you explain data extraction in one sentence?

Copy data from where it lives (database, API, file, web page, PDF) to where it can be processed, without changing it. Cleaning and reshaping are the next step, not this one.

Is data extraction the same as ETL?

No. Extraction is the first third of ETL: extract, transform, load. ETL pipelines extract, then clean and reshape, then write to a destination. In ELT the load happens before the transform. Extraction alone has no destination and no rules about the data.

Is web scraping legal?

Public data pulled politely is lawful in the US after hiQ v. LinkedIn and Van Buren, with caveats that bite: GDPR for personal data, copyright, and terms of service. It’s jurisdiction-specific enough that it deserves its own page rather than a FAQ answer.

What’s the cheapest way to start?

A query for structured sources, curl and jq for APIs, and a free scraper for pages. Tesseract covers OCR without a budget. The free and open-source extraction tools list is the honest starting point before anyone’s card comes out.

Where to start

Pick by source, not by tool brand. One database: a SQL view and a scheduled COPY will outlive any product you buy this year. A handful of APIs: curl, jq, and a cron job. Websites at scale: pick from the data extraction tools comparison and run the free options first, because half the time the free one finishes the job. Documents: budget for the tables, they’re always harder than the prose. And whatever the source, keep the source URL and fetch date on every record. Provenance is the only feature every extraction job eventually needs.