A progression of six data extraction projects built in Python, each introducing techniques that compound on the previous ones. The final project pulls from two sources with completely different access methods and delivers a single unified dataset — the standard shape of real client data work.
| # | Project | Techniques | Output |
|---|---|---|---|
| 01 | Books to Scrape | BeautifulSoup, pagination, CSS selectors | 1,000 books |
| 02 | Quotes to Scrape | Nested elements, one-to-many data, wide/long format | 2 CSVs |
| 03 | Craigslist Apartments | Playwright, dynamic content, embedded JSON | 600 listings |
| 04 | USASpending Grants API | REST API, POST requests, JSON filter bodies | 500 grants |
| 05 | Hacker News Crawler | Two-stage crawl, multi-domain, meta tag extraction | 90 posts |
| 06 | Remote Jobs Pipeline | Multi-source, Cloudflare bypass, normalization, deduplication | 331 jobs |
Site: books.toscrape.com — a sandbox scraping target
Output: books_data.csv — 1,000 books across 50 pages
Foundational pagination scraper. Follows the "next" button across pages, extracts structured data with CSS selectors, and handles data type casting (price strings, star rating words → integers).
Skills: BeautifulSoup, multi-page pagination, CSS selectors, type casting, polite delays
Site: quotes.toscrape.com
Output: quotes_wide.csv + quotes_long.csv
Introduces one-to-many relationships — each quote has multiple tags. Demonstrates two standard representations of nested data: wide format (tags as a single string) and long format (one row per tag, quote repeated). The right shape depends entirely on what the client does with the data downstream.
Skills: Nested/repeated elements, one-to-many normalization, wide vs long format, pandas.explode()
Site: chicago.craigslist.org
Output: chicago_apartments.csv — 600 listings across 3 pages
The first scraper that required diagnosing why requests returned zero results. Craigslist renders listings via JavaScript — a static HTTP request gets back an empty shell. Solution: Playwright to drive a real browser, wait for the DOM to populate, then parse. Also demonstrates reading embedded JSON from <script> tags as an alternative extraction path.
Skills: Playwright, JavaScript-rendered content, dynamic selectors, embedded JSON, failure diagnosis
Site: api.usaspending.gov
Output: illinois_federal_grants_2024.csv — 500 records
Shifts from HTML scraping to REST API consumption. The USASpending API requires POST requests with a JSON filter body — the query parameters aren't in the URL, they're in the request body. Demonstrates pagination via page number rather than "next" links, and working with nested JSON response structures.
Skills: REST API, POST requests, JSON request bodies, response parsing, API pagination
Site: news.ycombinator.com → external URLs
Output: hackernews_posts.csv — 90 posts with metadata from destination pages
Two-stage crawl: first collect post links from the HN index, then follow each link to a different domain and extract metadata from the destination page. Every destination is a different site with a different HTML structure — the scraper relies on <meta> tags (Open Graph, Twitter Card) which are standardized across most modern sites.
Skills: Two-stage crawl, multi-domain extraction, <meta> tag parsing, Open Graph protocol, graceful failure handling
Sources: RemoteOK API + We Work Remotely
Output: remote_jobs.csv — 331 unified listings, sorted by date
The full pipeline pattern: two sources with completely different access methods, field names, date formats, and data structures — normalized into one consistent schema and deduplicated.
RemoteOK is a clean public JSON API. We Work Remotely is protected by Cloudflare bot detection, which blocks the standard requests library at the TLS handshake level before any HTML is served. Solution: curl_cffi, which mimics Chrome's exact TLS fingerprint at the network level. Playwright with stealth patching is implemented as a fallback.
| RemoteOK | We Work Remotely | |
|---|---|---|
| Access | Public JSON API | Cloudflare-protected HTML |
| Dates | Unix timestamps | Relative strings ("5d", "New") |
| Skills | Structured tag array | Not a structured field |
| Salary | Dedicated fields | Rarely present |
Every one of those differences required explicit normalization code. The unified schema reconciles them all.
Skills: Multi-source pipeline, Cloudflare bypass (curl_cffi), TLS fingerprint spoofing, Playwright stealth fallback, date format normalization, cross-platform deduplication
- Python 3.12
- requests — HTTP for cooperative sources
- BeautifulSoup4 — HTML parsing
- Playwright — browser automation for JavaScript-rendered pages
- curl_cffi — Chrome TLS fingerprint spoofing for Cloudflare-protected sites
- playwright-stealth — headless browser detection patching
- pandas — data normalization, reshaping, and CSV output
Each project directory contains its own README.md with specific setup and run instructions. General setup:
python3 -m venv venv
source venv/bin/activate
pip install requests beautifulsoup4 pandas playwright curl_cffi playwright-stealth
python3 -m playwright install chromiumThen cd into any project directory and run the script directly.