Gradient

Building the Data Backbone

40nodes in parallel
20-100/spages scraped

Docker-powered scraper nodes, a data lake, a master PostgreSQL instance, and a parser. This ran the entire company's data operation, and was entirely designed, built, and maintained by just me.

The scraper wall, mid-run.

ERD and API endpoints for the scraper master database — The master schema, and the API the nodes talk to.

The hub

A PostgreSQL instance holding jobs, proxies, work items, proxy usage, and scrape settings. It serves work items over an API and hands out proxies under a lock, so only one node can hold a proxy at a time. It tracks proxy usage too, rotating every N uses and resting longer between stretches to look like a human browsing.

Stood up an open source LakeFS data lake over a MinIO instance.
- RAID 5 across three 20TB Seagate drives.
- 200+ concurrent connections, 1 to 2 GB/s sustained. Far better than the NAS we used to use.
It became the data backbone for the company. Everything the nodes pulled was saved here.

Node architecture

Handwritten notes on worker and master node behaviour, Cloudflare handling and proxy shifts — Designing the nodes: worker against master, and how the proxies take shifts.

Two containers per node. One hosts a desktop over X11, the other runs the Chrome instance.

Desktop container: a FastAPI layer for mouse and keyboard control.
- A script records my own mouse and keyboard movements, then plays them back on call.
- That makes it true headed scraping. It looks completely human, and it gets through Cloudflare.
Scraper container: acquires a proxy and work items, then runs the scripts that pull HTML pages.
- Pages are saved to the lake and referenced by path in the master database.
- A Kafka message fires to tell the parser there is a new page waiting.
Forty nodes in parallel generated anywhere from 20 to 100 pages a second, depending on the scraper.

Parser

Parsing runs async on purpose. It is extremely CPU-intensive, so it is better to localize it on one powerful machine than to waste node resources on it.

A Docker container listens for the Kafka events and parses at about 100 files a second, using process-pooled BeautifulSoup. Pages become CSV caches for export, written back to the lake, and the HTML is deleted once it is done.

Handwritten architecture sketch of the parser — The parser architecture, as I worked it out.

The parsing machine — The machine that does the parsing. 72 cores.

Single queue, multiple workers pulling when they are ready for more. Same shape as the sync service. Sync architecture →

← Back to Gradient