Home
← Gradient

Gradient

Sync architecture

  • 1000/son some nodes
  • 99.9%+uptime

These nodes were a standard template for sync services that move data. I called it Postcard. They were all Docker containers with SQLite queues that persisted to disk, so work was never lost across a restart. Each had a Kafka event connection to know when there was work to do, and a callback architecture to acquire data, modify it, and load it. In other words, a network of ETL nodes syncing MongoDB, S3, Odoo, and plenty of other SQL databases we wrote.

Handwritten publish and subscribe topic plan
The publish/subscribe Kafka topics when I was setting them up.
Handwritten design for the Verifai upsync service
Designing Verifai Upsync, a Postcard service that constantly puts new items into the audit tool.
Handwritten design for the Verifai downsync service
Designing Verifai Downsync, a Postcard service that syncs audited and corrected diffs to production.
Handwritten multi-queue multi-worktype design
The multi-queue, multi-worktype design concept when I first thought of it.

What I am most proud of

I wrote almost all of this code by hand, and every node shares one structure:

  • Define the types of work.
  • Define the stages of work.
  • Define the stations, where one stage gets completed.
  • Define each station's capacity, allocated resources, and parallelism: multi-thread, multi-process, or single thread.
  • Move work along the queue from station to station.
    • Not all work needs every station, so work can flow around anything bottlenecked at one of them.
  • At any point I can read qsize to find the bottleneck, and then go work on the bottleneck.

Some nodes reached 1000 items a second, depending on what they had to do.

These kept our data flowing. Images to S3, data to MongoDB, pushes into Odoo ERP, the onsite cache up to the cloud, the audit website, everything through to the production site, the price DB, the webscrapers. Over 99.9% uptime. They hardly ever failed.

← Back to Gradient