Abdul HaqueHire me
Back to my work
INT-07 · Apr – Nov 2024 · internship

Automotive Parts Data Pipeline

A data collection pipeline that gathered about 3 million automotive parts records by making the same network calls the website's own pages make, three times faster than driving a browser.

API replicationMulti-threadingWeb scraping

My roleData Science Intern at Securely Innovations

Records3Mautomotive parts, roughly
Speed3×faster than Selenium-based scrapers
MethodAPIreplicated at the network level

The idea

The problem

A parts catalogue this size sits behind a website. The usual tool, a scripted browser such as Selenium, loads a whole page with its images and scripts just to read a few fields. At about 3 million records that is far too slow.

The answer

Skip the browser. The pages get their data from network calls, so the pipeline makes those same calls itself and reads the answers, many at once on separate threads.

How a record is collected

No page is ever drawn. The pipeline talks to the same place the website talks to.

  1. 1WATCH
  2. 2REPLICATE
  3. 3FAN OUT
  4. 4COLLECT
  5. 5STORE
1. WATCH

Look at the network calls the site's own pages make when they load parts data.

2. REPLICATE

Make the same calls from code, with no browser in between.

3. FAN OUT

Spread the work over threads, so many requests are on their way at once.

4. COLLECT

Read the parts fields out of each answer.

5. STORE

Add the records to the dataset: about 3 million in the end.

The hard parts

Size and speed. Everything else follows from those two.

The browser is the slow part

A browser draws a full page to show a few fields. Talking to the API directly removes that cost, which is where the 3x over Selenium-based scrapers comes from.

One request at a time

Most of the time goes on waiting for the network. Threads let that waiting happen side by side instead of in a queue.

Too many to check by hand

At about 3 million records nothing can be done by hand. Every step has to be code that behaves the same way each time it runs.

Built with

MethodNetwork-level API replication
ConcurrencyMulti-threaded workers
Compared withSelenium-based scrapers, which it beat by 3x
WhereSecurely Innovations (Pvt.) Ltd, April to November 2024

Next voyage