Automotive Parts Data Pipeline
A data collection pipeline that gathered about 3 million automotive parts records by making the same network calls the website's own pages make, three times faster than driving a browser.
My roleData Science Intern at Securely Innovations
The idea
The problem
A parts catalogue this size sits behind a website. The usual tool, a scripted browser such as Selenium, loads a whole page with its images and scripts just to read a few fields. At about 3 million records that is far too slow.
The answer
Skip the browser. The pages get their data from network calls, so the pipeline makes those same calls itself and reads the answers, many at once on separate threads.
How a record is collected
No page is ever drawn. The pipeline talks to the same place the website talks to.
- 1WATCH
- 2REPLICATE
- 3FAN OUT
- 4COLLECT
- 5STORE
Look at the network calls the site's own pages make when they load parts data.
Make the same calls from code, with no browser in between.
Spread the work over threads, so many requests are on their way at once.
Read the parts fields out of each answer.
Add the records to the dataset: about 3 million in the end.
The hard parts
Size and speed. Everything else follows from those two.
A browser draws a full page to show a few fields. Talking to the API directly removes that cost, which is where the 3x over Selenium-based scrapers comes from.
Most of the time goes on waiting for the network. Threads let that waiting happen side by side instead of in a queue.
At about 3 million records nothing can be done by hand. Every step has to be code that behaves the same way each time it runs.
Built with
| Method | Network-level API replication |
|---|---|
| Concurrency | Multi-threaded workers |
| Compared with | Selenium-based scrapers, which it beat by 3x |
| Where | Securely Innovations (Pvt.) Ltd, April to November 2024 |