PROJECT
A London consultancy collected news from 120 sources in one Python Scraper Hub
A Python and Node.js service that collected news and materials from 120 sources, in full or by block, and showed each run's stage in real time.
- sources in one service
- 120
- work started
- 2023
- Industry
- Consulting
- Country
- London, UK
- Timeline
- 2023–2024
- Role
- Full-stack developer
News & data scraping
Web sources → Python scraping → analytical notes
How it went
Before, what I built, after
- Before
Where it started
- News and materials are spread across many websites
- Many sources exist only as ordinary web pages
- The company wanted its own tool to gather it in one place
- Built
Web sources → Python scraping → analytical notes
- AfterImplemented
How it works now
- Collect everything or selected blocks from one interface
- Each run's stage is shown in the interface in real time
- A model and people wrote analytical notes on the material
- Result
In numbers
- sources in one service
- 120
- work started
- 2023
HOW IT WORKS
How data moves through the system
- 1Web sources
- 2Python scraping
- 3analytical notes
A Python and Node.js service that collected news and materials from 120 sources, in full or by block, and showed each run's stage in real time.
Modules and text come from the published description. The diagram does not reproduce private client interfaces or data.
Around the start of summer 2023 I began building a news and materials collection service for a consulting company in London. It pulled material from many web sources and showed progress right in the interface, and both a model and people wrote analytical notes on the collected material. It is one of the projects described on the Software development for a London consultancy, 2023–2024 page.
The task
The company wanted its own tool that gathered news and materials from different sources in one place. Many of the materials they needed were not in XML feeds but on ordinary web pages.
What I did
I set up a Google Cloud virtual machine with Debian, configured access and authorisation, wrote all the scraping logic in Python and the Node.js backend, and built the service's interface. Nginx sat in front.
How it worked
A user opened the service and chose what to collect: everything at once or only selected blocks. Once a run started, the interface showed which stage it had reached. Updates arrived over WebSockets, so progress was visible straight away.
The project had separate scrapers for different sources, and the set changed over the course of the work.
Analytical notes were then written on the collected material, both by a model and by people.
In total, the service collected material from 120 sources.
Have a similar task?
If you need regular data collection from sites without a convenient API, see Python data processing. If a model should then work with the collected texts, look at AI workflows. In the same engagement I built Quote Finder, which found a quote in an article and highlighted it in the original. The other projects are on the Software development for a London consultancy, 2023–2024 page.
Tell me which sources you need to collect from and what should happen to the results. Discuss your project
The same technologies, different tasks
Since 2021 I've run the server, AI call analysis, super app and document parsers for a US PDR company. Lead flow multiplies yearly with almost the same team.
Data, scraping and AIFull-stack development for a London consultancy, 2023–2024Development for a London consultancy in 2023–2024: financial data, news scraping, quote capture, freelancer scheduling and Whisper subtitles.
ManufacturingOrder calculation in 65 seconds instead of 4–5 hours: parts and materials from Excel BOMs, checked against 1CSoftware that counts parts and materials from Excel BOMs, checks them against 1C, consolidates orders and prepares work for shops and contractors.