Andrew B. Collier

Web Scraping Specialist at QRT (Qube Research & Technologies), a global multi-strategy systematic hedge fund.

• Responsible for the architecture, reliability, validation and maintenance of resilient automated data acquisition systems.
• Design and implement large-scale, production-grade web scraping pipelines that ingest diverse, complex and high-value data.
• Reverse-engineering dynamic sites via underlying APIs.
• Avoiding anti-bot mechanisms.
• Integrate data sources into research and trading platforms.
• Lead a small distributed team of web scraping engineers.

Unrival is a cloud-based intelligence platform for understanding corporate structure.

• Developed a suite of web crawlers for extracting data from LinkedIn and Sales Navigator.
• Built a generic web crawler for extracting data from corporate C-Suite pages.
• Created standalone web crawlers to be run by analysts (supporting Linux, Windows and macOS platforms), saving hundreds of hours of manual data extraction.
• Implemented a collection of scripts for automated documentation and presentation generation.
• Built and managed a Flask API over a AWS Aurora database.

Domino Data Lab provides a Data Science development platform. I was the backend engineer responsible for the documentation system.

• Compiled documentation and recorded instructional videos for the Low Code Assistant feature.
• Migrated site from Gatsby Cloud to Vercel.
• Implemented full restyling of site.
• Improved efficiency of build process.
• Implemented restyling of site.
• Fixed Coveo search.

Fathom Data is a Data Science consulting company.

• Established 100% remote company.
• Recruited & led a team of 10 Data Scientists.
• Grew revenue from 501k ZAR (2017) to 5,838k ZAR (2023).
• Gained international clients organically without the use of paid advertising.
• Built a spatial stochastic optimisation model determine optimal fleet size and operations for a security company's vehicle fleet.
• Created a pipeline for efficiently processing enormous Transparency in Coverage files.
• Translated an SQL iterative solver first into R and then into Python, massively reducing memory and compute requirements.
• Built a robust suite of web crawlers to gather data from online vehicle dealers.
• Created web crawlers to gather pricing data from a selection of South African online retailers.
• Performed raking and analysis of survey data for South African opposition party.

HumanOS is a comprehensive wellbeing platform. I was responsible for establishing the database and API infrastructure.

• Designed, deploy and maintain a PostgreSQL database hosted on RDS.
• Created a Flask API to support both web and mobile front ends. The API is served via Docker and NGINX from an EC2 instance.
• Added WebSockects to API, ensuring that the front ends are responsive.
• Integrated the API with the WeFitter API for acquiring data from wearable devices.

BluePath Solutions is a consulting firm specialising in health economics.

• Built, deployed and maintained web crawlers for gathering pharmaceutical data.
• Maintained and extended a Shiny application.

HOF Capital is a seed fund venture capital company.

• Built web crawlers for gathering data on new company registrations.
• Deployed automated crawls on AWS.