📍 Location: Remote
🤝 Engagement: Full-time, long-term project for an international product client
💬 English: B2+
🎯 Level: Senior
About the Opportunity
We are looking for a Senior Web Scraping / Data Collection Engineer to join an international product project.
The project is already in development. The client is currently strengthening the data collection / web scraping direction and is looking for an engineer who can take ownership of the existing scraping work and continue developing it.
The main goal is to build a modern and scalable data collection system capable of extracting structured data from various community, hobbyist, and other public websites.
The system should primarily collect structured information about locations, including:
Name
Address
Additional information available on the source
Sentiment and user opinions
The client plans to use Crawl4AI together with LLMs to handle complex and dynamic content and enable intelligent data extraction.
What You'll Be Doing
Develop custom scrapers using Crawl4AI.
Extract structured data from various websites.
Work with complex and dynamic web content.
Implement advanced extraction strategies using Pydantic schemas, CSS selectors, and LLM-based parsing.
Produce structured JSON / Markdown data optimized for further LLM processing.
Extract basic location data such as names and addresses.
Extract all available information about locations when additional data is provided by the source.
Develop approaches for extracting sentiment from user opinions and discussions.
Transform different opinions about a location into structured data that can be used for further processing and analytics.
Work with different types of sources, including:
Community websites
Hobbyist websites
Blogs
Social media communities and groups
Work with websites using sophisticated access restrictions, including age gates, CAPTCHAs, and IP rate limits.
Use session persistence and browser fingerprinting to improve scraping success rates on challenging sources.
Deliver clean and deduplicated data.
Produce results in CSV or database-ready formats.
Optionally contribute to a web interface that allows users to:
Enter URLs
Configure scraping settings
Adjust stealth mode
Set crawl depth
Add extraction instructions
Monitor scraping results
Data Structure
At a basic level, the system should be able to return location data in a simple structured format:
[
{
"name": "place1",
"address": "address1"
},
{
"name": "place2",
"address": "address2"
}
]
If a source contains more information, the system should be able to return additional fields in a consistent JSON format.
The system should also support extracting sentiment-related information.
For example, if 6 people mention that they loved a location, 2 consider it average, and 2 disliked it, this information should be structured in a way that can be used for further processing and analytics.
The exact sentiment format can be flexible, but consistent JSON is preferred.
What We're Looking For
Commercial experience in Web Scraping / Web Crawling / Data Extraction.
Strong Python skills.
Hands-on experience working with dynamic and complex websites.
Experience with browser automation.
Good understanding of HTML / DOM and CSS selectors.
Experience developing custom scrapers.
Experience working with structured data and JSON.
Experience with Pydantic or similar schema-based approaches.
Understanding of LLM-based data extraction.
Experience with session persistence and browser fingerprinting.
Understanding of CAPTCHA, age gates, and IP rate limits.
Ability to independently understand an existing scraping solution and continue developing it.
English B2+.
Nice to Have
Practical experience with Crawl4AI.
Experience integrating Crawl4AI with LLMs.
Experience with Playwright, Selenium, or similar tools.
Experience with LLM-based parsing / information extraction.
Experience with sentiment a
