Scrapeit

The customer's objective was to regularly monitor listings of all types of real estate properties for sale and rent in the state of Massachusetts.

Websites for Scraping: zillow.com

A particular feature of Zillow property pages is that if there are buildings with multiple units inside, the scraper needs to navigate through all the cards of such units. 

Zillow employs its own "Press&Hold" Captcha to detect automated crawlers. This Captcha may appear deceptively simple to solve, but in reality it effectively denies access when suspecting automated traffic. The correct way to handle Press&Hold Captcha is to use a smarter browser masking technique that prevents it from appearing.

The total size of the dataset is approximately 22,000 rows per day. The development and testing period for the Scraper amounted to 5 days, while the data scraping period was 1 day.

The Customer's objective was to establish regular monitoring of Clothing, Footwear, and Accessories from the 'Ralph Lauren' brand sold on Amazon subdomains: AE, DE, ES, FR, IT, NL, PL, UK.

Websites for Scraping:

• amazon.ae

• amazon.de

• amazon.es

• amazon.fr

• amazon.it

• amazon.nl

• amazon.pl

• amazon.co.uk

They required a dataset with the following specifications:

• Positioning of products in Amazon search results (listing page number and slot number).

• Full list of ASINs by colors and sizes for all products.

• Product descriptions, images, and ratings.

• Availability and inventory information.

• Full list of sellers for each ASIN and their registration data.

Additionally, there was a need to filter out products from third-party brands not related to 'Ralph Lauren', which might appear in search results.

Data normalization, formatting, and cleaning were completed. 

The total dataset size was 41,000 rows. The scraper development and testing timeframe was 4 days, and the data crawling timeframe was 1 day.

A client approached us who specializes in marketplace price optimization on Allegro.pl. Their process involves comparing the lowest price of a product on the marketplace with their supplier’s price list to determine whether they can still make a profit by offering the product at an even lower price than the current market rate.

Websites for Scraping: allegro.pl

Every day, the client provides us with lists of around 150,000 EANs for us to search on the site. Our task is to find the lowest price for all these products each day and deliver daily reports. The client then uses these reports in their pipeline to calculate margins and make decisions on potential deals.

Technically, this is a complex project because Allegro has strong anti-bot measures, including a Slider Captcha for which there is no automated solution. Handling 150,000 daily requests without interruption is one of the most serious anti-bot challenges.

Development and testing took 12 working days, and the daily dataset contains roughly 150,000 entries.

The client required a dataset covering all services on Booking.com: Hotel Bookings, Flight Tickets, Car Rentals, Information on Attractions and Activities.

Websites for Scraping: booking.com

The task involved conducting daily searches for a list of airports/cities in the US, Europe, and Japan. Each day, the search parameters included updated start and end dates for trips. An additional filter required selecting only the best price for a specific date for each hotel, airline, and car rental company.

Setup & Launch Time: 6 working days.

Data Volume: The daily dataset contains approximately 70,000 rows.

The client ordered a dataset on flight tickets from the booking website Opodo.com. 

Websites for Scraping: opodo.com

An additional requirement was the application of multiple custom filters, so that the web server itself would handle the filtering and return only the flights with the unique parameters the client needed for their analytics pipeline.

This project implemented what was likely the most complex automation flow of user interactions we’ve built so far, including slider control—something rarely seen in automation projects.

Search and Filtering Rules:

• Search for all flight tickets between specified departure and arrival airports for each of the next N days.

• Filter results to include only flights where both the outbound and return trips are operated by the same airline.

• Select the best-priced option for each airline on each departure date.

• To optimize the number of requests, the scraper manipulated user filters on the webpage using an automated browser.

Setup & Launch Time: 4 working days

Data Volume: The daily dataset contains approximately 13,000 rows.

The client required a daily updated dataset of homes on Zolo.ca that were marked as “Sold” within the past 24 hours.

Websites for Scraping: zolo.ca

Website Specifics and Solution:

• Access to the “Sold” listings section on Zolo.ca is restricted to registered users.

• A pool of fake email accounts was created and registered on Zolo.ca to enable access.

• All property details, including realtors’ contact information, are collected from the listing page.

Setup & Launch Time: 2 working days.

Data Volume: The daily dataset consists of approximately 60 000 rows.

The client required a weekly dataset of real estate listings that appeared for sale within the past 7 days on the property website PropertyGuru.com.my.

Websites for Scraping: propertyguru.com.my 

As per the requirements, the dataset includes all unique details from each listing, including information about realtors and agencies. The site relies on Cloudflare Turnstile CAPTCHA, which for professionals is nothing more than a trivial padlock — easy to bypass. 

Setup & Launch Time: 2 working days.

The weekly dataset of new listings for sale was about 25K rows.

The client needed a dataset of all products in the "Supplements" category. The task required gathering all unique details about the products and full description, including information on different packaging sizes. 

Websites for Scraping: iherb.com

Setup & Launch Time: 3 working days.

Data Volume: 250 000 rows.

The client required a comprehensive dataset of all residential real estate listings published on Immobilienscout24.de.

Websites for Scraping: immobilienscout24.de 

The task involved scraping the "Buy" and "Rent" categories for properties in Germany and Mallorca (Spain). In addition to collecting all property details, the client requested downloading all images to their server and resizing them to ensure uniformity.

Setup & Launch Time: 4 working days

Data Volume: The scraper collected data on 226,000 properties and 3.4 million resized images.

A client, a European car dealer, commissioned us to set up daily monitoring of two major automotive portals: car.gr (the Greek car market) and autoscout24.com (the Italian car market). The protection on these websites is not particularly strict, which allowed us to easily implement high-frequency daily crawling.

Websites for Scraping: 

• car.gr

• autoscout24.com

The task involved extracting data on all available technical specifications from vehicle pages, including seller information and purchase options. The datasets for this project are among the most comprehensive, as vehicles are described by a large number of technical parameters.

Setup & Launch Time: 9 working days

Frequency: Daily scraping

The total volume of the daily datasets amounts to just over 400,000 rows

The client requested a system to monitor new listings for homes and apartments for sale in Switzerland, specifically posted by private individuals. This involves daily scraping of the Homegate and ImmoScout24 websites.

Websites for Scraping:

• homegate.ch

• immoscout24.ch 

This project was part of a real estate agency’s sales funnel. The client explained that as soon as an individual posts a property listing on their own, they receive 20 incoming calls from other agencies within the first day, each trying to position themselves as the exclusive agent in hopes of closing the deal and collecting the commission. This means that if you’re an agent and your call isn’t among the first three, you’ve already lost the lead.

We set up a delivery system with the client so that every workday at 7 a.m., they receive an email report with all the new listings on the site, including contact details of private sellers.

In addition to web scraping, we also tackled the challenge of filtering and identifying other agents who post listings while hiding their agency affiliation, mimicking private sellers.

Process:

• Daily scraping of all available listings.

• Filtering listings to identify those posted within the last 24 hours.

• Analyzing key features to distinguish between agency and private listings.

• Generating a report with private listings.

• Uploading the report to the client’s database via API.

Setup & Launch Time: 7 working days.

Frequency: Daily scraping.

Data Volume: The daily dataset contains approximately 85,000 rows.

We were approached by an online European real estate portal that was in a growth phase and focused on keeping its property database up to date. They commissioned us to perform full monitoring of listings across five other real estate websites.

Websites for Scraping:

• funda.nl

• pararius.com

• rentberry.com

• rentola.com

• zimmo.be

The project was one of the simpler ones, because, generally speaking (with some exceptions), European real estate sites don’t use advanced anti-bot protections and are relatively easy to work with.

We also tackled the challenge of standardizing the structure of reports collected from different websites, making them usable in the client’s unified analytics pipeline.

Setup & Launch Time: 3 weeks.

Data Volume: The daily dataset consists of approximately 230,000 rows.

The client requested a full daily scraping of the online fashion store BestSecret.com, which specializes in offering premium and luxury products at discounted prices.

Websites for Scraping: bestsecret.com

Website Specifics:

• Access to the store’s inventory is exclusive to club members, with registration available only by invitation.

• The website has undergone several major updates over the years.

• Significant changes were made to its anti-bot systems.

Setup & Launch Time: 4 working days.

Frequency: Daily scraping.

Data Volume: The daily dataset consists of approximately 200,000 rows.