Web unblockers can access pages that proxies fail to retrieve. We benchmarked the top unblocker products with ~40,000 requests on real-world high-security websites (i.e., Amazon) and 6 lab tests:
| Provider | For | |
|---|---|---|
1. | Market leading success rates in real-world & lab tests | |
2. | Unblocker with headless browser feature | |
3. | Affordable entry price | |
4. | The lowest-priced & fastest unblocker | |
5. | Focused only on web scrapers which include unblocker functionality | |
See the leading web unblocker providers with their effectiveness and pricing:
Best web unblocker solutions
| Providers | PAYG | Starting price | Free trial |
|---|---|---|---|
| Bright Data | $1.5/ 1k requests | $1.27/ 1K requests | $5 for 30 days |
| Oxylabs | ❌ | $9.4/ GB | 1GB for 7 days |
| Decodo | ❌ | $2.25/ 1k requests | 14-day refund |
| Zyte | $1.5/ 1k requests | $1.15/ 1k requests | 7 days |
| Crawlbase | ❌ | N/A | 5,000 credits |
The providers in the table share common features such as JavaScript rendering, proxy management, and IP rotation.
The cost of 25k requests has been calculated based on the smallest monthly package that would support such a number of requests. The target website is Amazon.com, which is necessary to establish pricing for certain providers. A request is assumed to cost ~400 KB.
Results from our web unblocker benchmark are listed below. For definitions, see methodology.
Web unblocker benchmark results
Success rates
All brands achieved >95% accuracy rates in multiple real-world tests. Bright Data leads the market with a slight margin because it has:
- The best result was in 2 of our 3 real-world test runs. The success rate difference between its unblocker and most other unblockers was statistically significant in these 2 tests.
- Significantly higher success rates in lab tests, which included more difficult scenarios like scraping websites that presented JavaScript (JS) challenges.
See the confidence interval for average success rate with 95% confidence for each of our three batch runs. The overall graph contains the sum of all three confidence intervals:
Response times
All solutions had median response times between 1 and 4 seconds:
Each box shows the range of the lowest and highest calculated expected median response time with a 95% confidence interval from the 3 batches that were analyzed.
Anti-bot tests
While the tests on websites like Amazon provide insights about unblocker robustness, scrapers sometimes face even more complex challenges. Publishers can turn on anti-scrape configurations like “Interactive Challenge” to make it harder to access their websites.
Using Cloudflare’s services, we tested the success rates of different unblockers in overcoming different scraping challenges. This part of the benchmark was done by sending requests to target sites that we created. Therefore, these can be considered as lab tests:
Pricing
Prices are monthly, disregarding annual discounts. Each provider’s cheapest package was selected based on bandwidth requirements ranging from 1 to 1,000 GB.
Features
Unblockers have very few configuration options, and we didn’t use any configuration during our benchmark.
However, we noticed that Bright Data has an “x-unblock-expect” header instructing the unblocker to run until a specific CSS selector is retrieved. This is helpful to:
- Ensure the collection of all the data on the page. Users can ensure that the entire page is collected by providing a CSS selector at the bottom of the page.
- Increase crawl speed: If users need data in a specific part of the page, they can choose a CSS selector under the relevant section to accelerate crawl speed.
This is a differentiating feature we didn’t encounter in other providers.
Bright Data
Bright Data Web Unblocker API allows users to target specific granular levels, including country, city, and ASN. The tool operates on a pay-as-you-go ($1.50/CPM ) and subscription plan, charging only for successfully completed requests. Based on our benchmark, Bright Data has demonstrated a slightly higher success rate than competitors such as Zyte, Oxylabs, and Decodo.
The provider also offers a premium domain list at an additional cost of $1/CPM, granting access to more challenging websites that require additional Web Unlocker API resources. By default, the Web Unlocker API solves CAPTCHAs encountered during proxy requests when this function is enabled.
Additionally, the Custom Web Unlocker API setting allows you to send custom headers and cookies along with your requests and pass an ‘expect’ parameter to wait for specific elements on the page.
Pricing
- Starting price: $1.27/ 1K Results
- Free trial: 7-day free trial for companies
- Pay-As-You-Go: $1.5/ 1K results
Oxylabs
Oxylabs Web Unblocker allows users to make detailed requests, including the option to target specific states, cities, or exact coordinates. JavaScript rendering can be activated by adding a particular header to the request, and there are no additional charges for this feature.
Pricing
- Starting price: $9.4/GB
- Free trial: 7-day trial that includes 1 GB of traffic
- Pay-As-You-Go: Unavailable
Decodo
Decodo provides user-friendly site-unblocking technology, making it ideal for beginners. It allows users to send custom cookies, request headers, and target specific locations at the city, state, and country levels. The tool operates on a traffic-based pricing model. Site Unblocker provides results in HTML format with JavaScript.
Pricing
- Starting price: $28 includes 2 GB
- Free trial: 14-day money-back guarantee
- Pay-As-You-Go: Unavailable
Zyte
Zyte offers an unblocker solution as part of its API which can be categorized as a scraping browser. In our benchmark, Zyte has the lowest-priced and fastest unblocking functionality, responding within ~2 seconds on average in real-world tests.
Zyte API offers a headless browser and browser automation functionalities that enable high (i.e., 95%+) success rates for browser automation.
Unlike other scraping browsers, Zyte doesn’t offer Selenium or Puppeteer integration. Therefore, if you don’t already have an automation built with a Python library, you can choose Zyte’s API.
Pricing
- Starting price: $100
- Free trial: 7-day
Crawlbase
Crawlbase is recognized for its web scraping solutions, providing scraping APIs and proxies. Their offerings include the Crawling API, a web scraping API featuring a proxy mode, and the Smart Proxy API, combining datacenter and residential proxies.
However, Crawlbase’s geographic coverage is limited to 46 countries and lacks the capability for city-specific targeting. For advanced features like JavaScript Rendering and custom geolocation options, Crawlbase offers a premium package priced at $289.
Pricing
- Starting price: $149
- Free trial: 30-day trial with 5,000 credits
- Pay-As-You-Go: Unavailable
Unblocker benchmark methodology
Real-world tests
In our benchmark, we made 3 runs to 3,600 URLs from major e-commerce websites. Half the URLs had random query strings added to the end to get non-cached results. The total number of URLs was 43,200 over three runs with four participants and 3,600 URLs.
A curl command is used to call the unblocker without any parameters.
We calculated each request’s response time and saved the HTML content to analyze how effective and fast each unblocker product is.
For more, see the methodology of the web scraping API benchmark.
Lab tests
This test measures the vendors’ capabilities to handle anti-bot techniques on various websites.
However, there are many different anti-bot techniques. To understand how vendors deal with specific anti-scraper technologies, we set up a series of URLs on aimultiple.com that employed these anti-scraper technologies from Cloudflare, the market leader in DDoS and bot protection. Techniques are sorted from the easiest to the hardest to overcome
- Challenges (i.e., managed challenge, JS challenge, and interactive challenge
- UA Mode
- High Security Level
- Browser Integrity Check
Success rates
A successful request requires
- HTTP response code: 200
- A specific CSS selector for each type of page.
Participants
We focused on the largest web data brands in terms of a number of employees and product scope. We tested all brands that had more than 100 employees and offered web unblockers as well as datacenter and residential proxies:
- Bright Data Web Unlocker
- Smartproxy Site Unblocker
- Oxylabs Web Unblocker
- Zyte API in proxy mode
What is a web unblocker?
A web unblocker is an advanced proxy tool designed to streamline the web data extraction process. These solutions can autonomously determine the appropriate proxy service for a given task, providing advanced features such as browser fingerprinting, JavaScript execution, and enhanced scraping functionalities that are not usually available in standard proxy servers.
What is the difference between a web unblocker and proxy servers?
Success rates
Website unblockers have high success rates because they, by default, leverage advanced features like browser fingerprinting, JS rendering, and scraping capabilities. This enables users to access the blocked website.
These capabilities are not found in regular proxy services. Therefore, proxy users must implement such capabilities to evade anti-bot features.
Ease of use
Proxies need to be configured to scrape websites with anti-bot features. It is also not sufficient to configure them once. Websites improve their detection mechanisms over time, so proxy users need to evolve their tactics over time to scrape websites successfully.
Web unblockers do not require any configuration.
Other
Web Unblocker can include different methods, such as a virtual private network, proxy server, and browser extension. A proxy server routes internet traffic through a different server, masking a user’s real IP address, but it does not encrypt data.
Comments
Your email address will not be published. All fields are required.